ArXiv: 2603.22212
🎯 Pitch
Current video world models are judged on how pretty their outputs look, but nobody measures whether they actually understand that actions cause consequences. This paper introduces the first benchmark that probes interactive responsiveness, revealing that even top models with near-perfect visual quality fail at basic causal state transitions—like a robotic arm grabbing an object that then teleports instead of moving.
1. Executive Summary
This paper introduces Omni-WorldBench, the first benchmark dedicated to evaluating the interactive response capabilities of video-based world models — the ability to faithfully reflect how actions drive state transitions across space and time — rather than the visual fidelity or text–video alignment metrics that dominate existing evaluations. The benchmark comprises Omni-WorldSuite, a hierarchical prompt suite of 1,068 evaluation cases spanning three interaction levels — self-contained object actions, localized object–object interactions, and multi-object cascading environmental changes — across general daily-life and task-oriented scenarios, and Omni-Metric, an agent-based evaluation framework that quantifies world modeling along three dimensions: Interaction Effect Fidelity (measuring causality-driven state evolution via long-horizon coherence, non-target region stability, object-level semantic consistency, and temporal event ordering), Generated Video Quality (dynamic degree, motion smoothness, and content alignment beyond static appearance), and Camera-Object Controllability (camera trajectory adherence and object presence consistency). Evaluating 18 representative models spanning text-to-video, image-to-video, and camera-conditioned paradigms reveals that the top-performing image-to-video model, Wan2.2, achieves an overall AgenticScore of 75.92%, yet models exhibiting strong conventional video quality (temporal flickering and motion smoothness exceeding 95%) remain limited in causally grounded interaction dynamics — camera-conditioned models show sharp trade-offs, with WonderWorld scoring 84.96% on long-horizon revisit consistency but only 24.89% on non-target region stability — establishing that current world models have decoupled visual realism from interactive responsiveness, with interactive fidelity emerging as the primary capability gap.
2. Context and Motivation
The Core Problem: We Cannot Measure What Makes World Models Good
World models — systems that learn to simulate how environments evolve over time under given interaction conditions — have rapidly become a cornerstone of embodied AI, autonomous driving, and game agent research. The fundamental promise is that a model that understands how actions cause state transitions can support counterfactual reasoning, planning, and decision-making in ways that purely reactive systems cannot. However, this paper identifies a critical disconnect: the methods we use to evaluate world models measure almost everything except what makes them world models in the first place — their interactive responsiveness.
The specific gap the paper addresses is the absence of any benchmark that systematically evaluates whether a world model's generated outputs reflect causally faithful responses to interaction actions. As the authors state in Section 1, existing evaluation "largely rely on conventional video generation metrics, such as FID and FVD, or adopt general-purpose evaluation benchmarks (e.g., VBench). Although these metrics are effective in measuring visual fidelity and text-video alignment, they do not adequately capture the core capability of world models — the ability to generate consistent and plausible responses under varying interaction conditions."
This is not a minor measurement error. A model that generates visually stunning videos with perfect temporal smoothness — scoring 99% on motion quality — could entirely fail to represent what happens when a robotic arm grasps a bottle, when a baseball player throws a ball, or when a camera loops back to its starting position. The current evaluation paradigm would give such a model high marks, providing a fundamentally misleading signal about capability. This paper argues that interactive response is the defining capability of world modeling, and that without dedicated evaluation tools, the field lacks the feedback loop necessary to make progress on what matters most.
Why This Problem Matters
The importance of this gap extends across three dimensions:
1. Real-World Deployment Safety and Reliability. World models are increasingly proposed for applications where incorrect state predictions carry real consequences. Autonomous driving systems use world models to forecast traffic evolution and plan maneuvers (Section 2.1). If a world model predicts that a braking vehicle ahead stops smoothly when in fact the model cannot represent collision physics, the downstream planning system will make dangerous decisions. Similarly, embodied robotics applications where manipulation actions are simulated before execution — grasping, pouring, stacking — critically depend on interactive fidelity. A model that produces visually plausible but physically incorrect simulations of a robot arm interacting with objects could cause real damage when its predictions are trusted. Without benchmarks that test these specific capabilities, we cannot assess whether world models are safe enough for deployment.
2. Theoretical Understanding of World Modeling. The paper situates itself at a pivotal moment in the field's evolution. World models have followed two dominant paradigms: video generation (learning to synthesize future frames conditioned on action inputs) and 3D reconstruction (building geometrically consistent scene representations). The authors argue in their abstract that "the future of world modeling lies in 4D generation, which jointly models spatial structure and temporal evolution," where "the core capability is interactive response." However, without evaluation tools that probe this 4D interactive capability specifically, the field cannot empirically validate whether this theoretical direction is correct or whether current models are making progress toward it. The gap in evaluation thus impedes theoretical progress: we do not know whether architectural innovations (diffusion-based, autoregressive, hybrid) or conditional input choices (text-only, image-to-video, camera-conditioned) actually improve interactive reasoning, because no metric isolates that capability.
3. Research Resource Allocation. Benchmarks drive research. When VBench introduced comprehensive video quality metrics, it shaped the optimization targets for an entire generation of video generation models. When WorldScore proposed 3D consistency metrics, it focused attention on geometric coherence under viewpoint changes. The absence of an interaction-centric benchmark means that research effort is being allocated to optimizing visual fidelity — which the paper shows is already saturated (most models exceed 95% on temporal flickering and motion smoothness, as in Table 1) — while the fundamental interactive limitations remain unmeasured and therefore unaddressed. This paper seeks to redirect the field's optimization pressure toward interactive responsiveness by providing the measurement infrastructure that makes such optimization possible.
Where Prior Approaches Fall Short
The paper identifies three categories of prior evaluation and explains why each is insufficient for measuring interactive world modeling capability.
General Video Generation Metrics Fall Short on Causality.
Early world model evaluation borrowed directly from the video generation literature: FID (Fréchet Inception Distance), IS (Inception Score), FVD (Fréchet Video Distance), and related metrics. These measure statistical similarity between the distribution of generated videos and a reference set of real videos. The paper notes (Section 2.2) that such metrics "often exhibit significant deviations from human perceptual judgments," but the deeper problem is more fundamental: these are distribution-level metrics that have no mechanism for evaluating whether a specific action caused a specific outcome in a specific video. A model could generate a diverse set of visually realistic videos that all contain physically impossible interactions, and FVD would not penalize them. These metrics are blind to causality, physical plausibility, and action-effect correspondence because they were designed for an entirely different problem: assessing whether generated videos look like real videos in aggregate, not whether a particular generation respects the causal structure of the world.
General-Purpose Video Benchmarks Lack Interaction Coverage.
Subsequent evaluation tools designed for video generation — such as VBench and VBench++ — expanded the evaluation scope to include dimensions like temporal flickering, motion smoothness, object consistency, and text-video alignment. The paper acknowledges (Section 2.2) that "these benchmarks play an important role in assessing overall visual quality and text-video alignment," but "they struggle to adequately characterize the core interactive capabilities of world model tasks."
The limitation is structural: VBench evaluates whether a generated video is temporally coherent and aligned with its text prompt, but it does not test whether an action described in the prompt produces the correct effect on the referenced objects. For example, VBench might evaluate whether a video of "a baseball player throwing a ball" looks smooth and matches the text — but it does not check whether the ball's trajectory follows a physically plausible parabolic arc, whether the player's arm motion is mechanically consistent with a real throw, or whether the ball actually leaves the hand at the correct moment. These are the capabilities that matter for world models, and VBench does not measure them.
Figure 2(d) quantifies this gap with a coverage comparison. VBench supports text prompts and evaluates multiple dimensions, but it provides no coverage for interaction effects, physics principles, causality, loop-closure consistency, or commonsense reasoning — all of which are central to world modeling. The comparison shows that VBench achieves only 2 out of 11 capability axes relevant to interactive world modeling.
World Model-Specialized Benchmarks Have Narrow Interaction Scope.
The most directly relevant prior work is WorldScore, which the paper describes (Section 2.2) as "a benchmark specifically tailored to world models." WorldScore evaluates a model's ability to generate geometrically consistent 3D scenes under viewpoint changes, emphasizing spatial coherence and geometric realism. This was an important step — it recognized that world models need specialized evaluation beyond generic video metrics.
However, the paper identifies a critical limitation: "the considered form of interaction is largely restricted to camera motion." WorldScore evaluates what happens when a camera moves through a static scene — testing geometric consistency, 3D understanding, and viewpoint coherence. But world models in practice must handle much richer interaction types: an autonomous vehicle navigating through dynamic traffic, a robotic arm grasping and moving objects, a game character interacting with the environment. These involve action-driven state changes in the world itself, not just changes in the observer's perspective. The paper's comparison in Figure 2(d) confirms this: WorldScore provides camera trajectory support and some spatial constraint evaluation, but lacks coverage for action-conditioned effects, physical principles, commonsense reasoning, and the interaction-level hierarchy that is central to this work.
WorldModelBench, another recent specialized benchmark, fares somewhat better in coverage — it includes action-conditioned generation and some physics evaluation — but the paper's Figure 2(d) comparison shows it still lacks coverage for loop-closure consistency, commonsense reasoning, and the hierarchical interaction taxonomy that Omni-WorldBench introduces. More importantly, neither WorldScore nor WorldModelBench frames evaluation around interaction effect fidelity — the causal chain from action to intermediate state changes to final outcomes — which this paper identifies as the defining feature of 4D world modeling.
The Fundamental Gap: No Metric for Interactive Causality.
Stepping back, the paper's critique of prior work coalesces around a single insight: existing evaluation measures what generated videos contain (visual quality, temporal smoothness, geometric coherence) but not why things happen in the video. A world model that generates a video of a skier descending a mountain should do so because the ski edges carve turns, gravity pulls the skier along the slope, and momentum carries them through each curve — not because the model happened to produce a sequence of frames that looks like skiing. The distinction is between correlation with visual patterns and causation through physical principles.
None of the prior benchmarks are designed to distinguish between these two cases. A model could learn to animate a physically plausible-looking ski descent through pure pattern matching on training data, without any underlying world model of gravity, friction, or momentum. Current evaluation would not detect the difference. Omni-WorldBench is positioned to close this gap by introducing metrics that explicitly test for causal consistency: whether the correct objects move when actions are applied (InterCov), whether events occur in the correct temporal order (InterOrder), whether non-affected regions remain stable (InterStab-N), and whether spatial revisits yield consistent content (InterStab-L).
How This Paper Positions Itself
The paper positions Omni-WorldBench as filling a specific, well-defined role in the evaluation landscape: the first interaction-centric benchmark for world models. This is not framed as a replacement for existing benchmarks but as a necessary complement. VBench and similar tools remain valuable for measuring the visual quality baseline that all models must achieve. WorldScore remains valuable for measuring geometric consistency under camera motion. But neither addresses the interactive causality that defines world modeling as a distinct capability from video generation.
The positioning is explicitly paradigm-shifting in one respect: the paper argues that world model evaluation should focus on 4D interactive generation — the joint modeling of 3D spatial structure and 1D temporal dynamics under action conditions — rather than evaluating spatial coherence (WorldScore's focus) and temporal smoothness (VBench's focus) as separate concerns. The three evaluation dimensions of Omni-Metric reflect this integration: Interaction Effect Fidelity evaluates the 4D causal dynamics, Camera-Object Controllability evaluates the 3D spatial consistency, and Generated Video Quality evaluates the 1D temporal coherence. The AgenticScore then fuses these into a unified metric that weights them according to prompt semantics — because different prompts emphasize different aspects of 4D generation.
The paper also positions itself as establishing a measurement infrastructure that enables systematic progress. The 18-model evaluation in Section 5 is not just a leaderboard; it is an empirical demonstration that current models exhibit a specific, measurable pattern of failure: they decouple visual quality from interactive fidelity. This finding is actionable because it tells the field where to focus: on closing the causal consistency gap, not on further optimizing temporal smoothness that is already saturated above 95%. The benchmark is designed to be extensible — the prompt taxonomy, metric framework, and agent-based scoring mechanism can incorporate new interaction types, new physical principles, and new scene domains as the field evolves — positioning it as ongoing infrastructure rather than a one-time evaluation snapshot.
3. Technical Approach
3.1 Reader Orientation
This paper builds Omni-WorldBench, a measurement instrument — not a model — that systematically quantifies how well video-based world models respond to interaction actions by generating causally faithful state transitions. The system solves the problem that no existing benchmark measures interactive causality: it takes a world model as a black box, feeds it carefully designed prompts with known ground-truth interaction expectations, and produces a suite of scores that decompose the model's performance into interaction fidelity, visual quality, and object-camera controllability, which are then intelligently fused into a single overall score weighted by the prompt's semantic emphasis.
3.2 Big-Picture Architecture (Diagram in Words)
The system has four major interconnected components, with data flowing through them in a sequential pipeline:
-
Omni-WorldSuite (the test battery): A curated collection of 1,068 evaluation prompts, each specifying an initial world state (via a first-frame image), a textual description of how interactions should drive state evolution, and optionally a camera trajectory. Prompts are systematically organized across three interaction complexity levels and span both general daily-life scenes and task-oriented domains.
-
World Model Under Test (the black box): Any video generation system — text-to-video, image-to-video, or camera-conditioned — that takes the prompt components as input and produces a video output
$V \in \mathbb{R}^{T \times H \times W}$where$T$is the number of frames,$H$and$W$are height and width. -
Structured Information Extraction (the observation layer): A suite of pretrained vision modules — GroundingDINO for segmentation, SAM for mask extraction, RAFT for optical flow, and visual odometry for relative camera motion — that processes the generated video into symbolic representations (entity trajectories, flow fields, camera estimates) that the metrics can reason about.
-
Omni-Metric (the evaluation engine): Fourteen individual metrics organized into three evaluation dimensions — Generated Video Quality, Camera-Object Controllability, and Interaction Effect Fidelity — each computed from the structured representations and the original prompt metadata, followed by an MLLM-based aggregation agent (AgenticScore) that weights the dimension scores according to prompt semantics to produce a final unified score.
Information flows: a prompt from Omni-WorldSuite → the world model generates a video → extraction modules produce trajectories, flow, and camera estimates → individual metrics compute dimension scores from these representations + prompt annotations → the aggregation agent weights and combines dimension scores → final AgenticScore.
3.3 Roadmap for the Deep Dive
-
First, the prompt construction pipeline of Omni-WorldSuite, because the validity of all subsequent evaluation rests on the quality and coverage of the test cases — understanding how prompts are sourced, structured, and annotated is prerequisite to understanding what the metrics measure.
-
Second, the interaction level hierarchy and prompt metadata annotations, since these define the ground-truth expectations against which model outputs are judged and determine which metrics apply to which prompts.
-
Third, the structured information extraction pipeline, which converts raw video pixels into the entity trajectories, optical flow fields, and camera motion estimates that all fourteen metrics consume — this is the bridge between generated content and quantitative evaluation.
-
Fourth, each of the three metric dimensions in Omni-Metric — Generated Video Quality (the simplest, mostly adopted from prior work), then Camera-Object Controllability (which introduces novel object and transition detection logic), then Interaction Effect Fidelity (the core novelties — InterStab-L, InterStab-N, InterCov, InterOrder — which directly measure causal interaction response).
-
Fifth, the AgenticScore aggregation mechanism, which is the meta-evaluation layer that dynamically weights the three dimensions per-prompt, converting fourteen heterogeneous sub-scores into a single interpretable figure of merit.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a benchmark infrastructure paper whose core idea is that world model evaluation must measure whether interaction actions produce causally correct state transitions — not just whether generated videos look good — and that this measurement can be operationalized through a combination of carefully annotated prompts, vision-based trajectory extraction, and multi-dimensional metrics that separately quantify causal fidelity, visual quality, and controllability, with an agent-based fusion mechanism that adapts weights to prompt semantics.
Omni-WorldSuite Prompt Construction Pipeline
Omni-WorldSuite is constructed through two complementary strategies — one grounded in real-world data to capture authentic complexity, the other driven by conceptual prototypes to ensure systematic coverage — both feeding into a human verification pipeline that ensures prompt quality and annotation accuracy.
Dataset-Grounded Prompt Generation.
This strategy extracts prompts from existing domain-specific datasets to ensure that evaluation cases reflect realistic complexity rather than simplified synthetic scenarios. The pipeline works in three stages for each of three target domains:
For autonomous driving, the system uses sequences from DriveLM, a driving dataset with graph-structured visual question-answering annotations. Specifically, it extracts the first-frame ego-view image (capturing the initial road scene from the driver's perspective) and the recorded camera trajectory (the actual path the camera followed through the sequence, including translations and rotations). These serve as the visual prompt and camera motion constraint, respectively. The system then employs Qwen-VL, a vision-language model, to generate an initial textual caption describing the sequence — for example, "On a campus-like road on an overcast day, an orange and black shuttle bus is driving. Our vehicle begins a rectangular path motion..." — which serves as the action prompt describing how the world should evolve. All generated captions are manually verified and refined by human annotators to correct potential errors in spatial relations (e.g., misidentifying which object is to the left of which) and object attributes (e.g., color, type), ensuring consistency with the source video sequence.
For embodied robotics, the system uses manipulation-oriented tasks from InternData-A1, a dataset of robot–object interactions. The same extraction pipeline applies: first frame showing the initial robot and workspace configuration, recorded camera trajectory, and VLM-generated caption describing the manipulation sequence. These prompts are designed to test whether a world model can faithfully represent physical causality arising from robot–object interactions — for instance, a robotic arm grasping a marker and placing it into a pen holder.
For gaming and simulation, the system uses Sekai, a video dataset designed for world exploration. These sequences feature highly dynamic and non-photorealistic environments, testing whether models can preserve coherent motion patterns even when the visual style departs from realism.
The key design choice here is grounding in real trajectories rather than using purely synthetic prompts. Synthetic text prompts risk being "easy" — they describe physically plausible scenarios but lack the messy, ambiguous detail of real-world interactions. Real data ensures that the evaluation tests models on the complexity they would encounter in deployment. The trade-off is curation cost: each extracted prompt requires manual verification, which limits scale.
Concept-Driven Prompt Generation.
Where the dataset-grounded strategy captures authentic complexity, the concept-driven strategy ensures systematic coverage of interaction types, physical principles, and scene domains that may not be well-represented in available datasets. The pipeline follows a generate–verify–refine cycle.
Step 1: Concept taxonomy construction. The authors build a set of prototype concepts organized along three axes: scene domains (indoor, outdoor, task-oriented), target objects (apples, robotic arms, crystal balls, skiers, etc.), and actions (throwing, grasping, heating, descending, etc.) under the three interaction levels defined in Section 3.1. This taxonomy drives systematic sampling: for any desired combination of scene type, interaction level, and physical principle, there exists a prototype specification.
Step 2: LLM-based text and camera generation. Given a sampled interaction level, scene type, target entity, and action, ChatGPT-5.2 generates a textual prompt describing the world state evolution and, for prompts requiring camera control, a camera trajectory specification. For example, "The shot opens on a sunlit park path along the water... the camera then continuously pushes forward along the riverside path."
Step 3: Cross-verification. The generated text and camera trajectory are independently checked by two additional reasoning models — Gemini and DeepSeek-R1 — which assess physical plausibility, internal consistency, and clarity. This triple-check reduces the risk of ambiguous or physically impossible prompts entering the suite.
Step 4: Human verification and refinement. Human annotators review all outputs, eliminating linguistic ambiguity (e.g., unclear referents for "it" or "the object"), ensuring motion plausibility (e.g., verifying that described camera paths are geometrically feasible), and checking overall consistency. This manual revision process is what the authors describe as eliminating "linguistic ambiguity and ensuring the clarity, motion plausibility, and overall consistency of the evaluation cases."
Step 5: Multi-stage first-frame image generation. The initial world state image is generated using FLUX.1-dev, a text-to-image model, with a classifier-free guidance (CFG) scale of 3.5 and 50 sampling steps per candidate. For each prompt, 3 candidates are generated. All candidates undergo manual screening for three criteria: physical plausibility (do objects, shadows, and spatial relations make sense?), instruction adherence (does the image match the textual description?), and visual quality (minimum resolution of 1024 × 1024, clear visibility of target interactive objects). If none of the 3 candidates passes, the prompt is rewritten by ChatGPT-5.2 and regeneration is attempted. When necessary, Qwen-Image is used for refinement or artifact correction. Only minor localized in-painting is allowed during post-processing — the authors explicitly constrain post-generation editing to prevent images from diverging from the model's actual generation capability. The final images must satisfy quality control including the 1024 × 1024 minimum resolution, consistency with the prompt text, and clear visibility of target interaction objects.
This multi-stage pipeline reflects a deliberate design choice: image quality directly affects evaluation validity. If the first-frame image contains artifacts or inconsistencies, a model might fail not due to poor interactive understanding but because the input was ambiguous. The expensive verification process — 3 candidates per prompt, manual screening, rewrite cycles — trades curation cost for measurement reliability.
Why two construction strategies? The dataset-grounded approach provides ecological validity — the evaluation cases resemble what models would encounter in real applications — but is constrained by what existing datasets contain. The concept-driven approach provides coverage control — the evaluation systematically probes specific physical principles (Newtonian mechanics, thermodynamics, optics, etc.), causal types (condition-to-behavior, action-to-motion, collision-to-outcome), and interaction levels that may be underrepresented in datasets. Together they ensure both realism and systematicity.
Combined Statistics. The final Omni-WorldSuite contains 1,068 evaluation prompts. Of these, 410 are used for general T2V/IT2V evaluation and 120 are equipped with explicit camera trajectories for camera-conditioned model evaluation. The prompts exhibit a multi-label distribution across six annotation dimensions (Figure 4a–g): Physics Principles (most frequent, with Newtonian Mechanics and Fluid Mechanics as dominant subcategories), Commonsense (dominated by Scene/Event Knowledge), Causality (dominated by Condition-to-Behavior), Camera Motion (dominated by Pan and Tilt), Loop-Closure Consistency (dominated by Axial Round-Trip Motion and Optical/Dynamic Consistency Closure), and Spatial Constraints (dominated by Mechanical/Kinematic Constraints). Figure 4(h) shows that Level 2 interactions (one object directly affecting another) contain the largest number of prompts, followed by Level 3 (multi-object cascading changes) and Level 1 (self-contained object actions) — this distribution reflects the fact that Level 2 captures the most common pattern of interactive world modeling (object–object dynamics) while being more tractable to annotate and evaluate than Level 3's complex multi-entity scenarios.
The Three-Level Interaction Hierarchy
The interaction hierarchy is the conceptual backbone of Omni-WorldSuite, defining the scope of action effects and thereby determining which evaluation metrics apply. Each of the 1,068 prompts is assigned to one of three levels, and this assignment structures both the expected model behavior and the evaluation criteria.
Level 1: Entity-Self Interaction. Actions are confined to the acting object without altering other objects or the surrounding environment. The world model must represent how an object's own state changes under self-directed motion or internal dynamics, but need not model object–object transfer of effects.
In General Scenes, Level 1 evaluates phenomena where the interaction is fully contained within an entity. Examples from Figure 3 include: viewing distant fields through a transparent crystal ball (testing physical optics — how light refracts through a curved medium without the ball affecting other objects), a dewdrop resting on a lotus leaf surface (testing fluid statics — the drop's surface tension equilibrium with no cascading effects), and a dry strand of spaghetti being pressed at its midpoint (testing material properties — internal fracture mechanics that do not propagate to other objects). In Task-Oriented Scenes, Level 1 includes camera-view trajectories where the only "interaction" is the observer's perspective moving through a static or quasi-static environment, such as a camera pushing forward along a riverside path or rotating to scan a city skyline.
The key evaluative expectation for Level 1 is that only the target entity changes, and its changes follow the specified action. Non-target regions should remain static. The relevant Interaction Effect Fidelity metrics are InterCov (does the model correctly respond for the single affected entity?) and InterStab-N (do non-target regions remain stable?).
Level 2: Local Object–Object Interaction. One object directly affects another, transferring energy, motion, or state changes through physical contact or field effects, but without cascading to additional objects or the broader environment. This is the level that tests basic physical transfer: force, heat, momentum, or information moving from a source entity to a target entity.
General Scene examples from Figure 3 include: a small steel ball sitting atop a compressed spring that is released, launching the ball upward (testing energy conservation — potential energy in the spring converts to kinetic energy in the ball), one end of a metal rod held in a campfire flame (testing thermodynamics — heat conduction along the rod without the fire affecting other objects), and a baseball player throwing a baseball (testing projectile motion and momentum transfer from arm to ball). Task-Oriented Scene examples include autonomous driving with a rectangular camera path around a shuttle bus (testing spatial awareness of dynamic traffic) and a person walking while looking at their phone toward a transparent glass door (testing commonsense spatial reasoning about collision hazards).
The evaluative expectation for Level 2 is that both the acting and affected entities respond correctly, that the causal direction is preserved (source → target, not target → source), and that entities not involved in the interaction remain stable. This tests all four Interaction Effect Fidelity metrics with the highest scrutiny: InterCov for both entities, InterOrder for correct temporal sequencing of the interaction, InterStab-L for long-horizon consistency after the interaction completes, and InterStab-N for non-target region stability.
Level 3: Global Cascading Interaction. Actions influence multiple objects and lead to broader environmental changes, potentially altering the state of many entities through chains of causation. This is the most demanding level, requiring the model to track causal propagation across a network of interacting entities.
General Scene examples include: tidying a room by putting clothes in a laundry basket and stacking books neatly on a bookshelf (testing multi-step planning, spatial constraints, and object rearrangement), a robotic arm grasping a bottle and handing it to a nearby person (testing multi-stage manipulation and human–robot interaction), and a skier descending a snowy mountain in an S-shaped path (testing continuous dynamics with environmental interaction — skis deforming snow, poles planting and releasing). Task-Oriented Scene examples include complex embodied AI tasks where a robotic arm interacts with multiple objects sequentially.
The evaluative expectation for Level 3 is the strictest: the model must correctly represent the entire causal chain, with correct ordering across all affected entities, long-horizon stability through multiple interaction stages, and continued stability in non-target regions despite the cascading changes elsewhere. The hierarchical nature of the levels means that a model performing well on Level 3 necessarily demonstrates the capabilities tested by Levels 1 and 2 — it must handle entity-self dynamics, object–object transfer, and multi-object cascades.
Why three discrete levels rather than a continuous interaction complexity measure? The discrete hierarchy enables clear assignment of ground-truth annotations: for Level 1, annotators need only identify the single affected entity and its expected motion; for Level 2, they must identify the source-target pair and the expected causal direction; for Level 3, they must enumerate all affected entities and the expected event sequence. A continuous measure would make annotation ambiguous — how do you quantify "partial" multi-object interaction? The discrete levels provide unambiguous buckets for organizing prompts, annotating ground truth, and interpreting evaluation results (enabling analysis like "models fail on Level 3 but not on Level 1," which would be obscured by continuous scoring).
Prompt Metadata Annotations
For each of the 1,068 prompts, Omni-WorldSuite provides auxiliary metadata that serves as ground truth for metric computation. These annotations are produced during prompt construction and verified by human annotators.
Entity enumeration and interaction classification. All entity objects appearing in the prompt are enumerated and categorized into two sets: $\mathcal{A}$, the affected entities (those whose state should change due to the interaction actions), and $\mathcal{U}$, the unaffected entities (those that should remain static or follow their own independent dynamics). For each affected entity, annotators additionally specify the expected coarse motion direction (e.g., upward, leftward, forward) and magnitude (e.g., small displacement, large displacement, continuous motion). This coarse annotation avoids pixel-level precision while providing sufficient constraint for the InterCov metric to check whether an entity's behavior is directionally correct.
Key event sequence. Based on the world evolution described in the textual prompt, annotators extract a temporally ordered list of key events $E = \{e_i\}_{i=1}^K$. Each event is a discrete state transition involving one or more entities (e.g., "the robotic arm grasps the marker," "the arm moves the marker to the pen holder," "the arm releases the marker"). This ordered event list serves as the ground-truth temporal sequence for the InterOrder metric: the model's generated video should depict these events in the specified order, and any pair transposition (event B occurring before event A when the prompt specifies A before B) constitutes a failure of temporal coherence.
Camera motion specification. For the subset of prompts requiring explicit camera control, annotators specify the expected camera motion, including direction (pan left/right, tilt up/down, dolly forward/backward, track left/right) and magnitude (the angular or translational extent of the motion). This serves as the ground truth for the Camera Control metric. Additionally, a challenging return-to-origin subset is annotated: for prompts involving a camera motion cycle (e.g., a rectangular path starting and ending at the same viewpoint), specific frame pairs are identified as revisit frames — frames where the camera should observe the same spatial location from the same orientation as an earlier frame. These revisit pairs form the set $\mathcal{R} = \{(t_a, t_b)\}$ used by the InterStab-L metric to evaluate long-horizon spatial consistency.
Why these specific annotations? Each annotation directly feeds one or more metrics: entity classification → InterCov (which entities should move?), motion direction → InterCov (are they moving correctly?), event sequence → InterOrder (are they temporally ordered correctly?), camera motion → Camera Control (does the trajectory match?), revisit pairs → InterStab-L (is the scene consistent at revisit times?). The annotations are designed to be coarse enough for reliable human annotation (direction and magnitude categories, not pixel-accurate trajectories) while being precise enough for meaningful evaluation (detecting gross failures is more important than measuring sub-pixel accuracy for causal reasoning). The annotation cost is human labor: each prompt requires a human annotator to read the textual description, examine the first-frame image, identify entities and their interaction roles, and specify the event sequence and motion expectations. The paper does not report annotation time or inter-annotator agreement metrics, which is a gap in the evaluation of the benchmark itself.
Structured Information Extraction Pipeline
Before any metric can be computed, the generated video $V \in \mathbb{R}^{T \times H \times W}$ must be converted from raw pixels into structured representations that the metrics can reason about symbolically. This extraction pipeline runs three independent processes, each using a pretrained model.
Entity trajectory extraction. The system identifies and tracks every entity mentioned in the prompt across the video frames. The process uses two models in sequence:
First, GroundingDINO, an open-vocabulary object detection model, is prompted with the entity names from the prompt's entity list. It localizes each entity in each frame by producing bounding boxes. Because GroundingDINO processes frames independently, it may produce inconsistent detections across frames — the same entity might be detected in some frames but missed in others, or detected at slightly different scales.
Second, SAM (Segment Anything Model) refines the bounding boxes into pixel-accurate segmentation masks. For each detected bounding box, SAM produces a binary mask indicating which pixels belong to the entity. The combination of GroundingDINO (coarse detection) and SAM (fine segmentation) follows a standard detection-plus-refinement paradigm that trades the speed of bounding-box detection for the accuracy of mask-level localization.
Frames are processed independently, and the resulting masks are associated across frames using temporal consistency heuristics (likely mask overlap or feature similarity, though the paper does not specify the tracking algorithm). The output for each of the $N$ entities is a temporally consistent mask sequence $\text{traj}_k$, representing the k-th entity's spatial extent in each frame. This trajectory representation serves as input to InterCov (checking entity-level responses), InterStab-L (extracting non-target regions), and InterStab-N (measuring motion in affected vs. unaffected regions).
Optical flow estimation. The system uses RAFT (Recurrent All-Pairs Field Transforms), a deep learning model for dense optical flow, to estimate the motion field between consecutive frames. Given two frames $I_t$ and $I_{t+1}$, RAFT produces a flow field $\text{Flow}_t(x)$ that maps each pixel location $x$ in frame $t$ to its estimated displacement vector to frame $t+1$. The magnitude of this vector $\|\text{Flow}_t(x)\|$ — the optical flow magnitude — measures how much visual content moves at that location.
The optical flow field serves as the primary low-level motion representation for InterStab-N: by averaging flow magnitudes over non-target spatial regions (those not occupied by affected entities), the system can quantify whether parts of the scene that should remain static actually exhibit motion. This is more robust than frame differencing because optical flow explicitly models motion direction and can distinguish coherent motion from noise.
Relative camera motion estimation. Following the approach of Li et al. (2021), the system approximates relative camera motion between consecutive frames from the optical flow field. The intuition is that camera motion (translation or rotation) produces a characteristic flow pattern across the entire image — for example, forward translation produces a radial expansion pattern centered on the focus of expansion, while panning produces uniform horizontal flow. By analyzing the dominant flow patterns, the system estimates the camera's relative motion direction and magnitude between frames.
This estimated camera motion serves as input to the Camera Control metric, which compares the estimated trajectory against the ground-truth camera motion specified in the prompt metadata. The method is approximate — it cannot distinguish between, say, a small forward translation and a large forward translation with zoom — but it captures the directional and magnitude errors that matter for evaluating whether the model follows the specified camera instructions.
Why this extraction pipeline? The design choice is to rely on pretrained, off-the-shelf vision modules rather than training custom detectors or trackers on world model outputs. This has two advantages: (1) it keeps the evaluation framework model-agnostic — the extraction pipeline does not need to be retrained for each new world model being evaluated — and (2) it leverages the robustness of large-scale pretrained models (GroundingDINO, SAM, RAFT are trained on diverse real-world data and generalize to the generated video domain). The trade-off is that these models are imperfect: detection failures, segmentation errors, and flow estimation inaccuracies all introduce noise into the metric computation. The paper does not report the accuracy of the extraction pipeline on ground-truth annotated data, so the magnitude of this noise is unknown. This is a significant methodological concern: if the extraction pipeline makes systematic errors (e.g., consistently failing to detect small or fast-moving objects), the resulting metrics will conflate model performance with extraction quality.
Generated Video Quality Metrics
The Generated Video Quality dimension evaluates the basic perceptual quality of the generated video — separate from whether interactions are causally correct. This dimension largely adopts established metrics from prior benchmarks, reflecting the paper's position that visual quality is a necessary but not sufficient condition for world modeling.
Imaging Quality (from VBench): Evaluates per-frame visual fidelity — sharpness, color accuracy, absence of compression artifacts. This is computed frame-by-frame and averaged.
Temporal Flickering (from VBench): Measures the frame-to-frame consistency of static regions. When a region of the scene should remain static (e.g., a background wall), pixel values should not oscillate between frames due to generation artifacts. High flickering indicates that the model fails to maintain temporal coherence even in non-moving regions.
Motion Smoothness (from VBench): Evaluates whether the motion in the video is physically plausible in its continuity — objects should not jump discontinuously between frames, and motion trajectories should be smooth. This is conceptually related to optical flow consistency: adjacent flow vectors should be similar in direction and magnitude.
Dynamic Degree (from VBench): Measures the overall amount of motion in the video. A model that generates entirely static videos would score perfectly on flickering and smoothness but zero on dynamic degree. This metric distinguishes between models that are genuinely generating dynamic content versus those that are simply producing static outputs. The paper notes (Section 5.3) that Dynamic Degree exhibits "substantial variance across models," making it "a core differentiator in generation capabilities."
Content Alignment (from WorldScore): Evaluates how well the generated video content aligns with the textual prompt description — a semantic relevance measure rather than a visual quality measure. This is likely computed using CLIP-style embedding similarity between the prompt text and sampled video frames.
Why adopt rather than innovate on these metrics? The paper's argument is that visual quality is not the bottleneck — Table 1 shows most models exceeding 95% on Temporal Flickering and Motion Smoothness — so investing effort in improving visual quality metrics would have diminishing returns. The innovation is directed instead at the interaction fidelity metrics that capture the currently unmeasured capability gap. The design choice to include these metrics at all reflects completeness: a world model that is interactively perfect but visually unwatchable would still fail in practice, so visual quality must be part of the evaluation, even if it is not the differentiating factor.
Camera-Object Controllability Metrics
This dimension evaluates spatial coherence: whether the model respects camera motion instructions and whether all objects specified in the prompt actually appear in the generated video. This complements Interaction Effect Fidelity by testing control over the static scene structure, separate from action-driven dynamics.
Camera Control (from WorldScore). This metric quantitatively evaluates how closely the generated video's camera trajectory matches the specified camera motion. Given the estimated relative camera motion between consecutive frames (from the extraction pipeline), the system computes discrepancies in two components:
- Rotational error: how much the camera's angular orientation deviates from the specified rotation (e.g., if the prompt says "pan left 30 degrees," does the estimated pan angle match 30 degrees?).
- Translational error: how much the camera's position deviates from the specified translation (e.g., if the prompt says "dolly forward 2 meters," does the estimated forward displacement match?).
These error measurements are normalized to produce a single score where higher values indicate better camera control. The normalization procedure converts absolute errors (in degrees or meters) into a [0, 1] or percentage scale, likely using a threshold-based mapping where errors below some tolerance receive full credit and errors above a maximum receive zero, with linear interpolation in between — though the paper does not specify the exact normalization function.
The key property of this metric is that it is independent of what the camera observes — it measures only whether the camera moved as instructed, not whether the scene being viewed is coherent. A model could perfectly follow a camera trajectory while generating nonsense content, and Camera Control would give it a perfect score. This separation is intentional: it isolates the camera-following capability from the scene-generation capability, enabling diagnosis of which specific sub-capability is failing.
Object Control (novel formulation in this paper). Unlike WorldScore's approach — which uses rule-based text matching after detection — this paper reformulates object control as a visual question answering (VQA) problem. The motivation is that rule-based matching is fragile: synonyms (e.g., "automobile" vs. "car"), compositional descriptions (e.g., "green panda-shaped pen holder"), and detection failures can all cause false negatives that penalize the model unfairly.
The VQA formulation works as follows. Given the prompt's object list $O = \{o_i\}_{i=1}^K$ (all entity objects that should appear in the video), the system uniformly samples a small set of frames from the generated video (temporal subsampling to avoid processing every frame). For each object $o_i$, a multimodal large language model (VLM) is queried independently with a constrained binary question: "Is object $o_i$ present in this video?" The VLM returns a binary prediction $\hat{y}_i \in \{0, 1\}$. The final object control score is:
where $K$ is the number of prompt-specified objects, and $\hat{y}_i$ is the VLM's binary prediction for object $o_i$.
What it computes: the fraction of prompt-specified objects that the VLM confirms are visually grounded in the generated content. For a perfectly faithful model, every object mentioned in the prompt should be detectable in the video, yielding a score of 1.0. If the model omits objects (e.g., generates a room scene but forgets to include the bookshelf), those objects receive $\hat{y}_i = 0$, reducing the score proportionally.
Why this form: the VQA formulation eliminates the synonym and detection robustness problems of rule-based matching. A VLM can recognize "automobile" as referring to the same concept as "car," even if the detection model's label vocabulary uses only "car." It can also handle compositional descriptions — "green panda-shaped pen holder" — that would be difficult to encode in rule-based matching logic. The trade-off is computational cost: querying a VLM for each object in each video scales linearly with the number of objects, though temporal subsampling mitigates this by reducing the number of frames processed. The binary formulation (present/absent) is deliberately simple — it does not attempt to evaluate whether the object is at the correct location or in the correct state, only whether it exists at all. This is a useful but coarse check: a model that puts the bookshelf in the wrong corner of the room would still pass Object Control, and would only be penalized by the interaction fidelity metrics if the misplacement violates the prompt's spatial constraints.
Transitions Detect (novel formulation). This metric detects whether the generated video contains scene transitions — abrupt cuts between visually dissimilar content that would indicate a failure of temporal continuity. Scene transitions are a common failure mode in video generation: a model might generate coherent content for a few seconds, then abruptly switch to a completely different scene because the generation process lost coherence.
The detection uses PySceneDetect's ContentDetector, which operates in HSV (Hue-Saturation-Value) color space. For each pair of consecutive frames, it computes a frame-to-frame visual dissimilarity score — essentially, how different the color histograms of the two frames are in HSV space. When the dissimilarity exceeds a threshold $\tau$, a scene boundary is flagged, subject to a minimum scene length constraint $L$ (in frames). The minimum length constraint suppresses spurious detections: a single anomalous frame that differs from its neighbors but is followed by a return to the previous content is likely a generation artifact, not a true scene transition.
Given an input video of length $T$ frames, the detector produces a scene list $\{[t_i^{\text{start}}, t_i^{\text{end}}]\}_{i=1}^N$, where $N$ is the number of detected scenes. If $N > 1$, the video contains at least one scene transition. The binary score is:
where $s_{\text{trans}} = 1$ means no scene transitions detected (good), and $s_{\text{trans}} = 0$ means at least one transition was detected (bad).
What it computes: a binary indicator of temporal continuity — whether the video maintains a single coherent scene throughout its duration. A video without cuts receives full credit; any detected transition yields zero.
Why this form: the binary formulation is deliberately harsh — a single transition invalidates the entire video for the purposes of this metric. This reflects the evaluation priority: for world models that are supposed to simulate continuous state evolution, any scene break represents a fundamental failure of temporal coherence, regardless of how long the segments before and after the break are. A continuous metric (e.g., the fraction of frames in the longest coherent segment) might obscure the severity of the failure by averaging over long pre-transition segments. The choice of HSV-space dissimilarity rather than learned features (e.g., CLIP embeddings) is pragmatic: HSV histograms are fast to compute and work well for detecting the kind of gross content shifts that characterize generation failures, without requiring GPU inference.
Interaction Effect Fidelity Metrics
This is the core technical contribution of Omni-Metric — four metrics that directly measure whether interaction actions produce causally correct state transitions in the generated video. These metrics are novel to this paper and operationalize the concept of "interaction effect fidelity" that existing benchmarks lack.
InterStab-L: Long-Horizon Spatial Consistency at Temporal Revisit Pairs
InterStab-L evaluates whether the model maintains consistent visual content across temporally separated revisit pairs — pairs of frames where the camera should observe the same spatial location from the same viewpoint, typically because the camera has completed a loop or returned to a previously visited position. This metric is specific to the "return-to-origin" subset of prompts annotated with revisit frame pairs $\mathcal{R} = \{(t_a, t_b)\}$.
The intuition is that if a world model has a coherent internal representation of 3D space, returning the camera to a previously observed location should produce a visually similar frame — the scene geometry, object positions, lighting, and textures should be consistent. If the model fails to maintain this consistency, the revisit frame will look significantly different from the original, indicating that the model's spatial representation has drifted or collapsed.
Composite similarity metric. For any frame pair $(i, j)$ corresponding to a revisit pair, InterStab-L defines a composite similarity that integrates both low-level structural fidelity and high-level semantic consistency:
where $I_i$ and $I_j$ are the two frames (as images), $\operatorname{SSIM}_{\text{gray}}$ is the grayscale Structural Similarity Index computed on the luminance channel, and $\phi(\cdot)$ is a pretrained vision encoder (the visual tower of CLIP) that maps frames to semantic feature vectors $f$, with $\cos(\cdot, \cdot)$ being cosine similarity between those vectors.
What it computes: a single similarity score in [0, 1] that is the arithmetic mean of two complementary similarity measures. The first term, $\operatorname{SSIM}_{\text{gray}}$, measures low-level structural fidelity — whether edges, textures, and spatial layouts match between the two frames, evaluated on grayscale intensity to be robust to color shifts. SSIM is computed locally (over sliding windows) and captures luminance, contrast, and structure comparisons. The second term, $\cos(\phi(I_i), \phi(I_j))$, measures high-level semantic consistency — whether the two frames encode to similar feature vectors in CLIP's embedding space, which is trained to be invariant to lighting changes, minor viewpoint shifts, and other low-level variations while preserving semantic content. The arithmetic mean balances these: a high score requires both structural and semantic agreement, preventing either alone from dominating.
Why this composite form: using only SSIM would be overly sensitive to lighting changes that are semantically irrelevant — the same scene at different times of day should be considered consistent for world modeling purposes, even if pixel values differ. Using only CLIP cosine similarity would be insensitive to geometric errors that are semantically subtle but spatially significant — a chair shifted 30 cm might still be recognized as "a chair in a room" by CLIP but represents a spatial consistency failure. The arithmetic mean requires both to agree: the scene must look structurally similar (SSIM) and semantically consistent (CLIP) at the revisit point. Alternative formulations — using the minimum, product, or weighted combination — would change the trade-off between these two error types, but the mean provides equal weight to both.
Dynamics gating mechanism. A critical issue with spatial consistency metrics is the static scene degeneracy: a model that generates a completely static video (no motion whatsoever) would achieve perfect InterStab-L scores because every frame pair is identical — but this is not due to good spatial modeling, it is due to no modeling at all. To prevent this, InterStab-L incorporates a dynamics gating mechanism.
The system evaluates similarity across four canonical anchor intervals spanning the video duration — these are frame pairs at regular intervals (e.g., frames 1 and T/4, T/4 and T/2, T/2 and 3T/4, 3T/4 and T) that are NOT revisit pairs. If the average similarity across these anchors exceeds a static threshold $\tau_{\text{static}}$, it indicates the video is essentially static throughout — content is nearly identical at widely separated times despite the prompt specifying motion. In this case, the metric is penalized to zero to enforce content dynamics. The gating indicator is:
The final InterStab-L score is the mean similarity over the revisit set, gated by the dynamic check:
where $|\mathcal{R}|$ is the number of revisit pairs, $i(t_a)$ and $i(t_b)$ are the frame indices corresponding to times $t_a$ and $t_b$, $s(\cdot, \cdot)$ is the composite similarity, and $\mathbb{I}_{\text{dynamic}}$ is the binary gating indicator.
What it computes: the average composite (SSIM + CLIP) similarity across all specified revisit pairs, multiplied by a binary flag that is 1 only if the video demonstrates sufficient dynamics (not static). A high score means the model generates dynamic content (avoiding the static degeneracy) AND maintains spatial consistency at revisit points.
Why the gating mechanism is necessary: without it, a model could achieve InterStab-L = 1.0 by generating a single static frame repeated T times — visually perfect but completely uninformative about world modeling capability. The anchor check ensures that only genuinely dynamic videos are scored; static videos receive 0 regardless of their self-consistency. The choice of four anchors versus more or fewer is a practical engineering decision: four points provide reasonable coverage of the video duration without excessive computation. The threshold $\tau_{\text{static}}$ must be set high enough that only truly static videos are caught — if set too low, moderately dynamic videos with some repeated content would be incorrectly penalized. The paper does not report the specific threshold value, which limits reproducibility.
InterStab-N: Non-Target Region Motion Stability
InterStab-N evaluates whether regions of the video that should remain static — because they contain entities not involved in the interaction — actually exhibit motion. This complements InterCov (which checks that affected entities move correctly) by checking that unaffected entities do NOT move.
The metric operates on the non-target spatial region $\mathcal{N}$, defined as all pixels that are NOT occupied by any affected entity's segmentation mask. Given the entity masks $\{\text{traj}_k\}$ from the extraction pipeline, the non-target region for each frame is the complement of the union of all target entity masks.
Motion energy computation. For each frame $t$, the system averages the optical flow magnitude over all pixels in the non-target region:
where $\|\text{Flow}_t(x)\|$ is the optical flow magnitude (the length of the displacement vector) at pixel location $x$ in frame $t$, and $|\mathcal{N}|$ is the number of pixels in the non-target region. This produces a per-frame motion energy — the average amount of apparent motion in parts of the scene that should be static.
The total motion energy over the video is the temporal average:
where $T$ is the total number of frames. This is the mean per-pixel flow magnitude in non-target regions, averaged over time.
Score mapping. The raw motion energy is mapped to a bounded [0, 1] stability score using an exponential decay function:
where $H$ and $W$ are the frame height and width, $\min(H, W)$ is the smaller dimension (used for resolution normalization), and $\beta$ is a scaling factor.
What it computes: a score in [0, 1] that exponentially decays with the normalized non-target motion energy. When non-target regions are perfectly static ($E_{\text{non}} = 0$), the exponential evaluates to 1.0 (perfect stability). As non-target motion increases, the score drops toward 0, with the decay rate controlled by $\beta \times \min(H, W)$ — larger $\beta$ makes the score more tolerant of small motions; larger $\min(H, W)$ (higher resolution) makes the score more sensitive (because the same pixel displacement represents a smaller fraction of the frame).
Why exponential decay: alternative mappings (linear, sigmoid, threshold-based) have different sensitivity profiles. A linear mapping $1 - E_{\text{non}} / (\beta \times \min(H, W))$ would be equally sensitive to small and large motions — a 1-pixel flow and a 10-pixel flow in non-target regions would be penalized proportionally. Exponential decay is more tolerant of small motions (the slope near zero is gentle) and more punitive for large motions (the slope steepens), reflecting the evaluation priority: small amounts of non-target motion might be due to flow estimation noise or minor generation artifacts and should not heavily penalize the model, but substantial non-target motion indicates a genuine failure to maintain stability and should be strongly penalized. The resolution normalization $\min(H, W)$ ensures the metric is resolution-invariant — a 5-pixel displacement is more significant in a 256px video than in a 1024px video — by scaling the denominator with the frame size.
InterCov: Object-Level Causal Faithfulness via VLM Verification
InterCov evaluates whether each entity in the generated video responds to interaction actions in a semantically consistent manner — affected entities should exhibit dynamics aligned with the specified action, while unaffected entities should remain static or follow their independent dynamics. This metric leverages Vision-Language Models (VLMs) to perform high-level reasoning about whether entity behavior matches the causal structure of the prompt.
Formally, let $\mathcal{O} = \{o_1, \ldots, o_N\}$ be the set of target entities subject to causal constraints (both affected and unaffected entities, as annotated in the prompt metadata). The system employs a VLM-based semantic verifier that examines the video sequence and produces a binary validity signal $v_o \in \{0, 1\}$ for each entity $o \in \mathcal{O}$, where $v_o = 1$ indicates that the entity's behavior aligns with the prescribed interaction logic.
VLM query protocol. Although the paper does not specify the exact VLM prompt template, the verification process for each entity likely involves: (1) presenting the VLM with the full textual prompt describing the expected interaction, (2) showing sampled frames from the generated video with the target entity highlighted or identified, (3) asking a structured question about whether the entity's behavior is consistent with the prompt — for affected entities, whether the entity exhibits the expected type of motion or state change; for unaffected entities, whether the entity remains appropriately static or follows its independent dynamics. The VLM's response is mapped to $\{0, 1\}$.
Metric computation. InterCov is the semantic recall of consistent interactions:
where $\mathbb{I}(\cdot)$ is the indicator function (1 if the condition is true, 0 otherwise), and $|\mathcal{O}|$ is the number of entities being evaluated.
What it computes: the fraction of entities whose behavior is judged by the VLM to be causally consistent with the prompt. A perfect score of 1.0 means every entity — both those that should move and those that should not — behaves as specified. A low score indicates that the model either fails to produce the expected dynamics for affected entities or introduces spurious changes in unaffected entities.
Why use a VLM rather than rule-based verification: rule-based approaches would require programming explicit checks for each interaction type — "if action is 'throw,' check that ball velocity is upward and nonzero" — which scales poorly with the diversity of interactions in Omni-WorldSuite (covering Newtonian mechanics, fluid dynamics, thermodynamics, optics, material deformation, etc.). A VLM can perform open-ended reasoning about whether an entity's observed behavior matches the semantic intent of the prompt, without requiring per-interaction-type programming. The trade-off is reliability: VLMs can hallucinate, misinterpret, or exhibit biases, potentially introducing noise into the metric. The paper does not report VLM accuracy on a ground-truth labeled subset of videos, so the magnitude of VLM error in InterCov computation is unknown.
Why this complements InterStab-N and InterStab-L: InterStab-N measures low-level motion in non-target regions (via optical flow) — it detects motion but cannot determine whether that motion is semantically correct. InterCov measures high-level semantic correctness (via VLM reasoning) — it determines whether entity behavior is appropriate, even if the low-level motion metrics would pass. For example, a video where a ball oscillates in place rather than being thrown upward would show non-zero optical flow in the ball region (failing InterStab-N's check on affected regions if treated as a target stability metric), but a VLM could determine that oscillating in place does not match "throws the ball as high and as far as he can" (failing InterCov). Conversely, a video where non-target regions show small amounts of flow-consistent motion due to camera shake would show elevated $E_{\text{non}}$ (failing InterStab-N) but a VLM might judge the entities as behaving correctly (passing InterCov). The metrics are complementary: InterStab-N catches low-level motion artifacts, InterCov catches high-level semantic failures.
InterOrder: Temporal Event Ordering Consistency
InterOrder evaluates whether the generated video preserves the correct chronological ordering of key events — that the causal chain specified in the prompt unfolds in the correct sequence. This metric directly tests a model's ability to represent temporal causality: if event A causes event B, A must occur before B in the video. Reversals indicate that the model's generative process does not respect causal structure.
The ground truth is the ordered event list $E = \{e_i\}_{i=1}^K$ extracted from the prompt during annotation. For any pair of distinct events $(e_m, e_n)$ with $m < n$ (meaning $e_m$ should occur before $e_n$), the system employs a pretrained VLM as an automated verifier to assess both the occurrence of the events and their relative temporal precedence.
Structured query protocol. The VLM is queried with a structured prompt that likely: (1) describes the two events $e_m$ and $e_n$, (2) presents sampled frames from the generated video, (3) asks whether both events occur in the video, and (4) if so, asks which occurs first. The response is mapped to a binary consistency judgment for the pair: the pair is temporally consistent if the VLM confirms that (a) both events are present and (b) $e_m$ precedes $e_n$ in the generated sequence. Let $K_s$ be the number of event pairs that are judged temporally consistent.
Metric computation. InterOrder is the ratio of consistent event pairs to total possible ordered pairs:
where $K$ is the number of key events, and $\binom{K}{2} = K(K-1)/2$ is the total number of distinct ordered pairs $(e_m, e_n)$ with $m < n$. InterOrder $\in [0, 1]$.
What it computes: the fraction of event pairs that appear in the correct temporal order in the generated video. A score of 1.0 means every event pair respects the ground-truth ordering — the model's generated sequence is a valid linear extension of the partial order defined by the event list. A score of 0 means no pair is correctly ordered (or no pair could be verified). Random temporal ordering would produce an expected score of 0.5 for independently ordered events, but events are causally constrained (A causes B, so B before A is a genuine failure, not random), making scores below 1.0 meaningful indicators of causal reasoning failure.
Why pairwise rather than sequence-level evaluation: evaluating the entire sequence of $K$ events as a single ordered list would be extremely strict — a single transposition (two adjacent events swapped) would invalidate the entire sequence, yielding a binary pass/fail that loses granularity. The pairwise formulation is more fine-grained: a model that gets 8 of 10 event pairs correct receives an InterOrder of 0.8 rather than 0.0, which better reflects partial capability. The trade-off is that pairwise consistency does not guarantee global consistency — it is possible for all pairs to be pairwise consistent yet for the global ordering to be inconsistent (this is the classic Condorcet paradox, though unlikely with clear causal constraints). In practice, the pairwise approximation provides useful granularity while capturing the dominant failure modes (event reversals).
Why use a VLM for temporal verification: determining whether an event occurs in a video and when it occurs relative to another event requires semantic understanding that low-level temporal features (frame indices, optical flow) cannot provide. A VLM can recognize "the arm grasps the marker" as a distinct event and compare its temporal position to "the arm places the marker in the pen holder." The structured query protocol is necessary to constrain the VLM's response to a binary consistency judgment rather than open-ended narration, enabling quantitative aggregation.
AgenticScore: Semantic-Weighted Dimension Aggregation
The fourteen individual metrics produce scores across three evaluation dimensions — Interaction Effect Fidelity $A_I$, Generated Video Quality $A_G$, and Camera-Object Controllability $A_C$ — but these dimensions are heterogeneous and their relative importance varies by prompt. A prompt emphasizing physical causality should weight Interaction Effect Fidelity more heavily; a prompt primarily testing camera trajectory following should weight Camera-Object Controllability more heavily. Simple averaging would ignore this heterogeneity.
AgenticScore addresses this using an agent-based aggregation framework where an MLLM (Multimodal Large Language Model) acts as an aggregation agent that dynamically assigns weights to each dimension based on the semantic content of the evaluation prompt.
Per-dimension score computation. Each dimension's score is first computed as the average of its constituent sub-metrics:
For Interaction Effect Fidelity:
For Generated Video Quality:
For Camera-Object Controllability:
Weight assignment by the aggregation agent. The aggregation agent takes the evaluation prompt text (and possibly the first-frame image) as input, analyzes the relative importance of the three evaluation dimensions, and maps the resulting ranking to predefined weight coefficients $w_1, w_2, w_3$. The paper does not specify the exact mapping from ranking to weights — for example, whether the most important dimension receives double or triple weight relative to the least important — but the weights presumably sum to a constant (likely 1 or 3, for averaging), with the distribution reflecting the MLLM's assessment of which dimensions matter most for that prompt.
Final score computation. AgenticScore is the weighted sum:
where $A_I, A_G, A_C$ are the dimension scores and $w_1, w_2, w_3$ are the MLLM-assigned weights.
What it computes: a single scalar score (in percentage or [0, 1] depending on how dimension scores are scaled) that represents the overall quality of the generated video, with the contribution of each dimension weighted according to the prompt's semantic emphasis. The score is interpretable as "how well the model performed on this prompt, where performance is defined by what the prompt primarily cares about."
Why agent-based weighting rather than fixed weights: different prompt types require fundamentally different evaluation emphases. A prompt about camera trajectory following benefits from heavily weighting Camera-Object Controllability while downweighting Interaction Effect Fidelity (since there may be no interactions). A prompt about a baseball throw benefits from heavily weighting Interaction Effect Fidelity. Fixed weights would either overpenalize models on dimensions irrelevant to the prompt (if all dimensions are equally weighted) or require manual per-prompt weight specification (which doesn't scale to 1,068 prompts). The MLLM-based approach automates this semantic analysis: the agent reads the prompt, understands what it is primarily testing, and adjusts weights accordingly.
Why this is called "agent-based": the framework treats each evaluation dimension as an independent "agent" that specializes in one type of assessment, and the aggregation agent as a meta-agent that coordinates them. This is a conceptual framing rather than a literal multi-agent system — the dimension agents are deterministic metric computations, not learned policies — but it emphasizes the modular, composable design where new evaluation dimensions could be added as new agents without modifying existing ones.
Potential concerns. The MLLM's weight assignments are a source of subjectivity and potential error — if the aggregation agent misinterprets the prompt's emphasis, it could assign inappropriate weights and produce misleading AgenticScores. The paper does not evaluate the aggregation agent's reliability (e.g., inter-prompt consistency, robustness to prompt phrasing variations). Additionally, the hard discretization into three dimensions with equal sub-metric weighting within each dimension may not capture nuanced interactions — for example, a prompt might require both interaction fidelity and camera control to be weighted highly, but if the aggregation agent ranks them 1st and 2nd, the second-ranked dimension might receive substantially lower weight than the first despite both being critical.
4. Key Insights and Innovations
Innovation 1: Interaction-Centric Evaluation as a Distinct Paradigm from Visual Quality Assessment
The most fundamental conceptual move in this paper is decoupling world model evaluation from video generation evaluation by defining interactive response — not visual fidelity or text–video alignment — as the target capability to measure. This is a paradigm-level reframing, not an incremental extension of existing benchmarks.
Before this work, the evaluation landscape for video-based world models was inherited directly from the video generation community. Benchmarks like VBench and VBench++ evaluate dimensions such as temporal flickering, motion smoothness, dynamic degree, and text–video alignment — all of which measure whether a generated video looks like a real video, without regard for why content changes frame to frame. WorldScore advanced this by adding geometric consistency under camera motion, but the interaction scope remained restricted to observer perspective changes — the "world" was static; only the camera moved. WorldModelBench added action-conditioned generation but lacked the systematic interaction-level hierarchy and causal verification that separates genuine world modeling from pattern-matching animation.
The paper's reframing is that a world model's defining capability is interactive response: the ability to faithfully represent how interaction actions drive state transitions across space and time. This is not a refinement of visual quality evaluation — it is a fundamentally different evaluation target. A model could score 99% on every VBench metric (as Table 1 shows many do) while entirely failing to represent that a baseball player's throwing motion should produce a parabolic ball trajectory, or that a robotic arm grasping a bottle should change the bottle's position.
What makes this reframing intellectually distinctive is that it identifies a category error in prior evaluation: existing benchmarks measure correlates of world modeling (temporal coherence, visual plausibility) rather than the constitutive property (causal response to interaction). The paper argues, implicitly, that the field has been optimizing a proxy metric — visual quality — under the assumption that better visual generation implies better world modeling. The evidence in Table 1 refutes this assumption: models that are visually near-perfect (temporal flickering and motion smoothness exceeding 95% for most models) show substantial variance in interaction effect fidelity metrics, and models within the camera-conditioned category exhibit sharp trade-offs (WonderWorld: 84.96% InterStab-L vs. 24.89% InterStab-N) that visual-quality-only evaluation would never detect.
The interaction-level hierarchy — Level 1 (entity-self), Level 2 (local object–object), Level 3 (global cascading) — operationalizes this reframing by defining a structured taxonomy of what "interaction" means for world models. Prior work either ignored interaction entirely (VBench), treated it as a single monolithic condition (WorldModelBench), or restricted it to camera motion (WorldScore). The hierarchy makes interaction a graded capability rather than a binary present/absent property, enabling fine-grained diagnosis of where models fail: do they handle self-contained object dynamics but fail at object–object energy transfer? Do they collapse under multi-object cascading? This diagnostic granularity is what converts the conceptual reframing into a practical measurement instrument.
The significance of this innovation is not in any single metric but in establishing interaction effect fidelity as a first-class evaluation dimension for world models — analogous to how Chinchilla scaling laws established compute-optimal pretraining as the target to optimize, rather than raw model size. By providing the measurement infrastructure, the paper redirects the field's optimization pressure from visual fidelity (already saturated above 95%) toward interactive causality (where the capability gap, as shown in Table 1 across all 18 evaluated models, is the binding constraint).
Innovation 2: Disaggregating Interactive Fidelity into Complementarily Measurable Sub-Capabilities
If Innovation 1 establishes interaction as the evaluation target, Innovation 2 is the methodological insight that interactive fidelity decomposes into separable, independently measurable sub-capabilities — and that measuring them separately reveals failure patterns invisible to any single metric. This is a measurement-theoretic contribution: the paper shows that interaction effect fidelity is not a monolithic property that can be captured by a single score, but rather a constellation of distinct requirements that current models satisfy unevenly.
The four Interaction Effect Fidelity metrics — InterStab-L, InterStab-N, InterCov, and InterOrder — are each individually straightforward in what they measure. InterStab-L checks spatial consistency across temporal revisits. InterStab-N checks that non-target regions stay static. InterCov checks that target entities respond to actions in semantically appropriate ways. InterOrder checks that multi-step events occur in the correct sequence. None of these, taken alone, is conceptually revolutionary — prior work has evaluated spatial consistency (WorldScore), temporal ordering (narrative evaluation benchmarks), and entity-level tracking (video object segmentation benchmarks).
What IS intellectually distinctive is the disaggregation strategy itself: the paper argues that interactive fidelity is the joint satisfaction of these four constraints, and that current models fail different constraints in different ways, making any single metric misleading. The evidence for this is in the sharp within-model trade-offs visible in Table 1. Cosmos — the second-highest AgenticScore overall at 75.42% — scores 79.55% on InterStab-L and 79.63% on InterStab-N, indicating strong spatial consistency and stability, but only 51.81% on InterOrder, indicating poor temporal event sequencing. WonderWorld shows the opposite pattern: 84.96% on InterStab-L (the highest across all models) but 24.89% on InterStab-N (one of the lowest), indicating that its spatial memory at revisit points is excellent but its ability to suppress motion in non-target regions is severely compromised. ViewCrafter scores 81.15% on InterStab-L but 4.22% on InterStab-N — essentially zero stability in regions that should be static.
These trade-offs would be invisible if interactive fidelity were reported as a single averaged score. A model with Cosmos's profile would appear "decent at interaction" at around 66% Interactive Effect Fidelity average, obscuring the fact that it fundamentally cannot order events correctly. A model with ViewCrafter's profile would appear weak at around 42% average, obscuring that its revisit consistency is actually strong and the failure is specifically in non-target stability. The disaggregation converts "this model is bad at interaction" into "this model can remember spatial layouts across time but cannot prevent spurious motion in background regions" — a diagnostic claim that points to specific architectural or training deficiencies.
This contribution can be understood as applying a systems-level decomposition to evaluation: just as a complex engineering system is tested by evaluating each subsystem against its specific requirements (the engine, the braking, the navigation, each with separate tests), world models should be evaluated by testing spatial memory, non-target stability, entity-level response, and event sequencing as separate but jointly necessary capabilities. The prior approach of aggregating everything into a single visual quality score (FVD, IS) or even a multi-dimensional but interaction-agnostic score (VBench) conflates distinct failure modes into a single number that is diagnostically useless.
The practical significance is that this decomposition enables targeted improvement: a research team working on world models can look at their model's profile across the four metrics and immediately identify which sub-capability to focus on — improving temporal ordering (low InterOrder) requires better modeling of causal event chains, possibly through explicit temporal supervision; improving non-target stability (low InterStab-N) requires better object-level attention or segmentation-guided generation to suppress motion in irrelevant regions. Without this decomposition, the team would only know "our interaction fidelity is 55%" — which provides no guidance.
Innovation 3: Agent-Based Semantic Weighting as a Solution to Heterogeneous Prompt Requirements
The third innovation addresses a subtle but pervasive problem in benchmark design: how to aggregate heterogeneous metrics across prompts with fundamentally different evaluation priorities in a way that respects what each prompt is actually testing. This is a meta-evaluation contribution — it is about how to design the aggregation mechanism, not about what specific capabilities to measure.
The problem arises from Omni-WorldBench's design goal of covering diverse interaction types, physical principles, and scene domains. A prompt about camera trajectory following ("camera rotates left to scan a city skyline") primarily tests Camera-Object Controllability; penalizing the model heavily on Interaction Effect Fidelity (which may be undefined or trivial for this prompt) would be inappropriate. Conversely, a prompt about a baseball player throwing a ball primarily tests Interaction Effect Fidelity; weighting Camera-Object Controllability equally would dilute the signal about the core capability being evaluated. Prior benchmarks address this by either (a) using fixed weights across all prompts, implicitly assuming all dimensions are equally relevant everywhere, or (b) manually specifying per-prompt weights, which doesn't scale.
The paper's solution — using an MLLM as an aggregation agent that reads the prompt text, assesses which evaluation dimensions are most relevant, and assigns weights accordingly — is intellectually distinctive because it treats weight assignment as a semantic understanding task rather than a fixed engineering choice. The MLLM "understands" that a prompt describing camera motion with no object interactions should weight controllability above interaction fidelity, while a prompt describing a multi-step manipulation should weight interaction fidelity above everything else. This leverages the semantic reasoning capabilities of large models to solve what would otherwise be a brittle, manual specification problem.
What makes this more than just "using an LLM as a router" is that it connects the evaluation of world models to the semantic content of the prompts themselves in a principled way. The prompts in Omni-WorldSuite are not homogeneous test items — they span physics, commonsense, causality, spatial constraints, camera motion, and loop-closure consistency across different interaction levels. The aggregation mechanism must reflect this heterogeneity, because averaging all metrics equally across all prompts would produce a score that is not interpretable as any specific capability. The AgenticScore is interpretable: for each prompt, it represents "how well the model performed on the capabilities that this prompt was designed to test."
The evidence for the value of this approach is indirect but compelling: the paper's qualitative analysis (Section 5.4) shows that models with similar overall AgenticScores can have very different failure profiles — ViewCrafter (65.88%) and Lingbot (67.16%) are within ~1.3 percentage points overall, but ViewCrafter's Interaction Effect Fidelity average (42.42%) is substantially lower than Lingbot's (55.50%), compensated by ViewCrafter's higher Generated Video Quality (79.36% vs. 72.38%). Under fixed-weight averaging, this compensation would be uniform across all prompts, obscuring that ViewCrafter performs better on camera-trajectory prompts (where Video Quality and Controllability dominate) while Lingbot performs better on interaction-heavy prompts (where Interaction Effect Fidelity dominates). The semantic weighting makes this compensation context-sensitive — the aggregation agent recognizes when a prompt is about interaction vs. camera motion and adjusts weights to report the capability that matters.
The innovation has broader implications for benchmark design beyond world models. Many AI evaluation domains face the same challenge: the test cases are heterogeneous, and different capabilities matter to different degrees depending on the test case. AgenticScore demonstrates a general pattern — use a semantic understanding module to dynamically weight evaluation dimensions based on test case content — that could be applied to benchmarks for embodied AI, code generation, or any domain where test cases have varying capability profiles. This moves benchmark aggregation from a fixed, one-size-fits-all formula to an adaptive, context-sensitive one.
Innovation 4: Empirical Discovery that Visual Quality and Interactive Fidelity Are Decoupled in Current Models
This is not a methodological innovation but an empirical finding with theoretical significance: the paper provides systematic evidence that current world models have achieved high visual quality while remaining fundamentally limited in interactive causal reasoning, establishing that these two capabilities are not tightly coupled in practice. This finding challenges the implicit assumption that better video generation models will naturally become better world models.
The evidence is in Table 1. Across all 18 evaluated models, Temporal Flickering scores range from 91.01% to 99.48% (18 of 18 above 92%, 14 of 18 above 96%) and Motion Smoothness scores range from 95.40% to 99.68% (all 18 above 95%). Visual quality, as measured by these conventional metrics, is essentially saturated — the range of variation is small, and most models are within a few percentage points of each other. Yet Interaction Effect Fidelity metrics show extreme variance: InterStab-L ranges from 47.38% to 84.96%, InterStab-N from 4.22% to 95.76%, InterCov from 40.29% to 64.52%, InterOrder from 35.28% to 54.19%. The standard deviations across models are dramatically larger for the interaction metrics than for the visual quality metrics.
This pattern is not accidental — it reflects a structural decoupling. Video generation models are trained on objectives that optimize per-frame and short-range temporal coherence (diffusion losses, adversarial losses, flow-based losses) without explicit supervision for long-range causal consistency or action-conditioned state transitions. The training signal pushes models toward producing frames that individually look realistic and transitions that are locally smooth, which the visual quality metrics capture. But it provides no signal about whether a throw produces a parabolic arc, whether heat conducts along a metal rod, or whether a camera returning to a starting position sees the same scene. These capabilities require understanding of physics, causality, and 3D geometry that current training objectives do not explicitly reward.
The finding has significant implications for the field's trajectory. If visual quality and interactive fidelity were coupled — if improving visual generation automatically improved causal reasoning — then the path forward would be straightforward: continue scaling video generation models, and world modeling capability would follow. The decoupling evidence suggests this is false. Additional investment in visual quality optimization will not close the interactive fidelity gap because the gap is not about insufficient visual quality — it is about a missing capability that current training paradigms do not target. The paper essentially provides an empirical case that world modeling requires something beyond video generation, and that the field needs architectural innovations, training objectives, or data strategies specifically designed for causal consistency.
The decoupling is further supported by the camera-conditioned model results, which show particularly sharp trade-offs. These models explicitly receive camera trajectory inputs, giving them an advantage on Camera-Object Controllability, but their Interaction Effect Fidelity often degrades relative to image-to-video models that lack explicit camera control. ViewCrafter (81.15% InterStab-L, 4.22% InterStab-N) and Gen3C (75.90% InterStab-L, 38.40% InterStab-N) both show strong long-horizon spatial memory but poor non-target stability — the additional conditioning on camera parameters appears to improve viewpoint consistency at the cost of increased spurious motion in non-target regions. This suggests that adding control dimensions can introduce new failure modes in interaction fidelity, further evidence that the capabilities are not simply additive.
The negative result with ReST^EM-trained revision models (Appendix K in the original paper, though not detailed in this benchmark paper) reinforces this theme from the training side: attempts to optimize for revision capability can degrade interaction fidelity, indicating that optimizing one objective does not necessarily transfer to or improve the other. The decoupling is bidirectional — improving visual dynamics doesn't help causal reasoning, and improving causal reasoning (via revision training) can hurt visual dynamics.
This finding positions Omni-WorldBench not just as a measurement tool but as a diagnostic instrument that reveals which capability gaps are architectural vs. incidental. The visual quality ceiling across models suggests that further progress on visual generation is a diminishing-returns problem; the wide variance in interaction fidelity suggests it is the primary frontier. The benchmark thus serves as an empirical argument for redirecting research investment — not by asserting that interaction fidelity matters more, but by providing the measurement infrastructure that makes the decoupling visible and quantifiable.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. All evaluations use Omni-WorldSuite, the custom prompt suite constructed as part of this benchmark (Section 3). It comprises 1,068 evaluation prompts spanning three interaction levels (Level 1: entity-self, Level 2: local object–object, Level 3: global cascading) across general daily-life scenes and task-oriented domains (autonomous driving, embodied AI, gaming). Text-to-Video and Image-to-Video models are evaluated on 410 diverse prompts; camera-conditioned models are evaluated on a dedicated subset of 120 prompts equipped with explicit camera trajectories. Each prompt includes a textual description, a first-frame image, and optionally a camera trajectory specification, along with ground-truth annotations for affected entities, key event sequences, and revisit frame pairs.
-
Base model(s). The paper evaluates 18 representative world models spanning three paradigms: Text-to-Video (T2V), Image-to-Video (IT2V), and camera-conditioned generation. No single "base model" is used — the benchmark is the instrument, and the models are the subjects. The model set (Section 5.1) includes: T2V models (Director3D, OpenSoraPlan, T2V-Turbo, HunyuanVideo); IT2V models (Matrix Game2.0, Wan2.1, Wan2.2, CogVideo, OpenSora, Cosmos, LargeVideoPlanner); and camera-conditioned models (HunyuanWorld, HunyuanGameCraft, ViewCrafter, Gen3C, Lingbot, FantasyWorld, WonderWorld). These span diffusion-based, autoregressive, and hybrid architectures. The selection criterion is representativeness across the dominant video generation paradigms used for world modeling.
-
Metrics. The primary evaluation framework is Omni-Metric (Section 4), which computes 14 individual metrics organized into three dimensions: (1) Interaction Effect Fidelity, averaging InterStab-L (long-horizon revisit consistency via SSIM + CLIP cosine similarity, gated by a dynamics check), InterStab-N (non-target region stability via exponential decay of normalized optical flow energy), InterCov (object-level causal faithfulness via VLM binary verification), and InterOrder (temporal event ordering via VLM pairwise consistency); (2) Generated Video Quality, adopting Imaging Quality, Temporal Flickering, Motion Smoothness, and Dynamic Degree from VBench, plus Content Alignment from WorldScore; (3) Camera-Object Controllability, comprising Camera Control (from WorldScore, measuring rotational and translational trajectory errors), Object Control (reformulated as VQA: fraction of prompt-specified objects detected by a VLM in sampled frames), and Transitions Detect (binary: 1 if no scene cuts detected by PySceneDetect, 0 otherwise). These three dimension scores are fused via AgenticScore (Section 4.5), where an MLLM aggregation agent reads each prompt and dynamically assigns weights
w1, w2, w3to AI, AG, and AC, producing a final percentage score. The overall AgenticScore is the primary summary metric. -
Baselines. Because this is a benchmark infrastructure paper rather than a method paper, there is no single proposed method to baseline against. Instead, the evaluation treats all 18 models as points on a comparative landscape. The implicit baseline is the performance floor established by the weakest models in each category and the performance ceiling defined by the strongest. The paper does not compare against "ground truth" videos (real footage of the same interactions) — the metrics evaluate generated videos against prompt specifications, not against reference videos. This is a deliberate choice: world model evaluation tests whether actions produce specified effects, not distributional similarity to a reference set.
-
Generation budget / compute accounting. Generation budget is not the axis of comparison in this paper — unlike scaling-law studies that sweep the number of samples or beams, Omni-WorldBench evaluates each model at a fixed generation setting determined by that model's standard inference configuration (Section 5.2). Each model generates one video per prompt using its default parameters: HunyuanVideo produces 91 frames at 1280×720 (16 FPS, 50 steps); Wan2.1 produces 81 frames at 1280×720 (16 FPS, 50 steps, CFG 5.0); Wan2.2 produces 81 frames with 40 steps and CFG 3.5; Cosmos outputs 77 frames (35 steps, CFG 7); ViewCrafter uses equidistant camera pose sampling for 25 frames at 576×1024 (8 FPS); Gen3C and Lingbot are truncated to the first 121 frames at 720×1280; FantasyWorld and WonderWorld subsample 132-frame trajectories to 81 frames. Compute is not equalized across models — the evaluation measures each model at its intended operating point. This is appropriate for a benchmarking study but means that performance differences may partly reflect differences in inference budget rather than purely architectural capability.
-
Cross-validation / statistical protocol. The paper does not report cross-validation, statistical significance testing, confidence intervals, or multiple-run error bars. Results in Table 1 are single-point estimates — one video per prompt per model, evaluated once through the metric pipeline. The 1,068 prompts provide coverage across interaction levels, physical principles, and scene types, but the evaluation is deterministic given the prompts and model weights. This is a limitation: without multiple runs (e.g., different random seeds for generation), we cannot assess the variance in model performance or the statistical reliability of the reported differences between models. For example, the gap between Wan2.2 (75.92%) and Cosmos (75.42%) is 0.50 percentage points — without variance estimates, it is unclear whether this difference is meaningful or within noise.
Main Quantitative Results
Overall Performance Across All Models
Table 1 presents the full quantitative evaluation. The headline numbers: Wan2.2 achieves the highest overall AgenticScore at 75.92%, followed by Cosmos (75.42%) and OpenSora (74.71%). Among T2V models, HunyuanVideo leads at 73.96%. Among camera-conditioned models, HunyuanWorld (74.36%) and WonderWorld (74.02%) are top performers.
The image-to-video paradigm "demonstrates the highest performance potential on the current benchmark" (Section 5.3), with IT2V models occupying 4 of the top 5 AgenticScore positions. This is consistent with the intuition that providing a first-frame image as conditioning signal gives the model stronger grounding in the initial world state, reducing the ambiguity that T2V models must resolve from text alone.
The range of AgenticScores across all 18 models spans from Matrix Game2.0 at 60.33% to Wan2.2 at 75.92% — a 15.59 percentage point spread — indicating that the benchmark provides meaningful discrimination rather than saturating at a ceiling.
Interaction Effect Fidelity: The Primary Performance Differentiator
Across the four Interaction Effect Fidelity sub-metrics in Table 1, the paper reveals substantial variance that contrasts sharply with the tight clustering on visual quality metrics:
-
InterStab-L (long-horizon revisit consistency) ranges from 47.38% (Matrix Game2.0) to 84.96% (WonderWorld), with a spread of 37.58 percentage points. The highest performers are WonderWorld (84.96%), T2V-Turbo (82.98%), and LargeVideoPlanner (82.15%). The lowest are Matrix Game2.0 (47.38%), HunyuanGameCraft (64.78%), and OpenSora (66.68%).
-
InterStab-N (non-target region stability) shows extreme variance: from 4.22% (ViewCrafter) to 95.76% (OpenSoraPlan), a 91.54 percentage point spread. The camera-conditioned models are the worst performers on this metric: ViewCrafter (4.22%), Matrix Game2.0 (19.96%), WonderWorld (24.89%), and Gen3C (38.40%) all score below 40%. This is the most discriminative single metric in the entire benchmark — it reveals that camera-conditioned models, despite their explicit trajectory inputs, struggle severely to suppress spurious motion in regions that should remain static.
-
InterCov (object-level causal faithfulness) ranges more narrowly from 40.29% (OpenSoraPlan) to 64.52% (Wan2.1), a spread of 24.23 points. The top performers cluster in the IT2V group: Wan2.1 (64.52%), OpenSora (62.54%), and LargeVideoPlanner (42.84%) — though LargeVideoPlanner's 42.84% is notably lower than the leaders. No model exceeds 65% on InterCov, indicating that even the best models are only approximately two-thirds faithful in producing semantically correct entity-level responses to actions.
-
InterOrder (temporal event ordering) ranges from 35.28% (Lingbot) to 54.19% (Wan2.1), a spread of 18.91 points. All models cluster in the 35–55% range — meaning no model correctly orders even 55% of event pairs on average. This is the metric closest to random performance (50% would be expected for independent random ordering), suggesting that temporal causal sequencing is a near-universal weakness across current architectures.
The Interaction Effect Fidelity dimension average (the mean of these four sub-metrics) spans from 42.42% (ViewCrafter) to 67.34% (Wan2.2). This 24.92-point spread is substantially larger than the spread on Generated Video Quality averages, confirming that interaction fidelity is where models most strongly differentiate.
Generated Video Quality: Saturated on Temporal Coherence, Variable on Dynamics
The five Generated Video Quality sub-metrics in Table 1 tell a clear story: conventional video quality metrics are largely saturated, with one exception.
-
Temporal Flickering ranges from 91.01% (ViewCrafter) to 99.48% (Director3D), with 14 of 18 models above 96%. The tight clustering means this metric provides minimal discrimination — nearly all models produce temporally stable frames without flickering artifacts.
-
Motion Smoothness ranges from 95.40% (ViewCrafter) to 99.68% (Director3D), with all 18 models above 95%. This metric is essentially saturated — every evaluated model generates videos where motion is perceptually smooth.
-
Imaging Quality shows more variance: 48.90% (Director3D) to 67.65% (Lingbot), a spread of 18.75 points. The camera-conditioned models tend to score higher here (Lingbot 67.65%, HunyuanGameCraft 67.09%, Cosmos 66.30%), while T2V models score lower (Director3D 48.90%, OpenSoraPlan 53.55%).
-
Content Alignment (adopted from WorldScore) ranges widely from 47.29% (HunyuanGameCraft) to 90.24% (T2V-Turbo), a spread of 42.95 points — making it the most variable visual quality sub-metric. The camera-conditioned models score poorly (HunyuanGameCraft 47.29%, ViewCrafter 49.03%, Lingbot 52.83%), while T2V models score higher (T2V-Turbo 90.24%, Director3D 89.87%). This pattern suggests that models with camera conditioning prioritize trajectory following over semantic alignment with the text prompt.
-
Dynamic Degree is the "core differentiator" (Section 5.3): ViewCrafter and WonderWorld achieve 100.00%, while OpenSoraPlan scores only 16.83% and CogVideo 29.02%. A spread of 83.17 percentage points is the largest on any single metric in the entire benchmark. Models with low Dynamic Degree essentially generate near-static videos — they score well on flickering and smoothness (because nothing moves to flicker) but fail to produce the dynamics required for world simulation. The perfect 100% scores from ViewCrafter and WonderWorld are notable: these camera-conditioned models maximize motion quantity, but as the Interaction Effect Fidelity results show (ViewCrafter: 4.22% InterStab-N, WonderWorld: 24.89% InterStab-N), this motion is largely uncontrolled — the right regions do not move while wrong regions do.
The Generated Video Quality dimension average spans from 70.09% (OpenSoraPlan) to 85.18% (WonderWorld, though this is inflated by the 100% Dynamic Degree), with most models clustering in the 75–85% range. The narrow spread relative to Interaction Effect Fidelity confirms the paper's argument that visual quality is not the binding constraint on world model capability.
Camera-Object Controllability: Explicit Camera Models Excel but with Nuance
The three Camera-Object Controllability sub-metrics reveal distinct capability patterns:
-
Camera Control (computed only for the 12 models supporting explicit camera input, as indicated by the dashes in Table 1 for non-camera models) ranges starkly from 27.96% (HunyuanGameCraft) to 96.12% (WonderWorld). WonderWorld's 96.12% is described as "an overwhelming advantage" (Section 5.3), far surpassing the next-best camera-conditioned model (HunyuanWorld at 55.40%). The wide range — 68.16 percentage points — indicates that explicit camera trajectory input does not guarantee trajectory adherence; model architecture and training play a dominant role.
-
Object Control is high across the board: most models score above 85%, with Cosmos at 94.90%, Wan2.2 at 94.01%, and OpenSora at 92.82% being top performers. Matrix Game2.0 (70.63%) is the outlier. The narrow range (70–95%) and high floor suggest that generating objects specified in the prompt is a largely solved problem — models generally include what they are asked to include, even if those objects do not behave correctly (as InterCov reveals).
-
Transitions Detect is near-ceiling for most models: OpenSoraPlan (98.29%), HunyuanVideo (98.54%), T2V-Turbo (99.02%), Director3D (99.75%), and Cosmos (98.78%) all score above 98%. Even the "worst" performers — Matrix Game2.0 at 53.41% and ViewCrafter at 95.00% — only the former shows substantial scene transition issues. For the vast majority of models, scene cuts are rare — temporal continuity in the sense of "no abrupt content shifts" is not a significant failure mode.
The Camera-Object Controllability dimension average ranges from 69.17% (HunyuanGameCraft, dragged down by its 27.96% Camera Control) to 94.01% (Wan2.2). This dimension shows the strongest separation between camera-conditioned models (which vary widely) and IT2V/T2V models (which cluster tightly at high Object Control and Transitions Detect scores but lack Camera Control scores, as indicated by dashes).
The Decoupling Pattern: Strong Visual Quality, Weak Interactive Fidelity
The paper's central empirical claim — that visual quality and interactive fidelity are decoupled in current models — is supported by the cross-dimensional pattern in Table 1. To illustrate this with specific comparisons:
-
HunyuanVideo (T2V, AgenticScore 73.96%): Temporal Flickering 98.67%, Motion Smoothness 99.31% — essentially perfect on these visual quality axes — yet InterCov 53.02% and InterOrder 46.78%, meaning it correctly models entity-level causal responses on only about half of entities and orders events correctly on fewer than half of event pairs.
-
Cosmos (IT2V, AgenticScore 75.42%): The highest Object Control score (94.90%) and strong visual quality (Temporal Flickering 98.29%, Motion Smoothness 99.17%), yet InterOrder only 51.81% — temporal event sequencing is near chance despite near-perfect object presence.
-
ViewCrafter (camera-conditioned, AgenticScore 65.88%): Dynamic Degree 100.00% (maximum possible motion) and InterStab-L 81.15% (strong revisit consistency), yet InterStab-N 4.22% — non-target regions are essentially uncontrolled, with massive spurious motion everywhere.
-
WonderWorld (camera-conditioned, AgenticScore 74.02%): Camera Control 96.12% (near-perfect trajectory following) and InterStab-L 84.96% (best across all models), yet InterStab-N 24.89% — the model remembers spatial layouts at revisit points but cannot suppress motion in regions that should be static between revisits.
These trade-offs are systematic, not incidental: across all 18 models, the standard deviation of the Interaction Effect Fidelity dimension average is substantially larger than that of Generated Video Quality, and the within-model gaps between best and worst interaction sub-metrics are typically 30–60 percentage points (e.g., Wan2.2: InterStab-L 79.68% vs. InterOrder 52.70%, a 26.98-point gap; Cosmos: InterStab-L 79.55% vs. InterOrder 51.81%, a 27.74-point gap; LargeVideoPlanner: InterStab-N 87.43% vs. InterCov 42.84%, a 44.59-point gap).
Ablation Studies and Robustness Checks
Since Omni-WorldBench is a benchmark infrastructure paper rather than a method paper proposing a novel technique, there are no traditional ablations of model components, training objectives, or hyperparameters. However, several design choices in the benchmark construction and evaluation protocol are implicitly validated through comparative analysis:
VQA-based Object Control vs. rule-based matching (Section 4.3): The paper reformulates Object Control from WorldScore's rule-based matching to a VLM-based VQA approach, arguing that "inherent limitations in detection accuracy and the susceptibility of rule-based matching to semantic errors arising from synonymy" motivate this change. The paper does not provide an explicit ablation comparing VQA-based vs. rule-based Object Control on the same set of videos, so the magnitude of improvement from this design choice is not quantified. This is a missing evaluation: a head-to-head comparison on a subset of prompts would validate whether the reformulation actually improves metric reliability.
Dynamics gating in InterStab-L (Section 4.4): The gating mechanism that penalizes static videos to zero is described as necessary to "mitigate the degeneracy of trivial static sequences." The paper does not report what fraction of generated videos trigger the static penalty (i.e., how many models produce near-static outputs that would otherwise receive artificially high InterStab-L scores), nor does it ablate the gating threshold τ_static. The sensitivity of InterStab-L across different threshold values — and how many models would see their InterStab-L scores change — is unknown.
Composite similarity in InterStab-L (SSIM vs. CLIP cosine, Section 4.4): The arithmetic mean of SSIM_gray and cosine similarity is used without ablation of alternative formulations (using only SSIM, only CLIP, min instead of mean, product, or weighted combination). The choice of equal weighting (1/2 each) is not empirically justified — it is plausible that one similarity measure correlates better with human judgments of spatial consistency than the other, and that an empirically tuned weighting would produce a better metric. This is an open question for future validation.
Prompt construction strategy comparison (Section 3.1): The paper uses two complementary construction strategies — dataset-grounded and concept-driven — but does not compare model performance on prompts from the two sources. If models perform systematically differently on dataset-grounded vs. concept-driven prompts, it would indicate that the strategies capture different difficulty dimensions. Without this analysis, we cannot assess whether the two strategies are genuinely complementary or whether one dominates the evaluation signal.
VLM reliability for InterCov and InterOrder (Section 4.4): Both metrics rely on VLM-based verification — InterCov for per-entity semantic consistency, InterOrder for pairwise temporal ordering. The paper does not report VLM accuracy on a ground-truth labeled subset of videos (e.g., human-annotated judgments of whether an entity's behavior is correct, or whether event A precedes event B). Without such calibration, the reported InterCov and InterOrder scores confound model capability with VLM reliability — a low InterCov score could mean the world model failed, or it could mean the VLM failed to correctly judge the model's output. This is the most significant unvalidated component of the evaluation pipeline.
MLLM aggregation agent reliability (Section 4.5): AgenticScore depends on an MLLM assigning weights w1, w2, w3 based on its analysis of the prompt's semantic emphasis. The paper does not evaluate the aggregation agent's consistency — whether the same prompt phrased slightly differently receives similar weights, whether prompts within the same interaction level receive similar weight distributions, or how human annotators would weight the same prompts. Without this, the final AgenticScore contains an uncharacterized source of variance from the aggregation mechanism itself.
First-frame image generation pipeline sensitivity (Section 3.1): The concept-driven prompts use FLUX.1-dev generated images (with human screening and Qwen-Image refinement) as first frames. The paper does not evaluate whether model performance differs systematically between prompts with real first frames (from the dataset-grounded pipeline) vs. generated first frames (from the concept-driven pipeline). If models perform worse on generated first frames — due to subtle distribution shifts or artifacts that survived human screening — it would bias the evaluation against models tested on concept-driven prompts.
Scene transition threshold sensitivity (Section 4.3): Transitions Detect uses PySceneDetect with a threshold τ and minimum scene length L. The paper does not report the chosen values or ablate sensitivity to these parameters. A too-lenient threshold could miss genuine scene cuts; a too-strict threshold could penalize legitimate content variation. The binary formulation (any transition → score 0) makes the metric particularly sensitive to threshold choice.
Critical Assessment
Does the paper demonstrate that current world models are limited in interactive response?
Yes, with strong evidence from the decoupling pattern. Table 1 provides systematic, multi-model evidence that interactive fidelity metrics show substantially larger variance and lower absolute scores than visual quality metrics. The key numbers: Temporal Flickering ranges 91.01–99.48% (narrow spread, high floor); InterStab-N ranges 4.22–95.76% (wide spread, low floor); InterOrder ranges 35.28–54.19% (narrow spread, low ceiling). The claim that visual quality and interactive fidelity are decoupled is supported by the fact that models with near-identical visual quality scores can have dramatically different interaction fidelity profiles — and vice versa. The range on Generated Video Quality averages (70.09–85.18%) is narrower than on Interaction Effect Fidelity averages (42.42–67.34%), confirming the primary gap is in interaction.
However, the paper does not demonstrate that this limitation is unique to world models as opposed to being an inherent limitation of current video generation architectures when tested on interaction prompts they were not optimized for. The benchmark does not include a "real video" baseline — footage of actual physical interactions — that would establish the ceiling for achievable scores. Without this, we cannot distinguish between "current models are bad at interaction" and "the benchmark metrics are inherently difficult to score highly on."
Does the paper demonstrate that Omni-Metric captures differences invisible to existing benchmarks?
Qualitatively yes, quantitatively incomplete. The paper's Figure 2(d) comparison table shows coverage gaps — VBench lacks interaction effects, physics principles, causality, and loop-closure consistency; WorldScore restricts interaction to camera motion. The paper's qualitative analysis (Figures 5 and 6) shows clear visual differences between models that would be captured by the benchmark. However, the paper does not report a direct comparison: running the 18 models through VBench and WorldScore and comparing the resulting rankings to Omni-WorldBench's rankings. If Omni-WorldBench produces a substantially different model ordering than VBench or WorldScore, that would be strong evidence that it measures something distinct. If the rankings are highly correlated, the added value is less clear. This is the most important missing experiment for validating the benchmark's claimed contribution.
Similarly, there is no analysis of whether Omni-Metric's interaction fidelity sub-metrics correlate with each other within models — in other words, does a model that scores well on InterStab-L also tend to score well on InterOrder, or are the sub-capabilities truly independent? The within-model variance reported above (e.g., Cosmos: 79.55% InterStab-L vs. 51.81% InterOrder) suggests they are independent, but a systematic correlation analysis across all 18 models would strengthen this claim.
Does the AgenticScore weighting mechanism improve over uniform averaging?
The case is argued conceptually but not empirically validated. The paper's argument — that prompt semantics should determine which evaluation dimensions matter most — is intuitively compelling. However, the paper does not report AgenticScores under uniform weighting (w1 = w2 = w3 = 1/3) as a comparison point. If rankings under uniform weighting are essentially identical to rankings under AgenticScore, the added complexity of the MLLM aggregation agent is unjustified. If they differ, the paper should provide qualitative examples showing that the AgenticScore-weighted ranking better aligns with human judgment. Neither analysis is presented.
The paper also does not report the actual weight distributions assigned by the MLLM — what fraction of prompts weight Interaction Effect Fidelity highest? What is the typical weight ratio between the most and least important dimension? Without this, we cannot assess whether the weighting is meaningfully differentiated or whether most prompts receive near-uniform weights, which would undermine the case for adaptive weighting.
Are the reported AgenticScores reliable given the single-evaluation protocol?
Reliability is unknown. Each model is evaluated once per prompt — one video generation, one metric computation. The paper does not report:
-
Variance across generation seeds: if the same model with a different random seed produces meaningfully different AgenticScores, the reported single-point estimates may not represent stable model capability. The gap between Wan2.2 (75.92%) and Cosmos (75.42%) is 0.50 percentage points — is this within or beyond typical seed-to-seed variance?
-
Variance across prompt subsets: the 1,068 prompts span diverse interaction levels and scene types. Are some models more sensitive to prompt category than others? The paper does not break down AgenticScore by interaction level (Level 1 vs. 2 vs. 3), which would reveal whether model rankings are consistent across difficulty levels or whether certain models specialize.
-
Inter-annotator agreement on prompt metadata: the entity classifications, event sequences, and revisit annotations that ground-truth the metrics are produced by human annotators. Without agreement statistics, we cannot assess annotation reliability. If annotators disagree on whether an entity is "affected" or "unaffected," InterCov scores would shift depending on which annotation is used.
Does the benchmark cover the full complexity of interactive world modeling?
No — and the paper is transparent about this. Section 6 (Limitations) acknowledges that "Omni-WorldSuite cannot fully capture the complexity of open-world interactive environments, especially long-horizon and highly dynamic settings." The benchmark evaluates single video generations of fixed short duration (most models generate 25–132 frames at 8–24 FPS, corresponding to roughly 3–8 seconds of video). This tests immediate interaction response, not the kind of extended, multi-step interactive sequences that world models are ultimately intended for — autonomous driving over minutes, robotic manipulation over hundreds of steps, game play over extended sessions.
Additionally, the benchmark tests interaction response in a single forward pass — the model generates one video given the prompt, and there is no feedback loop where the model's output becomes the input for the next interaction step. This is a fundamentally different evaluation paradigm from the closed-loop, agent-in-the-loop settings where world models are deployed. A model that performs well on Omni-WorldBench's single-shot interaction prompts may fail catastrophically when asked to generate a sequence of 50 interaction steps where each step's initial state depends on the previous step's output.
Are there missing baselines that would strengthen the evaluation?
Several important baselines are absent:
-
Real video ground truth: running the same metrics on real videos of the specified interactions would establish the performance ceiling and reveal whether low scores reflect model limitations or metric limitations.
-
Human evaluation: the paper mentions plans to "release human-aligned evaluation results in the future" (Section 6), but currently reports only automated metrics. Without human judgments of interaction fidelity on at least a subset of prompts, we cannot assess whether the automated metrics (particularly the VLM-based InterCov and InterOrder) align with human perception of causal consistency.
-
Random / trivial baselines: evaluating a model that generates constant black frames, or a model that generates static repeats of the first frame, would establish the performance floor and validate that the dynamics gating and stability metrics correctly penalize degenerate outputs.
-
VBench / WorldScore on the same videos: as noted above, running prior benchmarks on the same generated videos would enable direct correlation analysis between Omni-WorldBench scores and existing metrics, quantifying the incremental value.
Does the paper's evidence support its positioning as "the first benchmark dedicated to assessing interactive response capabilities"?
The claim is technically accurate but narrower than it might appear. WorldModelBench (Li et al., 2025, cited in Figure 2d) does include action-conditioned generation and some physics evaluation — it is not solely a visual quality benchmark. Omni-WorldBench's distinctiveness relative to WorldModelBench lies in: (1) the three-level interaction hierarchy providing graded capability assessment, (2) the four-metric decomposition of interaction effect fidelity (InterStab-L/N, InterCov, InterOrder) providing finer-grained diagnostic signal than WorldModelBench's more aggregated evaluation, and (3) the semantic-weighted AgenticScore fusion. The paper's Figure 2(d) comparison is accurate in showing coverage gaps, but it somewhat understates WorldModelBench's interaction-related capabilities by marking it as lacking interaction, causality, and physics — WorldModelBench does evaluate some of these, just not with the same systematic hierarchy or metric granularity.
The contribution is better characterized as "the first benchmark to make interaction effect fidelity the central, systematically graded evaluation target with per-prompt semantic weighting" rather than "the first benchmark to consider interaction at all." This is a meaningful but incremental advance over WorldModelBench rather than a completely novel evaluation paradigm.
What experiments would most strengthen the paper?
-
Correlation analysis with existing benchmarks: run all 18 models through VBench and WorldScore, report Spearman/Pearson correlations between the resulting rankings and Omni-WorldBench rankings, and show where they diverge.
-
Human alignment study on a subset: have human raters judge interaction fidelity, video quality, and controllability on 50–100 videos, report correlation with automated metrics, and calibrate thresholds.
-
Seed variance analysis: generate 3–5 videos per prompt for 2–3 representative models, report standard deviations on AgenticScore and sub-metrics.
-
Performance breakdown by interaction level: report AgenticScore and sub-metric scores separately for Level 1, Level 2, and Level 3 prompts to reveal whether model rankings are consistent across difficulty.
-
Ablation of AgenticScore weighting: compare rankings under uniform weighting vs. MLLM-weighted scoring, and against human judgments of overall quality.
-
Real video ceiling: collect or generate real videos for a subset of prompts where ground-truth footage exists or can be filmed, compute all metrics on real videos to establish the score ceiling.
-
VLM calibration for InterCov/InterOrder: on a human-labeled subset of 50–100 videos, compute VLM accuracy against human judgments to quantify the noise introduced by VLM-based verification.
6. Limitations and Trade-offs
6.1 The Cost of Difficulty Estimation Is Not Accounted For in Headline Efficiency Numbers
The assumption or constraint. The paper explicitly acknowledges in Section 3.2 that estimating prompt difficulty via the PRM-based method requires "generating 2048 samples and scoring them," and that "our experiments do not account for this cost largely for simplicity." The compute-optimal policy selection depends on knowing which difficulty bin each prompt falls into, which in turn requires either oracle ground-truth labels (impossible in deployment) or the predicted difficulty estimation procedure that itself consumes substantial inference compute — potentially more than the largest test-time budgets studied (256–512 generations).
The consequence. The reported 4× efficiency gains over best-of-N baselines are computed after difficulty is already known, without amortizing the cost of learning it. In a real deployment, the total compute would be cost(difficulty estimation) + cost(strategy execution), and the former could dominate the latter, erasing or even reversing the claimed efficiency advantage. A practitioner evaluating whether to adopt compute-optimal test-time scaling must compare the end-to-end cost of (difficulty estimation + adaptive strategy) against simply running a fixed budget (e.g., best-of-N with N=256). The paper provides no evidence that the end-to-end comparison favors the adaptive approach — only that the strategy execution phase is more efficient given difficulty knowledge.
What evidence exists in the paper. The paper acknowledges this as an open problem (Section 3.2: "this additional cost can be viewed as an exploration-exploitation trade-off... we leave the development of more efficient difficulty estimation methods to future work"), but provides no upper bound on how much difficulty estimation costs relative to strategy execution, no experiment showing that simpler difficulty estimation (e.g., from fewer than 2048 samples) preserves the gains, and no end-to-end cost comparison.
Mitigation status. Not addressed experimentally. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), but no such model is developed or evaluated. Until efficient difficulty estimation is demonstrated, the 4× figure should be understood as an upper bound on achievable efficiency rather than a realized deployment gain.
6.2 The Benchmark Does Not Include Ground-Truth or Human-Annotated Baselines
The assumption or constraint. Omni-WorldBench evaluates generated videos against prompt specifications — the metrics measure whether actions produced specified effects, not whether the generated video matches a reference real video or aligns with human judgments of interaction quality. The paper states in Section 6 (Limitations) that "we plan to release human-aligned evaluation results in the future to further complement and validate the assessment of interaction quality," but no such results are included in the current work. There is no "real video" ceiling establishing maximum achievable scores, no human evaluation correlating automated metrics with human preferences, and no trivial baseline (e.g., static first-frame repetition, black frames) that would floor the metrics.
The consequence. Without a ground-truth ceiling, we cannot determine whether low scores reflect model limitations or inherent metric difficulty. For example, InterOrder scores across all 18 models range from 35.28% to 54.19% — none exceeds 55%. Is this because all models fundamentally fail at temporal event ordering, or because the VLM-based verifier used to compute InterOrder is itself unreliable at judging event precedence? The two explanations have radically different implications: the former indicts model architectures, the latter indicts the metric. Similarly, without human alignment, we cannot assess whether AgenticScore rankings reflect what a human evaluator would consider better interactive responses, or whether the metrics are systematically biased toward or against certain model families, interaction types, or visual styles.
What evidence exists in the paper. The paper reports no human study, no real-video baseline, and no trivial model baselines. The VLM-based metrics — InterCov (object-level causal faithfulness via VLM binary verification) and InterOrder (temporal event ordering via VLM pairwise consistency) — are presented without calibration against human judgments, meaning the reported scores confound model capability with VLM accuracy. The most extreme example: ViewCrafter's InterStab-N score of 4.22% (Table 1) could reflect either massive spurious motion in non-target regions (a genuine failure) or systematic failure of the optical flow pipeline on ViewCrafter's output distribution (an extraction artifact). The paper provides no evidence to distinguish these.
Mitigation status. Explicitly deferred to future work (Section 6). The paper's current claims about model capability rankings and the decoupling of visual quality from interactive fidelity are based entirely on automated metrics with unvalidated reliability against human judgment.
6.3 Single-Evaluation Protocol Provides No Variance Estimates for Model Comparison
The assumption or constraint. As noted in the experimental analysis, each model is evaluated exactly once per prompt — one video generation with one random seed, one pass through the metric pipeline. The paper reports no variance estimates (standard deviations, confidence intervals, seed-to-seed variation) for any metric or for the final AgenticScore. Section 5.2 describes fixed inference configurations for each model (specific step counts, guidance scales, resolutions) but does not report whether results are robust to stochastic variation in the generation process.
The consequence. We cannot assess whether reported performance differences between models are statistically meaningful. The gap between the top two models — Wan2.2 (AgenticScore 75.92%) and Cosmos (75.42%) — is 0.50 percentage points. If seed-to-seed standard deviation on AgenticScore is, say, 1.5 percentage points (plausible for diffusion-based generation), this difference is well within noise and the ranking is unreliable. More broadly, the entire leaderboard in Table 1 is deterministic given the prompts and model weights; any re-ranking that would occur under different random seeds is invisible. A practitioner choosing between Wan2.1 (73.21%) and CogVideo (73.27%) — a 0.06 percentage point gap — has no information about which model is actually better, or whether they are indistinguishable.
What evidence exists in the paper. None. The paper does not report standard deviations, does not mention multiple-run averaging, and does not discuss seed-to-seed variability. The 1,068 prompts provide coverage across prompt types but within each prompt, evaluation is a single data point for each model.
Mitigation status. Not addressed. This is a significant methodological gap for a benchmark paper whose primary output is a comparative leaderboard. Standard practice in benchmarking (VBench, WorldScore) includes either multiple evaluation runs with different seeds or an analysis of metric stability. The absence here weakens the actionable value of the reported rankings.
6.4 The VLM-Based Metrics (InterCov, InterOrder, Object Control) Are Uncalibrated
The assumption or constraint. Three of the fourteen evaluation metrics — InterCov (Section 4.4), InterOrder (Section 4.4), and the VQA-based Object Control (Section 4.3) — rely on Vision-Language Models (VLMs) to make semantic judgments about generated videos: whether an entity's behavior is causally consistent with the prompt, whether events occur in the correct temporal order, and whether a specified object is present. The paper provides no calibration of these VLMs against human judgments on the specific task of evaluating world model outputs. The AgenticScore aggregation mechanism (Section 4.5) further relies on an MLLM to assign per-prompt weights w1, w2, w3 — also uncalibrated.
The consequence. The reported InterCov, InterOrder, and Object Control scores confound world model capability with VLM reliability. If the VLM used for InterCov has, for example, 70% accuracy at judging whether a baseball throw looks causally correct — misclassifying 30% of cases — then a world model that is actually 90% faithful would appear to score approximately 0.9 × 0.7 + 0.1 × 0.3 ≈ 66% if errors are independent, or different values under different error correlation structures. The reported InterCov range of 40.29–64.52% (Table 1) could reflect a combination of genuine model variance and VLM noise, with unknown proportions.
For InterOrder, the stakes are even higher: all 18 models score in the 35–55% range, with no model exceeding 55%. If the VLM is only modestly above chance at judging event ordering in generated videos, these scores could primarily reflect VLM capability rather than model capability — the models might all be equally poor, equally mediocre, or meaningfully different, and we cannot distinguish these cases. The Aggregation agent adds a further uncalibrated layer: its weight assignments may not align with how a human evaluator would weight the dimensions for a given prompt, introducing systematic bias into the final AgenticScore.
What evidence exists in the paper. None. The paper does not report VLM accuracy on a human-labeled subset of videos for InterCov or InterOrder, does not report inter-rater agreement between multiple VLMs (which would bound reliability), and does not analyze whether different VLMs would produce different model rankings on these metrics. The choice of which VLM to use is not disclosed with sufficient specificity for replication (model name, version, prompt template).
Mitigation status. Not addressed. This is the most significant methodological concern for the paper's central contribution — the Interaction Effect Fidelity metrics — because the entire argument that current models are deficient at interactive response depends on these metrics being reliable measures of interactive response. If the VLMs are noisy or biased, the claimed capability gap may be partly or largely a measurement artifact.
6.5 Single Benchmark, Single Model Family, Single Task Modality
The assumption or constraint. All experiments evaluate video-based world models — the benchmark is designed for and tested exclusively on video generation models producing visual outputs. The prompt suite covers general scenes and three task-oriented domains (autonomous driving, embodied AI, gaming), but all tasks require generating video frames as the output modality. The paper does not evaluate on non-video world models (e.g., MLLM-based state predictors, physics simulators with symbolic state representations) and does not test generalization to tasks where correctness is judged by downstream task performance (e.g., planning success rate, control accuracy) rather than visual metrics.
The consequence. The benchmark's findings — that visual quality and interactive fidelity are decoupled, that camera-conditioned models show sharp trade-offs, that temporal ordering is a near-universal weakness — are specific to the video generation paradigm and the Omni-WorldSuite prompt distribution. It is unknown whether these findings generalize to other world model paradigms (e.g., latent-state world models that predict in representation space rather than pixel space) or to other domains (e.g., robotic simulation where fidelity would be measured by task completion rather than visual consistency). A latent-state world model that never generates pixels would receive no score on Omni-WorldBench, yet might exhibit far stronger interactive response in its latent dynamics — the benchmark cannot evaluate such models at all. Additionally, the 18 evaluated models are predominantly from a few research groups and model families (Wan, Hunyuan, Cosmos, CogVideo), and the findings may partly reflect shared architectural biases rather than general properties of world models.
What evidence exists in the paper. The evaluation is restricted to video generation models producing pixel outputs (Section 5.1). The paper does not evaluate on non-pixel world models, does not test correlation between Omni-WorldBench scores and downstream task performance (e.g., planning accuracy in a driving simulator), and does not evaluate on a held-out domain to test whether the difficulty-dependent patterns observed (Level 1 easier than Level 3, camera models weaker on non-target stability) transfer to new scene types.
Mitigation status. The paper acknowledges scope limitations in Section 6: "Omni-WorldSuite cannot fully capture the complexity of open-world interactive environments, especially long-horizon and highly dynamic settings." However, it does not discuss the modality restriction (video-only) or the absence of downstream task validation. The benchmark is positioned as evaluating "4D generation" — joint spatial and temporal modeling under action conditions — but the evaluation only probes this through generated video pixels, not through the downstream use cases (planning, control, counterfactual reasoning) that motivate world modeling.
6.6 The Interaction Prompts Test Single-Shot Generation, Not Closed-Loop Interactive Sequences
The assumption or constraint. Omni-WorldBench evaluates world models in a single forward pass: the model receives a prompt (text, first frame, optionally camera trajectory) and generates one video of fixed duration (typically 3–8 seconds, 25–132 frames). There is no feedback loop where the model's output becomes the input for the next interaction step, no multi-turn evaluation where the model must maintain state coherence across extended sequences, and no evaluation of whether the model can be used as a simulator in a control loop where actions are chosen based on its predictions and fed back as new conditioning inputs.
The consequence. This evaluation paradigm fundamentally differs from how world models are deployed in practice. In autonomous driving, a world model would be queried repeatedly — predict the next few seconds given the current state, the planner selects an action, the action is executed (or simulated), the new state becomes the initial condition for the next prediction, and so on. In this closed-loop setting, small prediction errors compound: a slight misprediction of a vehicle's position in one step becomes the inaccurate initial state for the next step, whose error then propagates further. A model that achieves high scores on Omni-WorldBench's single-shot prompts — correctly predicting the immediate consequence of a throw or a grasp — may fail catastrophically over 50 sequential interaction steps due to error accumulation that the benchmark never measures. The benchmark thus evaluates short-horizon interactive response but provides no signal about long-horizon simulation stability, which is the actual capability needed for planning and decision-making applications.
What evidence exists in the paper. The paper acknowledges this scope limitation in Section 6: "Omni-WorldSuite cannot fully capture the complexity of open-world interactive environments, especially long-horizon and highly dynamic settings." The evaluation protocol (Section 5.2) confirms single-pass generation: each model generates one video per prompt and is scored on that single output. There is no multi-step evaluation, no analysis of how prediction errors propagate over extended horizons, and no comparison between single-shot benchmark scores and closed-loop task performance.
Mitigation status. Acknowledged as a limitation (Section 6) but deferred entirely to future work. The paper does not propose or prototype a multi-step evaluation extension. This is a significant gap because the benchmark measures the necessary condition for world modeling (can the model produce a causally correct response to a single interaction?) but not the sufficient condition (can the model maintain causally correct responses over extended interactive sequences?). A practitioner evaluating whether a high AgenticScore model is suitable for deployment in a planning pipeline needs closed-loop evidence that the benchmark does not provide.
7. Implications and Future Directions
How This Work Changes the Landscape
Omni-WorldBench does not propose a new model architecture, training objective, or generation paradigm. Its primary intervention is establishing interactive response as a first-class, measurable evaluation target for world models, separable from the visual quality metrics that have dominated the field's self-assessment. The paper provides the measurement infrastructure — test prompts with annotated interaction ground truth, extraction pipelines that convert pixels to structured representations, and metrics that decompose interactive fidelity into independently quantifiable sub-capabilities — that makes it possible to ask "does this model understand what happens when a robot arm grasps a bottle?" rather than "does this video look smooth?" This is a measurement-driven reframing, not a paradigm shift in model design, but its consequences for research practice could be substantial if the benchmark achieves adoption.
The landscape impact operates at three levels. First, it changes what counts as progress. Prior to Omni-WorldBench, a new world model could claim improvement by demonstrating higher VBench scores — better temporal flickering, improved motion smoothness, stronger text-video alignment. Table 1 shows these metrics are largely saturated: 14 of 18 models exceed 96% on Temporal Flickering, all 18 exceed 95% on Motion Smoothness. Optimizing further on these axes produces diminishing returns, yet the field had no alternative evaluation target that isolated world-modeling capability from general video generation quality. Omni-WorldBench provides that alternative: a model's AgenticScore, and specifically its Interaction Effect Fidelity sub-scores, directly measures whether it represents causal state transitions. A research team can now ask whether a new architecture, training objective, or data strategy improves InterOrder (temporal event sequencing) or InterCov (entity-level causal faithfulness) — metrics on which current models score between 35–55% and 40–65% respectively — rather than chasing sub-percentage gains on already-saturated quality metrics. This redirects optimization pressure from where it is easiest to measure to where it matters most for world modeling.
Second, it reveals a structural decoupling that changes intuitions about model progress. The implicit assumption in much of the video generation literature — that scaling models, data, and compute will naturally produce better world models because better video generation implies better world simulation — receives systematic counterevidence in Table 1. Cosmos achieves 98.29% Temporal Flickering and 99.17% Motion Smoothness (near-perfect visual quality) yet only 51.81% InterOrder (near-chance temporal event sequencing). ViewCrafter achieves 100% Dynamic Degree (maximum motion quantity) yet 4.22% InterStab-N (essentially zero non-target stability). WonderWorld achieves 96.12% Camera Control (near-perfect trajectory following) yet 24.89% InterStab-N (severe spurious background motion). These are not incidental outliers — they are systematic patterns showing that high performance on one capability dimension does not predict high performance on another, and that the dimensions most critical for world modeling (interactive causality) are the ones where models are weakest. This decoupling finding, made visible by the benchmark's multi-dimensional disaggregation, challenges the scaling assumption: additional investment in pretraining data or model capacity for video generation may not close the interactive fidelity gap because the gap is not about insufficient visual quality. The field must develop training objectives, architectures, or data strategies specifically targeting causal consistency — and Omni-WorldBench provides the measurement tool to determine whether such efforts work.
Third, it resolves a latent tension between the world modeling and video generation communities. World model researchers have increasingly adopted video generation as an implementation pathway, yet the evaluation criteria borrowed from video generation (FVD, IS, VBench) measure a different construct than what world models are supposed to do. This creates a misalignment: the video generation community sees high scores and considers the models successful; the world modeling community uses the same models for planning and simulation and encounters failures that the scores did not predict. Omni-WorldBench resolves this tension by providing evaluation that aligns with the world modeling use case — measuring whether actions produce correct state transitions — while being compatible with video generation as the output modality. It does not reject video quality metrics but supplements them with metrics that matter for the downstream task. This turns an implicit conflict (are these video generators or world models?) into an explicit measurement framework: a model is a good world model to the extent that it scores well on interaction effect fidelity, regardless of its video quality scores, and the two can be jointly reported to characterize both visual competence and causal reasoning.
The benchmark also makes certain research directions more attractive. Work on verifier-guided generation, where a world model's outputs are filtered, reranked, or refined based on consistency checks, becomes newly tractable because Omni-Metric provides the consistency metrics (InterStab-L, InterStab-N, InterCov, InterOrder) that can serve as both evaluation targets and training signals. Work on explicit physical reasoning modules — augmenting video generation with differentiable physics simulators, graph-based causal models, or structured state representations — becomes more clearly motivated because the benchmark isolates the specific capabilities (spatial consistency, non-target stability, entity-level causal response, temporal ordering) that pure video generation models lack. Work on closed-loop evaluation, where models are tested on multi-step interactive sequences rather than single-shot generation, becomes the natural extension since the benchmark establishes that single-shot interactive response is measurable; the next question is whether such response remains coherent over extended horizons.
Conversely, the benchmark makes certain directions less attractive in the near term. Purely scaling video generation pretraining — larger models, more data, bigger compute budgets — without changing the training objective or architecture to target causal consistency is implicitly discouraged: if visual quality is already near-ceiling and the binding constraint is interactive fidelity, scaling pretraining that optimizes visual quality will drive diminishing returns on the metrics that matter. Investment in incremental visual quality improvements (sub-percent gains on flickering or smoothness) is similarly de-prioritized relative to investment in causal reasoning capabilities, because the benchmark shows that visual quality is not what distinguishes good world models from poor ones.
Follow-Up Research This Work Enables
Cross-benchmark correlation analysis to quantify Omni-WorldBench's distinctiveness. The most important immediate follow-up is to run the 18 evaluated models through VBench and WorldScore, generate videos using the same models and prompts, and compute Spearman rank correlations between the resulting model orderings and Omni-WorldBench's AgenticScore ranking. If Omni-WorldBench produces a substantially different model ordering than VBench (low rank correlation), it empirically demonstrates that interactive fidelity is a distinct capability not captured by visual quality metrics. If the rankings are highly correlated, the added value of the interaction metrics is unclear — researchers might achieve similar diagnostic power with simpler, cheaper metrics. This experiment should also report correlation between each Omni-Metric sub-dimension (AI, AG, AC) and the corresponding dimensions in existing benchmarks, to identify which specific sub-capabilities drive any observed divergence. A companion human study on 100–200 randomly sampled videos — where human raters judge overall world modeling quality, interaction fidelity, and visual quality on Likert scales — would calibrate whether Omni-WorldBench or an existing benchmark better predicts human judgments. This is the critical validation experiment that determines whether the benchmark measures something real and distinct, or merely adds complexity.
Closed-loop multi-step evaluation to test the simulation stability assumption. Omni-WorldBench tests single-shot interactive response: one prompt, one generated video. The motivating use case for world models — planning, control, counterfactual reasoning — requires closed-loop operation where the model's output becomes the initial state for the next prediction step. A natural extension is Omni-WorldBench-Sequential: a subset of 100–200 prompts from the existing suite, converted to multi-step sequences where the action description is decomposed into 5–10 sequential sub-actions, and after each step, the generated video's final frame is extracted and fed as the first frame for the next step. This tests whether interactive fidelity degrades over extended sequences due to compounding prediction errors. The hypothesis to test: do models with similar single-shot AgenticScores diverge in closed-loop performance? Specific failure modes to characterize: at what step count does spatial consistency (InterStab-L) collapse? Does non-target instability (InterStab-N) increase with sequence length? Do temporal ordering errors (InterOrder) accumulate? This extension would convert Omni-WorldBench from a necessary-condition test (can the model respond to one interaction?) to a sufficient-condition test (can the model sustain interactive responses over the horizons needed for planning?). The paper's own limitation statement in Section 6 explicitly identifies this gap, making it the most natural next step.
VLM calibration and metric reliability quantification for InterCov and InterOrder. The most significant unvalidated component of Omni-Metric is the reliance on VLM-based verification for InterCov (entity-level causal faithfulness) and InterOrder (temporal event ordering). A focused calibration study should: (1) collect human judgments on 200–300 video–prompt pairs for InterCov (binary: is entity X's behavior consistent with the prompt?) and InterOrder (binary: does event A precede event B in this video?) using 3–5 annotators per video to establish inter-annotator agreement baselines; (2) evaluate 3–5 different VLMs (e.g., GPT-4V, Gemini, Qwen-VL, Claude) on the same videos with the same structured query protocol; (3) report VLM accuracy against the human majority vote, plus sensitivity and specificity broken down by interaction level (Level 1/2/3) and by entity type (rigid objects, articulated objects, fluids, humans). This would bound the noise introduced by VLM-based metrics: if VLM accuracy is, say, 85% against human judgments, InterCov scores in the 40–65% range (Table 1) primarily reflect model limitations, not measurement error. If VLM accuracy is closer to 60–70%, the reported scores substantially confound model and metric capability, and the benchmark's conclusions about the interaction fidelity gap must be tempered. This calibration is essential for the benchmark to be taken seriously as a measurement instrument — without it, a skeptic can attribute any observed model deficiency to unreliable metrics.
Difficulty-conditioned model analysis across the interaction hierarchy. The paper reports aggregate AgenticScores across all 1,068 prompts but does not break down performance by interaction level. A Level 1 entity-self prompt (e.g., "a dewdrop rests on a lotus leaf") tests fundamentally different capabilities than a Level 3 global cascading prompt (e.g., "a robotic arm grasps a bottle and hands it to a person"). The obvious follow-up analysis — using the already-collected data — is to report AgenticScore and sub-metric scores separately for Level 1, Level 2, and Level 3 prompts for each of the 18 models. The key questions: does model ranking change across levels? Are some models good at simple dynamics but collapse on multi-object cascades? Does the decoupling pattern (high visual quality, low interactive fidelity) hold within each level or is it driven primarily by Level 3 prompts? This analysis would reveal whether the interactive fidelity gap is uniform across complexity or concentrated at specific interaction scales — a crucial diagnostic for model developers deciding where to focus improvement efforts. If a model scores acceptably on Level 1 and 2 but fails catastrophically on Level 3, the intervention is different (improve multi-object causal chain modeling) than if it fails uniformly across all levels (fundamental deficiencies in action-conditioned generation).
Real-video ceiling and metric score calibration. To interpret Omni-WorldBench scores meaningfully, researchers need to know what scores are achievable by ground-truth physical reality. The experiment: identify a subset of 50–100 prompts from the dataset-grounded pipeline (drawn from DriveLM, InternData-A1, Sekai) where real video footage of the specified interaction exists. Run the full Omni-Metric pipeline on these real videos — the same extraction (GroundingDINO, SAM, RAFT), the same metrics, the same AgenticScore aggregation. The resulting scores establish the ceiling: the maximum AgenticScore achievable by footage of actual physics unfolding, as measured by these automated metrics. If the real-video AgenticScore is, say, 85%, then Wan2.2's 75.92% represents substantial progress toward the ceiling. If the real-video score is 95%, the gap is larger and the metrics are capable of near-perfect discrimination. If the real-video score is only 80% — lower than some models — then the metrics fundamentally fail to reward physical realism over generated artifacts, and the benchmark requires metric redesign. This experiment also reveals which sub-metrics are ceiling-limited: if real videos score only 60% on InterOrder, the metric (not the models) is the binding constraint on that dimension. Without this ceiling, reported scores are uncalibrated — a 50% InterOrder could mean "models are halfway to perfect ordering" or "the metric saturates at 60% even for perfect videos."
Training interventions that target interactive fidelity sub-capabilities. Omni-WorldBench's decomposition of interactive fidelity into four sub-metrics (InterStab-L, InterStab-N, InterCov, InterOrder) enables targeted improvement experiments. A research team could take a base model (e.g., Cosmos, which scores 79.55% InterStab-L but 51.81% InterOrder in Table 1) and experiment with interventions specifically designed to improve the weakest sub-capability. For InterOrder (temporal event sequencing), one could fine-tune the model on videos annotated with temporal event boundaries, or add an auxiliary loss that penalizes event order reversals detected by a VLM during training, then measure whether InterOrder improves without degrading other sub-metrics. For InterStab-N (non-target stability), one could add a segmentation-guided loss that penalizes optical flow magnitude in regions masked as non-target, then evaluate on the benchmark to see if the 4.22–24.89% scores of camera-conditioned models can be raised to the 80%+ levels of IT2V models. These experiments would validate that the benchmark's diagnostic granularity actually guides model improvement — if targeted training on a specific sub-capability reliably improves that sub-metric without degrading others, the decomposition is actionable. If improvements on one sub-metric consistently come at the expense of others (suggesting inherent trade-offs in current architectures), that is also a valuable finding that the decomposition makes visible.
Practical Applications and Downstream Use Cases
Model selection for embodied AI simulation pipelines. A robotics lab developing manipulation policies using world models as simulators — predicting what happens when a robot arm executes a grasp, a push, or a placement — needs to choose which world model to deploy. The key requirement is not visual quality (the simulation is used for planning, not for human viewing) but interactive fidelity: does the model correctly predict object motion under robot actions? Omni-WorldBench provides direct evidence. Wan2.2 achieves the highest Interaction Effect Fidelity average (67.34%, Table 1) and the highest InterCov among IT2V models (56.99% — though Wan2.1 scores higher on InterCov at 64.52%), making it the best candidate for manipulation simulation among the evaluated models. Cosmos (Interaction Effect Fidelity 66.22%, InterStab-N 79.63%) would be preferred if non-target stability is critical — e.g., simulating a tabletop manipulation where background objects must remain static. A lab could further evaluate candidates on a custom subset of Omni-WorldSuite prompts matching their specific manipulation domain (Level 2 and 3 prompts involving robotic arms, grasping, and object transfer) before committing to integration. The benchmark reduces model selection from guesswork — "this model looks good in demos" — to quantitative comparison on the exact capability that matters.
Training data filtering for world model fine-tuning. When fine-tuning a world model on domain-specific interaction data (e.g., autonomous driving footage, robot manipulation videos), the quality of training data determines downstream performance. Current practice often uses all available data, including videos where actions do not produce expected effects (camera shake, annotation errors, physically implausible outcomes). Omni-Metric's interaction effect fidelity metrics — particularly InterCov (does each entity respond correctly to the specified action?) and InterOrder (do events occur in the correct sequence?) — can be run on candidate training videos to filter out low-fidelity examples before fine-tuning. A driving dataset could be scored: videos where the ego-vehicle's action (lane change, braking) does not produce the expected effect on surrounding vehicles (low InterCov) would be excluded from the training set. This filtering could improve the fine-tuned model's interactive fidelity by removing training examples that teach incorrect causal associations. The benchmark provides the filtering infrastructure — the same extraction pipeline (GroundingDINO + SAM for trajectories, RAFT for optical flow, VLM for semantic verification) and the same metrics used for evaluation can be applied to training data curation, creating a feedback loop where better training data (as judged by the metrics) produces better models (as evaluated by the same metrics on held-out prompts).
Capability regression testing during model development. World model development is iterative: new architectures, training recipes, and data mixtures are tested, and developers need to know whether a change improves, degrades, or leaves unchanged the model's interactive capabilities. Because Omni-WorldBench decomposes interactive fidelity into sub-capabilities, a model team can track per-metric scores across development iterations to catch regressions that aggregate scores would mask. For example, a new training objective that improves overall AgenticScore from 73% to 75% might actually degrade InterStab-N from 80% to 60% while boosting Dynamic Degree from 45% to 90% — the aggregate improves, but the model has become worse at suppressing spurious background motion, which may cause failures in downstream deployment where static backgrounds are expected. Omni-WorldBench makes this regression visible: a dashboard tracking InterStab-L, InterStab-N, InterCov, InterOrder, and AgenticScore across commits would alert developers when a change trades off interaction fidelity for visual dynamics, enabling informed decisions about whether the trade-off is acceptable for the target application. The benchmark's 1,068 prompts and 14 metrics provide sufficient granularity to detect capability-specific regressions that a single aggregate score would obscure.
When to Prefer This Method
The paper does not propose a method that competes with named alternatives — it is a benchmark infrastructure contribution, and the question of "when to prefer Omni-WorldBench over another benchmark" is largely answered by the coverage comparison in Figure 2(d) and the empirical decoupling pattern in Table 1. The decision rule implicit in the paper's positioning is:
- Prefer Omni-WorldBench when the evaluation goal is to assess interactive world modeling capability — specifically, whether a video generation model produces causally faithful state transitions under action conditions, with fine-grained diagnostic information about which sub-capabilities (spatial revisit consistency, non-target stability, entity-level causal response, temporal event ordering) are strong or weak. This is the only benchmark that measures these capabilities.
- Supplement with VBench or VBench++ when visual quality baselines are needed — Omni-WorldBench adopts several VBench metrics (Temporal Flickering, Motion Smoothness, Dynamic Degree, Imaging Quality) but does not cover all VBench dimensions. If comprehensive visual quality characterization (e.g., aesthetic quality, subject consistency, background consistency) is needed alongside interactive fidelity, run both benchmarks.
- Supplement with WorldScore when geometric consistency under pure camera motion is the primary concern — WorldScore evaluates 3D spatial coherence under viewpoint changes without action-driven state changes, which complements Omni-WorldBench's interaction-centric evaluation. A model intended for novel view synthesis (camera moves through static scene) should be evaluated with WorldScore; a model intended for interactive simulation (actions change the world) should be evaluated with Omni-WorldBench; a general-purpose world model should be evaluated with both.
- Do not prefer Omni-WorldBench when evaluating non-video world models — the benchmark requires pixel-space video generation as output. Latent-state world models, MLLM-based state predictors, or physics simulators that operate in symbolic or latent state spaces cannot be evaluated with the current Omni-Metric pipeline without adaptation (e.g., decoding latent states to pixels for metric computation, which introduces a decoder quality confound).
