Rethinking Video Generation Model for the Embodied World

ArXiv: 2601.15282

🎯 Pitch

State-of-the-art video generation models consistently fail at basic robot tasks, earning a mere 58% fidelity score on actions like putting a teapot on a stove, despite excelling in visual smoothness. The novel RBench benchmark reveals this stark domain gap with a 0.96 human correlation, while its paired 4-million-clip RoVid-X dataset provides the first dedicated resource to close it.

---

1. Executive Summary

This paper introduces RBench, a systematic benchmark for evaluating video generation models on embodied robotics tasks, and RoVid-X, a complementary large-scale training dataset containing 4 million annotated robot video clips designed to address the physical realism deficiencies that the benchmark exposes. RBench evaluates 25 representative models—spanning commercial, open-source, and robotics-specific families—across five task domains and four robot embodiments using reproducible automated metrics that assess both task-level correctness (physical-semantic plausibility, task-adherence consistency) and visual fidelity (motion amplitude, robot-subject stability, motion smoothness), achieving a Spearman correlation of 0.96 with human evaluations. The benchmark reveals a persistent "domain gap" where consumer-oriented models optimized for visual smoothness underperform on physically grounded tasks, and where even the strongest models struggle with fine-grained manipulation and visual reasoning, establishing that proficiency in creative video generation does not automatically transfer to embodied AI requirements without physically-grounded training data and evaluation protocols that transcend perceptual metrics.

2. Context and Motivation

The Core Problem: We Can't Tell How Well Video Models Understand the Physical World

The fundamental question this paper tackles is deceptively straightforward: when a video generation model produces a clip of a robot performing a task, how do we know whether it actually understands the physical interaction it's depicting? This matters because video generation is rapidly evolving from a creative media tool into a critical component of embodied AI pipelines — a shift that demands entirely different evaluation criteria than the ones the field currently uses.

To understand why this problem is acute right now, we need to appreciate the convergence of two trends. First, video generation models have undergone a dramatic capability leap. Models like Wan, Sora, Seedance, and Veo can now synthesize high-resolution, temporally coherent videos from text or image prompts at scales that were unthinkable just a few years ago. Second, the robotics and embodied AI communities have begun treating these models not merely as media generators but as world simulators — systems that can generate synthetic robot trajectories for policy learning, predict future states for planning, or serve as differentiable simulators for training visuomotor policies. The paper cites a growing body of work using video models to synthesize robot data as an alternative to expensive human teleoperation, to extract executable actions through inverse dynamics models, and to initialize robot policies through video-based pretraining.

This dual use creates a tension. A video model optimized for cinematic appeal — with smooth camera motions, aesthetically pleasing color grading, and visually coherent but physically impossible transitions — might score highly on conventional perceptual metrics while producing outputs that are actively harmful if used as training data or simulation inputs for a physical robot. A robot policy trained on generated videos where grippers phase through objects, where objects float without support, or where action sequences skip critical steps will inherit those physical misunderstandings when deployed in the real world. The evaluation problem is therefore not academic: it directly impacts the safety and reliability of embodied systems built on top of video generation models.

Why Existing Benchmarks Don't Answer the Right Questions

The paper identifies three categories of existing evaluation frameworks, each of which falls short for embodied robotics in specific ways.

Perceptual quality benchmarks (VBench, EvalCrafter, Video-Bench) focus on pixel-level fidelity. These benchmarks measure attributes like frame clarity, texture quality, motion smoothness, and text-video semantic alignment. They ask questions like "is this video sharp?", "are the colors natural?", and "does the motion look fluid?" While these metrics are valuable for consumer video applications, they are fundamentally indifferent to physical correctness. A video of a robot arm that clips through a table to grasp an object — physically impossible in the real world — can score perfectly on perceptual quality if the pixels are sharp and the motion is smooth. The paper demonstrates this failure mode explicitly in Figure 2, where high-scoring generated videos exhibit structural distortion, floating components, and key action omissions that perceptual metrics would never flag.

Physics-focused benchmarks (VideoPhy, PhyBench, T2VPhysBench) isolate physical constraints but lack task grounding. Recent work has introduced benchmarks that test whether generated videos respect basic physical laws — does an object fall when dropped? Does a collision produce a plausible reaction? Does a liquid pour realistically? These benchmarks represent an important step forward, but the paper identifies a critical limitation: they evaluate physics in isolation from task context. A benchmark might ask "does this ball bounce realistically?" but it doesn't ask "did the robot successfully pick up the ball and place it in the basket?" The physical violations that matter most in embodied AI are not generic physics puzzles — they are violations that occur during task execution, such as a gripper failing to close around an object while the object nonetheless moves with the arm, or a robot skipping the "grasp" step entirely in a pick-and-place sequence. Existing physics benchmarks, the paper argues, "often rely on isolated physical constraints or local visual metrics, failing to capture the complex interplay between robotic actions and environmental responses."

Story and dynamics benchmarks (StoryBench, T2VBench, FETV) emphasize narrative coherence rather than physical interaction. These benchmarks evaluate whether models can maintain consistent characters, follow event sequences, and produce logically ordered narratives over long time horizons. While long-range coherence is certainly relevant to robotics (where tasks unfold over multiple sequential steps), these benchmarks focus on storytelling coherence — does character A appear consistently across scenes? — rather than physical coherence — does the robot's end-effector trajectory respect joint limits and contact constraints?

The result of this gap is what the paper calls "overly optimistic conclusions, where high scores are assigned even to videos containing unnatural movements or incomplete tasks." A model might be celebrated for its video generation capabilities based on perceptual benchmarks, while producing outputs that are fundamentally unusable for embodied AI applications. The field lacked a benchmark that asks the questions that matter for robotics: Did the task get completed? Are the actions physically plausible? Is the robot's morphology stable over time?

The Data Scarcity Problem Underneath the Evaluation Problem

The paper argues that the evaluation gap is not merely a measurement issue — it reflects a deeper data scarcity problem that has constrained the entire field of video-based robot learning. While the benchmark provides the diagnostic tool, the underlying disease is the lack of large-scale, diverse, high-quality training data for robotic video generation.

The paper provides context for why this is the case, categorizing existing robot learning datasets into three types (Section 2.2):

Real-world robot data (Open X-Embodiment, DROID, Agibot World, BridgeData) is expensive and narrow. These datasets are collected through teleoperation — human operators controlling robots to perform tasks while cameras record the interactions. The cost structure is brutal: each hour of robot data requires an hour of skilled human operator time, plus the capital cost of the robot hardware and the physical space. The result is that even the largest real-world datasets (Open X-Embodiment at ~1.4M trajectories, Agibot World at ~1M trajectories) are orders of magnitude smaller than the web-scale image and text datasets that power foundation models in vision and language. Moreover, most real-world datasets are collected on a small number of robot platforms in constrained laboratory environments, producing narrow distributions over robot morphologies, tasks, objects, and environments. The paper notes that these datasets "predominantly focus on similar types of robots, resulting in issues of limited diversity and restricted environments."

Human video data (Ego4D, Something-Something, EPIC-Kitchens) lacks robot embodiment. Large-scale egocentric video datasets capture humans performing everyday tasks, and there is a growing body of work attempting to transfer human manipulation knowledge to robots. However, the embodiment gap is significant: human hands have different kinematics, degrees of freedom, and contact dynamics than robot grippers. A model trained exclusively on human videos may learn to generate realistic hand movements but struggle with the mechanical constraints of a parallel-jaw gripper or the joint limits of a 7-DOF arm.

Synthetic robot data (MimicGen, RoboCasa, RoboGen) can be diverse but suffers from the simulation-to-reality gap. Simulation-based data generation can produce unlimited quantities of labeled robot trajectories with perfect ground truth, but simulators make approximations to physics, rendering, and sensor modeling that don't perfectly match reality. The paper doesn't dwell on this category extensively, but it's part of the landscape that makes real-world video data valuable despite its scarcity.

Cross-dataset incompatibility compounds the problem. The paper highlights a practical barrier: "inconsistent data collection and storage methods across different datasets make it difficult to enable effective cross-dataset co-training." Different datasets use different camera setups, different control interfaces, different data formats, and different annotation schemes. A video model that attempts to train across multiple existing robot datasets faces a data engineering challenge before it ever gets to the modeling challenge.

The consequence is that video generation models, when applied to robotics, are either trained on insufficiently diverse data (leading to poor generalization across embodiments and tasks) or trained primarily on non-robot data (leading to the physical understanding failures that RBench reveals). The paper's dual contribution — a benchmark to measure the problem and a dataset to address it — is motivated by this diagnosis.

How the Paper Positions Itself

The paper positions RBench and RoVid-X as a synergistic ecosystem for advancing video-based embodied intelligence. The authors explicitly state this framing in the abstract: "Collectively, this synergistic ecosystem of evaluation and data establishes a robust foundation for rigorous assessment and scalable training of video models, accelerating the evolution of embodied AI toward general intelligence."

This is more than a rhetorical flourish — it reflects a specific thesis about what the field needs. The paper argues that evaluation and data are coupled problems that must be solved together. Without RBench, researchers lack the tools to identify which physical failures are most prevalent and where different model families fall short. Without RoVid-X, even those armed with diagnostic insights lack the training data to address the deficiencies they've identified.

The paper also positions itself within the broader trajectory of video generation research. It observes that video models are "evolving into unified foundation models for machine vision" and are "being increasingly used in robot learning and action prediction, as well as controllable simulators for synthesizing robotic video trajectories." This situates the work not as a niche robotics contribution but as addressing a gap that becomes increasingly critical as video models become more central to embodied AI pipelines. The paper draws an implicit analogy to how standardized benchmarks like ImageNet enabled the deep learning revolution in computer vision — RBench and RoVid-X aim to play a similar catalytic role for video-based robot learning.

A key element of the paper's positioning is its claim to be the first comprehensive benchmark specifically designed for robotic video generation (Section 1: "To the best of our knowledge, it is the first comprehensive benchmark with fine-grained metrics for robotic video generation"). The paper distinguishes this from general video benchmarks by emphasizing three design principles that are absent from prior work: (1) evaluation across multiple robot embodiments rather than generic scenes, (2) metrics that jointly assess physical plausibility and task completion rather than treating them independently, and (3) reproducible automated metrics that correlate with human judgment rather than relying on costly human evaluation or simplistic perceptual scores.

The paper also positions itself as filling a gap that existing robotics-specific video models have exposed. It evaluates models like Cosmos 2.5 (NVIDIA's world simulation model), DreamGen (a video-based robot policy model), and Vidar (an embodied video diffusion model), finding that even these purpose-built systems struggle significantly on the benchmark. This is a crucial point: the paper is not merely critiquing general-purpose video models for failing at robotics — it's showing that even models designed for robotics have substantial room for improvement, which validates the benchmark's difficulty and the dataset's necessity.

The Practical Stakes

To make the motivation concrete, consider what happens in the current state of practice. A research team wants to use video generation to augment their robot manipulation dataset. They take a state-of-the-art image-to-video model, condition it on frames from their robot's camera and text descriptions of desired tasks, and generate synthetic trajectory videos. They then use an inverse dynamics model to extract actions from these videos and train a policy. If the generated videos contain the failure modes that RBench is designed to detect — floating objects that move without contact, grippers that change shape mid-trajectory, objects that spontaneously appear or disappear — the extracted actions will encode these physical impossibilities, and the resulting policy will either fail in deployment or learn compensation strategies that don't generalize. The paper's contribution is to make these failure modes measurable (through RBench) and addressable (through RoVid-X), moving the field from anecdotal awareness of these problems to systematic diagnosis and treatment.

3. Technical Approach

This is primarily a systems and benchmark paper whose core idea is that evaluating video generation models for embodied robotics requires jointly measuring task completion and physical fidelity through a suite of automated VQA-style metrics, and that improving performance on these metrics demands a purpose-built dataset with rich physical property annotations that existing robot datasets lack.

3.1 Reader Orientation

The paper builds two interconnected artifacts: RBench, a diagnostic toolkit that automatically grades robot-generated videos on whether they complete their assigned tasks and obey basic physics, and RoVid-X, a 4-million-clip training dataset engineered to give video models the physical interaction priors they need to perform well on such evaluations. Together, they solve the problem that neither perceptual video benchmarks nor generic physics benchmarks ask the right questions about robot behavior — RBench defines those questions and RoVid-X provides the curriculum to answer them.

3.2 Big-Picture Architecture (Diagram in Words)

The system has two major components that function as a coupled evaluation-and-training ecosystem:

1. RBench (the evaluation infrastructure): a curated set of 650 image-text pairs spanning five task categories and four robot embodiments, paired with five automated metrics implemented via multimodal large language models (MLLMs) and low-level vision tools. These metrics ingest generated videos and produce scalar scores along two dimensions: task completion (did the robot do what was asked?) and visual quality (is the motion physically coherent?).

2. RoVid-X (the training data pipeline): a four-stage data processing workflow that ingests raw videos from internet platforms and 20+ open-source embodied datasets, filters for quality and relevance, segments into task-annotated clips with captions, and augments each clip with physical property annotations (optical flow, depth maps, super-resolution enhancement). The output is a unified, annotated dataset for training video generation models.

Information flows as follows: raw video sources → GPT-5-based content filtering → quality scoring and scene segmentation → video understanding model for task segmentation and captioning → physical property annotation (FlashVSR for resolution, AllTracker for optical flow, Video Depth Anything for depth) → unified RoVid-X dataset. Separately, curated evaluation samples are constructed from non-overlapping sources → MLLM-based metric computation → aggregated task completion and visual quality scores → model rankings. The two pipelines interact when RoVid-X-trained models are evaluated on RBench (Section 5.4).

3.3 Roadmap for the Deep Dive

• First, the RBench benchmark construction (Section 3.1 of the paper), including how evaluation samples are selected, how tasks and embodiments are organized, and how data leakage is prevented — since the design of the evaluation set determines what the metrics measure and what conclusions the benchmark supports.

• Second, the five automated metrics (Section 3.2 and Appendix B), explained in detail with their mathematical formulations, because these metrics are the core technical contribution that distinguishes RBench from prior work — they operationalize "physical plausibility" and "task adherence" as computable scores.

• Third, the RoVid-X dataset construction pipeline (Section 4.1), walking through each of the four stages and the specific tools and models used, because the dataset's value depends on the quality controls and annotation richness built into each stage.

• Fourth, the dataset analysis and statistics (Section 4.2), which characterize what RoVid-X actually contains and how it compares to existing datasets, because the claimed benefits — diversity, scale, physical annotations — must be substantiated.

3.4 Detailed, Sentence-Based Technical Breakdown

RBench Evaluation Set Construction

RBench is constructed as a curated collection of 650 evaluation samples, each consisting of an initial reference image (a keyframe extracted from a high-quality robot video) and a text prompt describing the task the model should generate. The samples are organized along two orthogonal dimensions: task-oriented categories (250 samples) and embodiment-specific categories (400 samples).

Task-oriented categories (50 samples each). The paper defines five task families designed to stress-test different capabilities of video generation models:

• Common Manipulation: basic object interaction — grasping, placing, pushing, rotating, pressing. The evaluation focuses on whether the model can generate "physically plausible, temporally coherent, and natural manipulation behaviors" for everyday object handling. Scenes include single-arm, dual-arm, and humanoid robots.

• Long-Horizon Planning: multi-stage action sequences requiring the model to maintain causal and temporal coherence across sub-actions. Examples include sequences like "open the refrigerator door, take out the green box, close the door" and physical motion sequences like "perform a front flip, land steadily, lean forward and balance upside down on front legs." The evaluation assesses "action decomposition, stage transitions, and cross-time reasoning."

• Multi-Entity Collaboration: scenarios with a Primary Entity (any robot type) and a Secondary Entity (human, animal, or another robot) that must interact. Examples include object handover, dressing assistance, collaborative task completion, following, and guidance. The evaluation assesses whether the model can generate "natural, temporally coherent, and task-consistent multi-entity collaboration behaviors."

• Spatial Relationship: tasks requiring correct relative positioning — above/below, left/right, front/behind. The evaluation assesses "whether the model can generate physically plausible, temporally coherent, and natural manipulation behaviors that achieve the specified goals" while maintaining correct spatial layouts and motion trajectories.

• Visual Reasoning: tasks requiring fine-grained visual-semantic understanding — color recognition (e.g., "pick up the sky-blue book"), numerical ordering, attribute matching, geometric understanding (e.g., "the tallest orange object"), text reading (e.g., a dispenser labeled "Jasmine Tea"), and visual feature recognition (e.g., "the book with a portrait of a person on its cover"). The evaluation assesses "fine-grained visual grounding, logical consistency, and the ability to align robot actions with high-level visual-semantic reasoning requirements."

Embodiment-specific categories (100 samples each). The paper defines four embodiment types to capture the substantially different morphologies, degrees of freedom, and control modes that video models must handle:

• Dual-arm robots: emphasizing coordinated bimanual manipulation.
• Humanoid robots: prioritizing tool use and natural full-body postures.
• Single-arm robots: focusing on precise object interactions.
• Quadruped robots: predominantly testing terrain adaptation and motion continuity.

Each subset includes diverse robot models, action types, manipulated objects, scene environments, and both first-person and third-person camera perspectives. The paper notes a deliberate design choice: "embodiment-aware datasets like RBench are pivotal for advancing video foundation models in robotics" because they reveal biases in model capabilities — for instance, models pre-trained on large-scale human activity datasets tend to perform better on humanoid scenarios while struggling with fine-grained single-arm manipulation.

Sample curation protocol. Reference images are keyframes extracted from high-quality videos sourced from public datasets or online sources. Each image is manually verified for accuracy. Critically, the paper ensures that "the selected videos in the evaluation set do not appear in the subsequent training database" — a data leakage prevention measure that prevents models trained on RoVid-X from having an unfair advantage on RBench. Additionally, "new task prompts for each reference image" are designed by human annotators to avoid content overlap with training data. All samples are "verified and filtered by human annotators to ensure that the generated task prompts align with realistic logic."

Metadata recording. For each sample, the paper records the manipulated object, embodiment type, and camera viewpoint (first-person or third-person) to support detailed analysis along these dimensions.

Automated Metrics: Overview and Design Philosophy

RBench evaluates generated videos using five fine-grained metrics that produce two aggregated indicators: Task Completion (TC) and Visual Quality (VQ) . The metrics are implemented through a combination of multimodal large language model (MLLM) evaluation (using open-source Qwen3-VL and closed-source GPT-5) and low-level computer vision computation. The paper selects these two MLLMs to provide both an open and a closed-source evaluation, following "previous practices" of using MLLMs as zero-shot evaluators.

The key design philosophy is that each metric targets a specific failure mode that standard perceptual metrics miss. Rather than asking a generic "is this video good?" question, each metric asks targeted questions about physical violations, task adherence, morphological stability, motion amplitude, and temporal smoothness. The metrics are designed to be reproducible — they use fixed prompts, fixed MLLM configurations, and deterministic computation where possible — so that different research groups can obtain consistent results when evaluating the same models.

Score aggregation. The five fine-grained metrics (denoted as PSS, TAC, RSS, MS, and MA — defined below) are normalized to [0, 1] using:

$s \leftarrow \text{clip}_{[0,1]}\left(\frac{s - s_{\min}}{s_{\max} - s_{\min}}\right)$

where $s$ is a raw metric value, $s_{\min}$ is the minimum possible value for that metric, and $s_{\max}$ is the maximum possible value.

The normalization maps each raw score linearly into the unit interval and then clips to enforce the bounds. This makes scores from different metrics comparable and prevents any single metric from dominating the aggregate.

Task Completion aggregates Physical-Semantic Plausibility and Task-Adherence Consistency:

$\text{TC} = \frac{\text{PSS} + \text{TAC}}{2}$

Visual Quality aggregates Robot-Subject Stability, Motion Smoothness, and two penalty terms:

$\text{VQ} = \max\left(0, 0.8 \cdot \text{RSS} + 0.2 \cdot \text{MS} - P_{\text{MA}}(\text{MA}) - P_{\text{RSS}}\right)$

where $P_{\text{MA}}(\text{MA})$ is a motion-amplitude penalty applied when the robot barely moves, and $P_{\text{RSS}}$ is a stability-consistency penalty based on robot and object stability grades. The weights (0.8 for RSS, 0.2 for MS) encode the paper's judgment that morphological stability is more critical for robotic video quality than temporal smoothness.

The penalties ensure that videos with insufficient motion or unstable visual composition are penalized even if other quality scores are high — a design choice targeting the specific failure mode where models generate visually smooth but essentially static videos.

Model-level aggregation. For each model, TC and VQ are computed for all evaluation samples. The final model score is the mean of TC and VQ across all samples. This averaging across samples and across the two dimensions produces the single "Avg." column in Table 2 that determines model rankings.

---

Metric 1: Physical-Semantic Plausibility (PSS)

This metric targets physical and semantic violations that "standard perception scores often miss." The core mechanism is a VQA-style protocol using MLLMs, where the model inspects a temporal grid of uniformly sampled key frames from the generated video and answers targeted questions about physical anomalies.

Input representation. Rather than processing the entire video, the system constructs a grid image composed of key frames sampled at uniform temporal intervals. This grid-based input is a practical compromise: it gives the MLLM access to temporal information without requiring video-native processing (which many MLLMs don't support well), while reducing the token count compared to processing every frame individually.

Targeted violation categories. The MLLM is prompted to detect four specific types of violations:

• Floating/Penetration: parts of the robot or objects that are not grounded on any surface or that interpenetrate with solid objects. The paper provides concrete examples in Appendix B, Figure 6 (a metallic spoon and a single-arm robot's manipulator suspended in mid-air without support) and Figure 7 (a humanoid robot hand "unrealistically penetrates the box, indicating a severe violation of rigid-body constraints").

• Spontaneous emergence: entities that appear, disappear, or duplicate without causal motion. Appendix B, Figure 8 shows three sub-examples: (a) a robotic arm suddenly disappearing in later frames, (b) human hands and a new notebook suddenly appearing, and (c) instant noodle packs being spuriously duplicated.

• Non-contact attachment/Incorrect grasp: objects that move rigidly with the robot without visible contact or with improper gripper closure. Figure 6(a) illustrates a metallic spoon moving with the gripper even though "there is no clear contact or gripper closure, resulting in an unrealistic 'sticking' effect."

• General physical implausibility: beyond these specific categories, the evaluator is required to assess "whether the overall action sequence and causal progression are reasonable."

What this metric captures that others miss. Standard perceptual metrics like FID, FVD, or CLIP-based similarity scores measure distributional similarity or semantic alignment but cannot detect that a spoon is floating in mid-air — the pixels might be sharp, the colors natural, and the text description well-matched, while the physics is completely wrong. PSS is specifically designed to catch these "severe physical violations that significantly reduce the credibility of the generated video."

Why VQA-style evaluation? The paper adopts MLLM-based evaluation rather than training specialized detectors for each violation type because the violations are diverse, context-dependent, and hard to enumerate exhaustively. A floating spoon looks different from a floating robot arm; interpenetration between a gripper and a box looks different from interpenetration between a quadruped's leg and the ground. The MLLM's general visual reasoning capability is leveraged to handle this diversity without requiring per-violation training data.

---

Metric 2: Task-Adherence Consistency (TAC)

This metric evaluates whether the generated video aligns with the intent and sequence defined by the text prompt. While PSS checks physical plausibility, TAC checks whether the right thing happened — even a physically perfect video of a robot moving its arm is wrong if the prompt asked it to pick up a specific object.

Core mechanism. Like PSS, TAC uses a temporal grid of key frames evaluated by an MLLM via a VQA checklist. The checklist covers two dimensions:

• Task responsiveness: whether the goal state is reached without premature interruption. The MLLM checks whether the robot actually attempts to do what was asked, or whether it remains static or performs an unrelated action.

• Key actions: whether all required actions specified in the prompt (e.g., grasp, place, open, close) occur and align with the prompt in the correct order.

Task-specific instantiations. The paper tailors the TAC evaluation criteria to each of the five task families, because what counts as "task adherence" differs fundamentally across them:

For Common Manipulation, TAC is assessed through:

• Task Completion: whether the robot successfully accomplishes the manipulation objective while exhibiting "reasonable intermediate phases (e.g., approach → grasp → move → place)."
• Action Effectiveness: whether the manipulation is physically plausible, including "natural gripper closure, appropriate contact locations, and smooth trajectories." Attempts with "discontinuous, incomplete, or physically implausible actions" are treated as failures.

For Multi-Entity Collaboration, TAC is assessed through:

• Task Completion: whether both entities execute their respective roles and complete all required interaction steps in a "temporally coherent and logically consistent manner."
• Action Effectiveness: for contact-based interactions, whether a full sequence of "approach → contact → release/transfer" is produced; for non-contact interactions, whether a coherent process of "initiation → alignment → sustained coordination" is produced. Missing stages, asynchronous responses, or logically inconsistent behaviors are treated as unsuccessful.

For Spatial Relationship, TAC is assessed through:

• Spatial Relation Accuracy: whether spatial relations between entities (e.g., above/below, left/right) match the textual description with "consistent orientation, scale, and viewpoint."
• Manipulation Feasibility: whether the direction, trajectory, and intent of the robot's motion are compatible with the described spatial relations. Trajectories that contradict the described direction or result in physically unreasonable motions are incorrect.

For Visual Reasoning, TAC is assessed through:

• Visual Reasoning Accuracy: evaluated via an automatic Question Chain mechanism. The MLLM first generates a set of stepwise verification questions covering the trigger-feedback-outcome logic of the prompt, then answers those questions based on the generated video. The score is computed as:

$\text{Score} = 5 \times \frac{\text{completed questions}}{\text{total questions}}$

where completed questions count those where the MLLM's answer confirms that the expected visual or logical condition holds. Missing or incorrect events are treated as unfulfilled steps. The multiplication by 5 maps the score to a 0–5 scale.

• Action Effectiveness: physical plausibility and dynamic coherence of the robot's motions, penalizing "clearly discontinuous, incomplete, or physically implausible actions even if some high-level reasoning appears correct."

For Long-Horizon Planning, TAC is assessed through:

• Event Completion Rate: for a sample with an ordered event list (e.g., "1. open the refrigerator door; 2. take out the green box; 3. close the door"), the score is computed as:

$\text{Score} = 5 \times \frac{\text{completed events}}{\text{total events}}$

This measures "how completely the required event sequence is executed." The multiplication by 5 again normalizes to a 0–5 range.

• Action Effectiveness: physical plausibility and temporal coherence of the underlying motions, ensuring that "partially correct high-level event ordering without valid execution is not over-rewarded."

Why task-specific criteria? The paper's key insight here is that a single "task adherence" question would be too vague for an MLLM to answer reliably. By decomposing task adherence into task-family-specific sub-questions with explicit checklists and scoring rubrics, the evaluation becomes more reproducible and discriminative. The Question Chain mechanism for Visual Reasoning is particularly notable: rather than asking "did the robot reason correctly?", the MLLM is asked to generate and then verify a structured set of binary verification questions, making the evaluation process transparent and auditable.

What this metric captures that others miss. A video where the robot approaches an object, hovers near it, and then the object teleports into a basket would score well on perceptual quality (smooth motion, sharp frames) and might even pass basic physics checks (no floating or interpenetration). TAC would catch that the "grasp" action was never performed — a task-level failure that makes the video useless for training a robot policy.

---

Metric 3: Motion Amplitude (MA)

This metric measures whether the robotic subject actually moves meaningfully, distinguishing genuine robot articulation from apparent motion caused by camera movement. The motivation is a specific failure mode: "a common failure mode in robotic video generation is that the robot remains nearly static while the generated frames appear visually smooth." Pure smoothness-based metrics would score such videos highly, penalizing only jittery motion while ignoring the absence of motion.

Pipeline. The metric follows a four-stage computational pipeline adapted from VMBench:

Stage 1: Robot localization and tracking. The robot is first localized using GroundingDINO (an open-set object detection model), and temporally stable segmentation masks are obtained via SAM2 (Segment Anything Model 2). CoTracker (a dense point tracking model) is then used to track a dense grid of keypoints inside the robot mask. The critical design choice is that tracking is constrained to the robot mask, ensuring that "the estimated motion truly reflects robot articulation rather than background drift or mask leakage."

Stage 2: Frame-level motion computation. Let $p_{t,k}$ denote the 2D location of the $k$-th tracked point at frame $t$. The raw frame-to-frame displacement is:

$\bar{D}_t = \frac{1}{K} \sum_{k=1}^{K} \|p_{t,k} - p_{t-1,k}\|_2$

where $K$ is the number of tracked keypoints and $\|\cdot\|_2$ is the Euclidean distance.

The motion is then normalized by the video diagonal to ensure consistency across resolutions:

$\tilde{D}_t = \frac{\bar{D}_t}{\sqrt{W^2 + H^2}}$

where $W$ and $H$ are the video width and height in pixels.

What it computes: the average Euclidean displacement of all tracked points on the robot between consecutive frames, expressed as a fraction of the video's diagonal length. A value of 0.01 means the robot moved by about 1% of the screen diagonal from one frame to the next; a value of 0.5 means it moved by half the screen diagonal.

Why normalize by diagonal: pixel displacements mean different things at different resolutions — 50 pixels is a large movement in a 256×256 video but negligible in a 1920×1080 video. Normalizing by the diagonal makes the metric resolution-independent.

Stage 3: Camera-motion compensation. To estimate camera-induced movement, the robot mask is inverted and the same tracking procedure is applied to the background region. Let $\tilde{D}_t^{\text{bg}}$ denote the normalized background motion. A "soft-zero" strategy is applied:

$\hat{D}_t = \begin{cases} \tilde{D}_t - \tilde{D}_t^{\text{bg}}, & \tilde{D}_t > \tilde{D}_t^{\text{bg}} \\ \tilde{D}_t, & \tilde{D}_t \leq \tilde{D}_t^{\text{bg}} \end{cases}$

What it computes: if the robot's apparent motion exceeds the background motion (camera movement), the excess is attributed to actual robot articulation; otherwise, the full robot motion is retained (the "soft-zero" approach). This avoids zeroing out genuine small robot motions that happen to coincide with camera movement.

Why this form: simply subtracting the background motion ($\tilde{D}_t - \tilde{D}_t^{\text{bg}}$) would be correct if robot and camera motion were independent, but could produce negative values when tracking noise or partial occlusion makes the background appear to move more than the robot. The soft-zero approach is more robust: it "retains the small residual value" rather than forcing a zero, which "improves robustness against tracking noise or partial occlusion, while effectively treating the robot as 'static' when appropriate."

Stage 4: Final score computation. The compensated displacement is clipped to stabilize extreme values, and the Motion Amplitude Score is:

$\text{MAS} = \frac{1}{T} \sum_{t=1}^{T} \min(\hat{D}_t, 1)$

where $T$ is the total number of frames.

What it computes: the temporal average of the per-frame compensated robot displacement, clipped at 1.0 to prevent any single large displacement from dominating the average. A lower MAS indicates insufficient subject motion.

Why clip at 1: a displacement larger than the screen diagonal (mathematically possible if the robot moves dramatically while the camera pans) would inflate the average. Clipping ensures the score remains in a well-behaved range where the primary signal is "how much of the time is the robot moving?" rather than "how extreme was the single largest movement?"

Design rationale. MAS complements smoothness-based metrics by revealing "smooth-but-inactive failure modes" — videos where the background shifts slightly due to camera simulation but the robot itself is essentially frozen. This is a common artifact in video generation models that learn to produce visually coherent but temporally minimal outputs to avoid generating unnatural motion artifacts.

---

Metric 4: Robot-Subject Stability (RSS)

This metric assesses whether the robot's morphology and the target object's appearance remain consistent over time. The failure modes it targets include: gripper/hand shape drifting into non-mechanical forms, extra or missing manipulators, changes in link length or topology, joint inversion, object misidentification or attribute drift (class, color, position), and impossible deformation of rigid items.

Core mechanism. RSS uses a contrastive VQA setup based on MLLM. Unlike PSS and TAC which evaluate temporal sequences, RSS operates on pairs of frames: a reference frame and a generated frame. The MLLM is prompted to judge "how well the entity's appearance, structure, and semantics are preserved between the two frames."

Two sub-components. RSS separately evaluates:

• Robot structural stability: whether the robot maintains its embodiment type, link count, and morphology. Appendix B, Figure 10 provides concrete examples: (a) a humanoid robot degenerating into a single-arm robot, (b) a quadruped robot morphing into a small humanoid robot, (c) a single-arm robot transforming into a humanoid robot, and (d) a parallel gripper deforming into a dexterous robotic hand. These represent "structural drift and inconsistency in robot embodiment over time."

• Subject appearance stability: whether manipulated objects maintain their identity. Appendix B, Figure 11 shows (a) a rectangular knitted sleeve transforming into a long-sleeve sweater, and (b) a green plastic cup becoming a round yellow mug — both "indicating a loss of identity-preserving appearance."

Beyond these explicit examples, the paper notes a broader range of anomalies: changes in the number of robot links or arms during task execution, spontaneous generation of extra manipulators, unnatural variations in arm length or connectivity, changes in joint bending direction, and "unrealistic material changes, such as a rigid object bending like a deformable one."

Scoring. The MLLM assigns a stability grade to the robot and the manipulated object using a letter-grade scale (A through E, where A indicates perfect stability and E indicates severe degradation). These grades are mapped to penalty magnitudes $p(g) \in \{0.2, 0.4, 0.6, 0.8\}$ for grades B through E, with grade A incurring zero penalty. The stability-consistency penalty is:

$P_{\text{RSS}} = \begin{cases} \frac{p(g_r) + p(g_o)}{2}, & \text{if both exist} \\ p(g_r), & \text{if only } g_r \text{ exists} \\ 0, & \text{otherwise} \end{cases}$

where $g_r$ is the robot stability grade and $g_o$ is the object stability grade.

What it computes: the average penalty from robot and object stability degradation, ranging from 0 (perfect stability for both) to 0.8 (severe degradation for both). This penalty is subtracted from the Visual Quality aggregate.

Why a penalty rather than a direct score: the letter-grade mapping to penalties creates a non-linear relationship where stability issues are heavily penalized — a "B" grade already incurs a 0.2 penalty — reflecting the paper's judgment that even minor morphological drift is unacceptable for embodied AI applications. A smooth scoring function would under-penalize the transition from "slightly wrong" to "very wrong."

What this metric captures that others miss. Object permanence and morphological consistency are fundamental assumptions in robotics — a policy trained on video data assumes that a "gripper" in frame 1 is the same entity as the "gripper" in frame 30. If the video model spontaneously changes the gripper type or the number of fingers, the extracted actions will be physically meaningless. Standard video quality metrics (which measure per-frame or short-clip quality) cannot detect these long-range identity preservation failures.

---

Metric 5: Motion Smoothness (MS)

This metric evaluates the temporal continuity and naturalness of motion, targeting artifacts ranging from low-level frame-to-frame discontinuities (aliasing, flicker) to high-level motion blur and jitter. The paper notes that "various robot embodiments, including quadruped robots, humanoids, and single-arm manipulators, exhibit different degrees of motion-induced distortion that substantially degrade perceived video quality."

Core mechanism. MS is adapted from VMBench and uses Q-Align, a model that predicts aesthetic quality scores for images, as the perceptual quality estimator. The key insight is that temporal smoothness can be estimated by measuring how much the aesthetic quality fluctuates from frame to frame — a video with smooth, natural motion should have consistent per-frame quality, while one with temporal artifacts will show sharp quality drops at artifact frames.

Step 1: Per-frame quality scoring. For each video, frames are processed with a sliding window of size $w$ (default $w = 3$). Each window is fed into Q-Align to obtain a per-frame quality score sequence $\{Q_t\}_{t=1}^T$.

Step 2: Temporal fluctuation measurement. The magnitude of adjacent-frame quality differences is:

$\Delta Q_t = |Q_t - Q_{t-1}|, \quad t = 2, \ldots, T$

where $Q_t$ is the Q-Align aesthetic quality score for frame $t$, and $\Delta Q_t$ measures how much the perceptual quality changed from the previous frame.

Step 3: Adaptive thresholding. A critical design choice is that the threshold for flagging temporal anomalies is not fixed — it adapts to the video's motion amplitude $m$ (the MAS value from Metric 3). The paper defines a piecewise adaptive threshold function:

$\tau_s(m) = \begin{cases} 0.01, & m < 0.1 \\ 0.015, & 0.1 \leq m < 0.3 \\ 0.025, & 0.3 \leq m < 0.5 \\ 0.03, & m \geq 0.5 \end{cases}$

What it computes: a threshold on acceptable per-frame quality fluctuation that depends on how much the robot is actually moving. Low-motion videos get a strict threshold (0.01) because smooth videos shouldn't have quality fluctuations; high-motion videos get a relaxed threshold (0.03) because rapid legitimate motion can cause legitimate quality variations that shouldn't be penalized.

Why adaptive: a fixed threshold would either be too strict for high-motion videos (flagging natural motion blur as artifacts) or too loose for low-motion videos (missing subtle flickering). The piecewise function is "determined through grid search on a validation split to ensure reproducibility."

Step 4: Anomaly detection and scoring. A frame $t$ is marked as temporally abnormal if its quality fluctuation exceeds the adaptive threshold:

$I_t = \mathbb{I}[\Delta Q_t > \tau_s(m)]$

where $\mathbb{I}[\cdot]$ is the indicator function (1 if the condition holds, 0 otherwise). To "robustly capture abrupt artifacts such as frame drops or transient distortions," adjacent frames of each abnormal index are also flagged.

The Motion Smoothness Score is the proportion of normal frames:

$\text{MSS} = 1 - \frac{1}{T} \sum_{t=2}^{T} I_t$

What it computes: the fraction of frames that are not flagged as temporally abnormal. MSS = 1.0 means no frame had a quality fluctuation exceeding the adaptive threshold; MSS = 0.0 means every frame was flagged.

Why this form: the fraction-of-normal-frames formulation is simple, bounded, and interpretable. It directly answers the question "what proportion of the video is temporally smooth?" without requiring modeling of the magnitude of quality fluctuations (which would be harder to calibrate across different Q-Align versions or video domains).

Design rationale. MS complements MAS: MAS checks whether there is motion, while MS checks whether the motion that exists is smooth. Together they prevent the "smooth but static" failure mode (high MS, low MAS) and the "active but jittery" failure mode (high MAS, low MS) from being conflated with genuine high-quality motion (high MS, high MAS).

---

Motion-Amplitude Penalty

A penalty term is applied to down-weight videos with insufficient subject motion in the Visual Quality aggregate:

$P_{\text{MA}}(\text{MA}) = \begin{cases} (t - \text{MA}) + \delta, & \text{MA} < t_{\text{low}} \\ t - \text{MA}, & t_{\text{low}} \leq \text{MA} < t \\ 0, & \text{MA} \geq t \end{cases}$

where $t = 0.1$ is the primary threshold, $t_{\text{low}} = 0.05$ is a lower threshold, and $\delta = 0.1$ is an additional penalty for extremely low motion.

What it computes: when the normalized motion amplitude (MA) falls below 0.1, a penalty equal to the shortfall is applied. When MA falls below 0.05, an additional 0.1 penalty is added. When MA is above 0.1, no penalty is applied.

Why this form: the piecewise structure with a steeper penalty below $t_{\text{low}}$ creates a "cliff" that strongly penalizes essentially static videos while being more lenient on videos with some but insufficient motion. This reflects the practical reality that a video with zero robot motion is completely useless (worse than a video with minimal motion), and the scoring should reflect that non-linearity.

---

RoVid-X Dataset Construction Pipeline

The dataset construction is a four-stage pipeline designed to transform heterogeneous raw video sources into a unified, annotated dataset suitable for training video generation models on embodied robotics tasks.

Stage 1: Robot Video Collection. Raw robotic videos are collected from two source types: large-scale internet video platforms and "over 20 open-source embodied video datasets." These datasets "cover a variety of robot types and task scenarios, ensuring the breadth and diversity of the data."

The initial filtering is performed by the GPT-5 model, which is used to "automatically filter the content of each video and remove low-quality or irrelevant video clips that do not align with the research objectives." GPT-5 identifies videos related to robotic tasks and actions based on visual content and subtitles. After this filtering, approximately 3 million raw robotic video clips are identified, "covering different actions, tasks, and robot types."

Design choice: why GPT-5 for filtering? The scale of raw video collection makes manual filtering infeasible. Using an MLLM for initial content filtering is a practical compromise: GPT-5 can make coarse relevance judgments (robot vs. non-robot, task-relevant vs. irrelevant) at scale, reducing the human annotation burden to verification and refinement rather than primary filtering.

Stage 2: Video Quality Filtering. This stage applies a rigorous filtering procedure to remove low-quality and irrelevant clips. The process has two sub-steps:

First, scene segmentation detection is applied to "remove all video data unrelated to robots." The paper doesn't specify the exact detection method, but the intent is to filter out clips that passed the coarse GPT-5 filter but don't actually contain robot content — false positives from the initial automated filtering.

Second, a video quality scoring system assesses videos from multiple dimensions: clarity, dynamic effects, aesthetic performance, and optical character recognition (OCR), among other metrics. Each video clip is assigned a quality score based on these criteria, and "videos retained in the final dataset meet high-quality standards."

Design choice: multi-dimensional quality scoring. Rather than using a single quality metric (e.g., resolution or compression artifacts), the scoring system considers multiple dimensions including aesthetic performance and OCR. The inclusion of OCR is notable — it suggests that text legibility (e.g., labels on objects, text on robot interfaces) is considered important for robot video quality, perhaps because text provides semantic grounding that video models need to learn.

Stage 3: Task Segmentation and Captioning. This stage transforms raw video clips into task-annotated segments with text descriptions. The process uses "a video understanding model [Seed1.5-VL] and a specially designed prompt template to automatically analyze the robot actions within the videos."

The segmentation and captioning follows a structured procedure:

• Action identification: the system "identifies all dynamic actions within the video and excludes static scenes or irrelevant actions (e.g., waiting or remaining still)." The time range for each action (start and end times) is "precisely labeled to ensure accuracy."

• Action description: using the MLLM model, textual descriptions of each action are automatically generated, including three key elements: the action subject (e.g., "right arm" or "left gripper"), the object being manipulated (e.g., "nameplate" or "box"), and the specific operation details (e.g., "grasp and move" or "remove from the table").

• Standardized output: subtitles for each task segment are output in a standardized format, "ensuring that the action descriptions for each video clip are clear, concise, and aligned with the task requirements."

Design choice: structured captioning with subject-object-action triplets. Rather than generating free-form captions (which can be verbose, inconsistent, or miss key elements), the system produces structured descriptions that explicitly identify the acting entity, the target object, and the operation. This structured format is likely beneficial for training video generation models because it provides consistent conditioning signals — the model can learn to associate specific subject-object-action combinations with specific visual patterns.

Stage 4: Physical Property Annotation. This stage augments videos with physical property annotations designed to provide "more precise reference data, aiding in the training and evaluation of robot video generation models and offering richer physical data support for future research." Three types of annotations are applied:

• Resolution enhancement: FlashVSR is used to "improve the video resolution, making the images clearer and enhancing the details of the actions." FlashVSR is a diffusion-based streaming video super-resolution model.

• Optical flow annotation: AllTracker is used to "annotate a unified optical flow for the subjects in the videos, ensuring consistency in tracking and recording robot actions across different scenes." AllTracker is a dense point tracking model that can handle high-resolution inputs.

• Depth map generation: Video Depth Anything is used to "generate relative depth maps to accurately describe the spatial relationships and depth information of objects in the scene." Video Depth Anything is a video-specific depth estimation model that produces temporally consistent depth maps.

Design choice: why these three annotations? The paper selects annotations that directly address known failure modes in video generation. Resolution enhancement compensates for the low resolution of many source videos (as shown in Table 1, many existing datasets are at 480P or lower). Optical flow provides explicit motion information that can help models learn physically plausible trajectories rather than visually smooth but physically meaningless motion. Depth maps provide spatial layout information that can help models learn correct object relationships (preventing floating and penetration artifacts). The combination addresses the three dimensions of visual quality, motion quality, and spatial understanding that RBench evaluates.

RoVid-X Dataset Analysis

The resulting dataset, RoVid-X, contains 4 million robotic video clips spanning thousands of tasks and robot types. The paper provides statistical characterization through Figure 4(b), which illustrates the variety in action skills, task types, and interaction objects.

Comparison to existing datasets. Table 1 provides a systematic comparison across representative robotic video datasets along seven dimensions: year of release, number of videos, number of skills, resolution, optical flow availability, diverse robotic forms, and diverse captions.

RoVid-X is the only dataset that satisfies all seven criteria simultaneously:

• Scale: 4M videos, exceeding Open X-Embodiment (1.4M) and Agibot World (1M).
• Skill diversity: 1300+ skills, far exceeding the next largest (Humanoid Everyday at 221).
• Resolution: 720P, matching or exceeding most datasets (many are at 480P or lower).
• Optical flow: the only dataset providing this annotation.
• Diverse robotic forms: covering multiple robot morphologies.
• Diverse captions: providing structured text descriptions.

A critical qualitative distinction is that existing datasets "predominantly focus on similar types of robots, resulting in issues of limited diversity and restricted environments," while RoVid-X is specifically designed to span "various robot morphologies and entities" with "a unified set of physical attributes for all data sources, thereby advancing cross-entity research in robot learning."

What the statistics reveal. The dataset's breadth — 1300+ skills, thousands of tasks, multiple robot types — is designed to address the generalization challenge identified in the benchmark results: models fine-tuned on narrow robot-specific datasets (like Vidar and UnifoLM) perform poorly on RBench because they lack "World Knowledge provided by large-scale pretraining." RoVid-X aims to provide that missing breadth while maintaining robot-specific physical grounding.

The resolution enhancement from potentially lower-resolution sources to 720P is notable: many source datasets (RoboNet at 240P, portions of Open X-Embodiment at 64P-720P) would produce training data that is too low-resolution for modern video generation models to learn fine-grained manipulation details. FlashVSR upscaling is therefore not merely cosmetic — it's a practical necessity for creating a training dataset at the resolution modern video models expect.

Evaluation Setups for Benchmark Experiments

The paper evaluates 25 models across three families. The implementation details are important for reproducibility:

Generation protocol. All open-source models generate videos using their "official default configurations to ensure consistency with the model's preset settings." Closed-source models use "official APIs, strictly following the methods recommended by the developers for invoking and using the models." For each image-text pair in the benchmark, three videos are generated per model sample and the scores are averaged to minimize variance from stochastic generation.

Model configurations. The paper provides model-specific generation parameters (Appendix C):

• Wan2.2_A14B: 5 seconds, 1280×720, 16 fps, 81 frames
• Wan2.2_5B: 5 seconds, 1248×704, 24 fps, 120 frames
• Wan2.1_14B: 5 seconds, 832×480, 16 fps, 81 frames
• HunyuanVideo: 5 seconds, 1248×704, 24 fps
• HunyuanVideo 1.5: 5 seconds, 848×480, 24 fps
• CogVideoX-5B: 6 seconds, 720×480, 8 fps, 160 frames
• LTX-Video: 5 seconds, 832×480, 16 fps
• LTX-2: 5 seconds, 1536×1024, 24 fps
• SkyReels-V2: ~4 seconds, 960×544, 24 fps
• FramePack: 5 seconds, 832×480

The variation in resolution and frame rate across models is substantial (from 720×480 at 8 fps to 1536×1024 at 24 fps), which the paper acknowledges but doesn't attempt to control for — the evaluation treats each model at its intended operating point.

MLLM evaluators. The paper uses two MLLMs: open-source Qwen3-VL and closed-source GPT-5. Using both provides a robustness check — if the two evaluators produce similar rankings, the metrics are less likely to be artifacts of a particular MLLM's biases.

Finetuning experiments (Section 5.4). For validating RoVid-X, models initialized with Wan2.1 14B and Wan2.2 5B weights are finetuned using MSE loss exclusively on 200k randomly sampled instances from RoVid-X. The paper notes that this is a subset due to "computational constraints," meaning the full 4M dataset's potential impact is not directly measured — the reported improvements (Table 4) represent a lower bound on what full-dataset training could achieve.

Summary of Key Design Choices and Their Justifications

• VQA-style MLLM evaluation over specialized detectors: MLLMs provide general visual reasoning that can handle diverse, context-dependent violation types without requiring per-violation training data. The high Spearman correlation (0.96) with human judgments validates this choice.

• Task-specific criteria for TAC: a single generic "task adherence" question would be too vague for reliable MLLM evaluation. Decomposing into task-family-specific sub-questions with explicit checklists makes the evaluation reproducible and discriminative.

• Adaptive thresholding for motion smoothness: a fixed threshold would either penalize legitimate high-motion videos or fail to catch subtle artifacts in low-motion videos. The piecewise function adapts to the video's actual motion content.

• Penalty-based aggregation for VQ: rather than a weighted sum, penalties ensure that specific failure modes (static robot, morphological instability) can sink the VQ score regardless of other quality dimensions — reflecting the practical reality that these failures make videos unusable for embodied AI.

• Four-stage data pipeline with physical annotations: raw collection → filtering → captioning → physical annotation produces a dataset that addresses all the failure modes RBench measures: resolution enhancement for clarity, optical flow for motion quality, depth for spatial understanding.

• GPT-5 for initial content filtering: enables scale — manual filtering of millions of raw clips would be prohibitively expensive, and GPT-5 provides a reasonable coarse filter that can be refined by subsequent automated quality scoring.

• Non-overlapping evaluation and training data: the paper explicitly ensures that evaluation samples don't appear in RoVid-X, preventing inflated benchmark scores from memorization.

• Three-generation averaging: generating three videos per sample and averaging scores reduces variance from stochastic generation, making model comparisons more reliable without excessive compute cost (since 3 × 650 = 1950 generations per model is manageable).

4. Key Insights and Innovations

Innovation 1: Task-Level Correctness as a First-Class Evaluation Dimension for Video Generation

The paper's most fundamental conceptual move is elevating task completion from an afterthought to a co-equal dimension with visual quality in video generation evaluation. This is not merely "adding more metrics" — it redefines what it means for a generated video to be "good" in embodied contexts.

Prior to this work, the dominant paradigm for video generation evaluation treated perceptual quality as the primary (often sole) axis of assessment. Benchmarks like VBench, EvalCrafter, and Video-Bench measure frame clarity, texture fidelity, motion smoothness, and text-video semantic alignment — all dimensions of how visually appealing the video is. Even physics-focused benchmarks like VideoPhy and PhyBench, which introduced physical plausibility checks, treated physics as an attribute of the video's appearance (does the ball bounce realistically?) rather than as a constraint on task execution (did the robot successfully pick up the ball and place it in the basket?).

The paper's innovation is to argue — and demonstrate empirically — that task completion and visual quality are orthogonal dimensions that can conflict. A video can be perceptually flawless while being task-incorrect (the robot moves smoothly but never grasps the object), and a video can be task-correct while being visually degraded (the robot completes the action but with jittery motion). Standard benchmarks conflate these dimensions; RBench disentangles them.

This is a fundamentally different diagnostic stance. Rather than asking "how good does this video look?", RBench asks "would a robot policy trained on this video learn the right thing?" The five task-oriented categories in RBench — Common Manipulation, Long-Horizon Planning, Multi-Entity Collaboration, Spatial Relationship, and Visual Reasoning — are not arbitrary. They correspond to capabilities that embodied AI systems need: basic manipulation skills, sequential planning, multi-agent coordination, spatial understanding, and visual-semantic reasoning. Evaluating video models on these dimensions tests whether they can serve as effective world simulators, not just as media generators.

The significance of this reframing extends beyond benchmarking. By making task completion measurable and reproducible, the paper creates a feedback signal that can guide model development. Prior work could only say "this model generates visually appealing videos"; now researchers can say "this model achieves 0.531 on Visual Reasoning but only 0.364 on Multi-Entity Collaboration," which directly pinpoints capability gaps. This is analogous to how the GLUE and SuperGLUE benchmarks transformed NLP evaluation from perplexity-based metrics to task-specific capability assessment — the paper aims to do the same for video generation in embodied contexts.

The empirical evidence for the orthogonality of task and visual quality comes from the model rankings in Table 2. Some models (like Wan 2.6 at Rank 1) score highly on both dimensions, but the relationship is not monotonic. The consumer-oriented Sora series demonstrates the conflict most starkly: Sora v2 Pro ranks 17th overall (Avg 0.362) despite being a highly capable creative video generator, because its task completion scores on robotic manipulation are poor (0.208 on Common Manipulation, 0.186 on Multi-Entity). The paper's qualitative results (Figure 5) make this concrete: in the Visual Reasoning task, Wan 2.5 "mistakenly identifies the woven basket as the hollow basket" — a task-level failure that perceptual metrics would never catch.

This is a fundamental shift in evaluation philosophy, not an incremental addition of metrics. It creates a new axis of assessment that the field had no way to measure before, and the strong human correlation (Spearman ρ = 0.96, Table 3) validates that this axis captures something perceptually meaningful, not just an arbitrary computational artifact.

---

Innovation 2: Difficulty-Conditioned Diagnosis Through Multi-Dimensional Failure Taxonomy

The paper's second conceptual innovation is that RBench doesn't just output a single aggregate score — it provides granular, dimension-specific failure diagnostics that reveal why models fail on particular tasks and embodiments, enabling targeted improvement rather than blind scaling.

This matters because the aggregate ranking (Table 2) hides enormous heterogeneity. The paper's analysis reveals that different model families fail for different reasons on different tasks, and these patterns are diagnostically rich:

The "Media-Simulation Gap" (Section 5.2.1). The paper identifies a specific failure mode where "models optimized for media consumption prioritize visual smoothness and cinematic transitions, often at the expense of physical fidelity and precise motion control." This is not merely an observation that consumer models underperform — it's a specific diagnostic that isolates why: the training objectives and data distributions that produce cinematic quality are orthogonal to those that produce physical correctness. Sora v2 Pro's poor performance (Rank 17) is not because it's a weak model overall, but because its optimization target (cinematic appeal) actively competes with the evaluation target (physical task execution).

The "Cognitive Gap" (Section 5.2.1). The paper identifies that even top-tier models show a sharp performance drop on Visual Reasoning compared to execution-oriented tasks. Wan 2.6 scores 0.655 on Spatial Relationship but only 0.531 on Visual Reasoning — a drop of 0.124 points. This pattern holds across model families, suggesting that current video generation architectures have a specific weakness in translating visual-semantic reasoning into physically grounded action. This is not a generic "models are bad at reasoning" claim; it's a specific diagnostic that suggests where architectural or training improvements are most needed.

The "Manipulation Gap" (Section 5.2.1). The paper finds that "models consistently score higher on coarse-grained locomotion tasks (Quadruped, Humanoid) than on fine-grained manipulation." The embodiment-specific scores in Table 2 bear this out: across all models, Quadruped and Humanoid scores are systematically higher than Single Arm and Dual Arm scores. This is a specific finding about the relative difficulty of generating different types of robot motion, and it has implications for training data composition — if models struggle with fine-grained contact dynamics, training data should emphasize manipulation-heavy scenarios.

The "Specialization Dilemma" (Section 5.2.1). The paper identifies a fundamental tension: robotics-specific models like Cosmos 2.5 perform better than general open-source models of comparable size, but "domain-specific data... cannot fully compensate for the deficit in 'World Knowledge' provided by large-scale pretraining." Conversely, models fine-tuned on very narrow robot data (Vidar, UnifoLM) rank at the bottom despite being purpose-built for robotics — they sacrifice too much general visual knowledge. This is not a simple "more data is better" finding; it's a specific diagnosis of the tradeoff between domain specialization and general pretraining, with practical implications for how to allocate training compute.

The innovation is that RBench provides the lens to see these patterns. Prior benchmarks would report that Model A scores 0.70 and Model B scores 0.65, with no insight into why or where the gap exists. RBench's task-level and embodiment-level breakdowns enable the kind of fine-grained failure analysis that the paper performs. This is a conceptual contribution to the practice of evaluation — it shifts benchmarking from a ranking exercise to a diagnostic tool.

The significance is amplified by the fact that these diagnostic patterns are not obvious a priori. One might have guessed that larger models would uniformly outperform smaller ones, or that robotics-specific training would uniformly help. The paper reveals that the relationship is more nuanced: robotics-specific training helps on some dimensions but hurts on others (the Specialization Dilemma), scaling helps but hits diminishing returns on specific capabilities (the Cognitive Gap), and high visual quality can actively mask task failures (the Media-Simulation Gap). These are actionable insights for researchers deciding how to allocate training resources, not just descriptive statistics.

---

Innovation 3: The Synergistic Ecosystem Framing — Evaluation and Training as Coupled Problems

The paper's third conceptual innovation is its argument that evaluation and training data are coupled problems that must be solved together in embodied video generation. This is more than a "we built a benchmark and a dataset" paper — it's a specific thesis about how progress happens in this domain.

The standard paper structure would treat the benchmark (RBench) and the dataset (RoVid-X) as independent contributions, each with its own motivation and evaluation. The paper instead frames them as a synergistic ecosystem where each component serves the other. RBench reveals what is broken (specific failure modes, diagnostic gaps); RoVid-X provides the training signal to fix those failures. The finetuning experiments in Table 4 close the loop: models trained on RoVid-X show improvements specifically on the dimensions that RBench measures (task completion across all five task categories, visual quality across all four embodiments).

This framing matters because it addresses a chicken-and-egg problem in the field. Without RBench, researchers can't tell whether their video models are physically competent, so they can't prioritize data collection to address specific failures. Without RoVid-X, even those armed with diagnostic insights from a benchmark like RBench lack the training data to fix the identified problems — existing datasets are too narrow in robot morphology, too low-resolution, or too small in scale. The paper argues that simultaneous investment in both evaluation infrastructure and training data is necessary to break this deadlock.

The intellectual move here is to recast the problem from "we need better video models" (a modeling problem) or "we need better benchmarks" (an evaluation problem) to "we need a coupled measurement-and-improvement cycle" (a systems problem). This is analogous to how ImageNet simultaneously provided a standardized benchmark AND a large-scale training dataset, and how the combination catalyzed the deep learning revolution. The paper is making an implicit bet that embodied video generation needs a similar catalytic pairing — and that RoVid-X is the necessary complement to RBench's diagnostic capability.

The evidence for the coupling claim comes from the finetuning experiments (Table 4). Models finetuned on RoVid-X show stable improvements across diverse dimensions: Wan2.1_14B improves from 0.399 to 0.446 (total score), with gains in every task category and every embodiment. The improvement is not limited to easy cases — the hardest task categories (Long-Horizon, Visual Reasoning) show some of the largest relative gains (from 0.335 to 0.389 on Long-Horizon, from 0.205 to 0.298 on Visual Reasoning for Wan2.1_14B). This suggests that the dataset's diversity and physical annotations are providing the training signal that general video pretraining misses.

However, this is the innovation where the paper's claims outrun its evidence most clearly. The finetuning experiments use only 200k samples (5% of the full dataset) on two specific base models (Wan2.1_14B and Wan2.2_5B). The paper does not demonstrate that full-dataset training produces proportionally larger gains, that the improvements transfer to other model architectures, or that the RBench-measured improvements translate to downstream task performance in real robot policy learning. The synergistic ecosystem framing is therefore a compelling hypothesis supported by initial evidence rather than a fully validated claim. The paper's contribution here is to articulate the coupled-problem thesis and provide preliminary validation, creating a research program that others can pursue.

---

Innovation 4: The Limitation of Pre-Trained Motion Priors for Robotic Physical Interaction

The paper surfaces a subtle but important finding through its embodiment-specific evaluation: pre-training on human-centric video data creates motion priors that are actively mismatched with the physical constraints of non-humanoid robots. This is not framed as a central claim in the paper, but it emerges as one of the most instructive patterns in the results and represents a conceptual contribution to understanding why video models fail on robotic tasks.

The evidence comes from comparing model performance across embodiments. The paper notes that "due to extensive pre-training on large-scale human activity datasets, many models tend to exhibit higher task completion rates and better visual quality in humanoid-robot scenarios, while they often struggle with fine-grained single-arm manipulation." Table 2's embodiment scores confirm this pattern: Humanoid scores are systematically higher than Single Arm scores across almost all models (e.g., Wan 2.6: 0.667 Humanoid vs. 0.666 Single Arm; the trend is even more pronounced in weaker models: CogVideoX-5B: 0.496 Humanoid vs. 0.338 Single Arm).

This is a deeper finding than "models are better at some embodiments than others." It reveals that the pre-training data distribution shapes the model's implicit physics engine. Models trained on internet-scale video data see orders of magnitude more examples of human movement than robot movement, and humanoid robots happen to share enough kinematic structure with humans (bipedal, two arms with hands, similar joint configurations) that the human motion prior transfers partially. Single-arm industrial manipulators, in contrast, have fundamentally different kinematics (fixed base, articulated joints, parallel grippers) with few analogs in the human video training distribution. The models have no strong prior for how a 6-DOF arm should move when picking up an object, so they resort to blending human-like arm motions with mechanical constraints they don't fully understand — producing the morphological drift and physical implausibility that RBench's metrics capture.

This insight has practical implications for training data strategy. It suggests that simply scaling up general video pre-training (the dominant paradigm) will asymptotically improve humanoid robot generation but leave a persistent gap for non-humanoid embodiments. The paper doesn't frame it this way explicitly, but the implication is that embodiment diversity in training data is not just nice to have — it's necessary to overcome the human-centric bias in pre-trained motion priors. RoVid-X's inclusion of quadruped, single-arm, and dual-arm data alongside humanoid data is therefore not merely about breadth; it's about providing counterexamples to the dominant motion mode that the model has internalized from internet-scale pretraining.

The reason this qualifies as an innovation rather than just an observation is that it provides a mechanistic explanation for the Media-Simulation Gap that the paper identifies. Consumer video models are bad at robotic physical interaction not because they're poorly trained per se, but because their training objective (realistic video generation) and training distribution (human-centric video) jointly optimize for a different kind of "realism" than the one robotics demands. The paper's contribution is to make this mismatch visible and measurable, turning what could have been a vague complaint ("video models don't understand physics") into a specific, diagnostic finding about the relationship between pre-training data and embodiment-specific physical accuracy.

5. Experimental Analysis

Evaluation Methodology

• Dataset. RBench consists of 650 image-text pairs, with 250 samples organized into five task-oriented categories (50 each: Common Manipulation, Long-Horizon Planning, Multi-Entity Collaboration, Spatial Relationship, Visual Reasoning) and 400 samples organized into four embodiment-specific categories (100 each: Dual-arm, Humanoid, Single-arm, Quadruped robots). Reference images are keyframes extracted from high-quality videos sourced from public datasets or online sources, manually verified for accuracy, and explicitly selected to avoid overlap with the RoVid-X training database. Text prompts are designed and verified by human annotators to ensure "realistic logic" and to avoid content overlap with training data (Section 3.1, Appendix A).

• Base models. The evaluation covers 25 video generation models grouped into three families. Commercial models (8 evaluated): Wan 2.6, Seedance 1.5 Pro, Wan 2.5, Hailuo v2, Veo 3, Seedance 1.0, Kling 2.6 Pro, Sora v2 Pro, and Sora v1. These are accessed via official APIs with developer-recommended settings. Open-source models (12 evaluated): Wan2.2_A14B, HunyuanVideo 1.5, LongCat-Video, Wan2.1_14B, LTX-2, Wan2.2_5B, SkyReels-V2, LTX-Video, FramePack, HunyuanVideo, and CogVideoX-5B. These are run with official default configurations. Robotics-specific models (5 evaluated): Cosmos 2.5 (NVIDIA's world simulation model), DreamGen (two variants: gr1 and droid), Vidar, and UnifoLM-WMA-0. These are purpose-built for robotic video generation or policy learning. The models span a wide range of scales (from 5B to 14B+ parameters for open-source models), architectures (Diffusion Transformer, latent diffusion, MoE), and training paradigms (general video generation, world simulation, policy-conditioned generation). The paper argues this diversity "provides a comprehensive perspective on model performance" (Section 5.1).

• Metrics. Five fine-grained metrics are computed for each generated video, aggregated into two final indicators. Task Completion (TC) averages Physical-Semantic Plausibility (PSS, detecting floating, penetration, spontaneous emergence, and non-contact attachment via MLLM VQA on temporal grids) and Task-Adherence Consistency (TAC, checking task responsiveness and key action completeness via MLLM VQA with task-specific criteria). Visual Quality (VQ) combines Robot-Subject Stability (RSS, contrastive VQA on reference vs. generated frames for morphological and object identity consistency) and Motion Smoothness (MS, Q-Align-based temporal quality fluctuation with adaptive thresholding), penalized by a Motion Amplitude penalty (for insufficient robot movement) and an RSS degradation penalty (for morphological instability). The final model-level score is the mean of TC and VQ across all evaluation samples. The paper validates these automated metrics against human judgments from 30 participants, reporting a Spearman rank correlation of ρ = 0.96 on a ten-model subset (Table 3, Section 5.3, Appendix D).

• Baselines. The paper does not use baseline "methods" in the traditional sense — it benchmarks all 25 models against each other on the same evaluation set using the same metrics. The key comparative baselines are therefore implicit: commercial vs. open-source models (to measure the capability gap), general-purpose vs. robotics-specific models (to measure the value of domain-specific training), and model iterations (e.g., Wan 2.1 → 2.5 → 2.6, Seedance 1.0 → 1.5 Pro) to measure the effect of scaling and iterative refinement. The paper reports all models' scores in Table 2, enabling pairwise comparisons at a glance.

• Generation budget / compute accounting. The paper does not use a FLOPs-based compute budget. Generation is measured at the level of individual model invocations: for each image-text pair in the benchmark, three videos are generated per model and scores are averaged to "minimize errors" from stochastic generation (Section 5.1). Each model generates at its intended operating point — the paper does not equalize resolution, frame rate, or generation duration across models. Video specifications range from 832×480 at 16 fps (Wan2.1_14B, LTX-Video) to 1536×1024 at 24 fps (LTX-2), with durations of 4–6 seconds. This means the comparison is not FLOPs-matched; it reflects each model's performance at its default quality-compute tradeoff. The finetuning experiments in Section 5.4 use 200k randomly sampled instances from RoVid-X due to "computational constraints" — roughly 5% of the full 4M dataset.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for the main benchmark evaluation — each model is evaluated on all 650 samples, and the reported scores are means across these samples. For the finetuning experiments (Section 5.4), the 200k training samples are randomly sampled from the full dataset, but the paper does not report multiple seeds or confidence intervals. The human preference study (Section 5.3) uses 30 participants, with votes aggregated via a 5/3/1 win-tie-loss scoring scheme and Spearman rank correlation computed against RBench scores. The correlation is reported as ρ = 0.96 with a two-sided p < 10⁻³, indicating statistical significance.

---

Main Quantitative Results

Overall Rankings and the Divide Between Model Families

The central quantitative result is the comprehensive model ranking in Table 2, which reports per-task scores, per-embodiment scores, and an overall average ("Avg." column) for all 25 models. The headline finding is a clear hierarchy across model families:

• Commercial closed-source models occupy the top 7 positions, with Wan 2.6 at Rank 1 (Avg 0.607), Seedance 1.5 Pro at Rank 2 (0.584), Wan 2.5 at Rank 3 (0.570), Hailuo v2 at Rank 4 (0.565), Veo 3 at Rank 5 (0.563), Seedance 1.0 at Rank 6 (0.551), and Kling 2.6 Pro at Rank 7 (0.534).

• Open-source models trail substantially, with the best open-source model (Wan2.2_A14B) at Rank 8 (0.507) — a gap of 0.100 points (roughly 16.5%) below the top commercial model. The open-source tier shows sharp internal stratification: Wan2.2_A14B (0.507) > HunyuanVideo 1.5 (0.460) > LongCat-Video (0.437) > Wan2.1_14B (0.399), representing a spread of 0.108 points within the open-source family alone.

• Robotics-specific models occupy a middle-to-low tier, with Cosmos 2.5 at Rank 9 (0.464) outperforming most open-source models but trailing all top-tier commercial systems. Narrowly-trained robotics models rank at the bottom: Vidar at Rank 24 (0.206) and UnifoLM-WMA-0 at Rank 25 (0.123).

The key insight from these rankings is not merely "commercial models are better" — it's that the performance spread (0.607 to 0.123) is enormous, indicating that the benchmark is discriminative across a wide capability range and that current models have dramatically different levels of physical understanding.

The Media-Simulation Gap: Consumer Models Underperform on Physical Tasks

A striking pattern in Table 2 is the counter-intuitive underperformance of widely-recognized consumer models. The Sora series — widely celebrated for creative video generation — performs poorly: Sora v2 Pro ranks 17th (Avg 0.362) and Sora v1 ranks 22nd (0.266). This places Sora v2 Pro below several open-source models (Wan2.2_5B at Rank 16, 0.380; LTX-2 at Rank 15, 0.381) and far below its commercial peers.

The per-task breakdown reveals why: Sora v2 Pro scores 0.208 on Common Manipulation, 0.186 on Multi-Entity Collaboration, and 0.115 on Visual Reasoning (Table 2, Tasks columns). These are task completion failures — the model generates videos that may look smooth but don't execute the specified robotic actions. The paper's qualitative analysis (Figure 5, Appendix F) confirms this: in the Long-Horizon Planning task, Hailuo "lacks the 'turn-on' action, leading to a violation of physical logic," while in the Spatial Relationship task, several models "mistakenly place [the object] inside the pan" rather than to the left of it.

This finding validates the paper's central thesis: proficiency in creative video generation does not naturally transfer to embodied AI tasks. The Sora models are optimized for visual smoothness and cinematic transitions — objectives that are orthogonal to, and in some cases actively compete with, physical task execution. The paper's metrics, particularly TAC and PSS, capture this distinction in a way that perceptual benchmarks like VBench cannot.

Difficulty-Conditioned Performance: The Cognitive and Manipulation Gaps

The per-task breakdown in Table 2 reveals systematic difficulty patterns across all model families. The paper identifies two specific gaps:

The Cognitive Gap. Even top-tier models show a sharp performance drop on Visual Reasoning compared to execution-oriented tasks. Wan 2.6 scores 0.655 on Spatial Relationship but only 0.531 on Visual Reasoning — a gap of 0.124 points (19%). This pattern holds across models: Wan 2.5 shows a similar gap (0.576 on Spatial vs. 0.437 on Visual), and Hailuo v2 drops from 0.636 to 0.473. In contrast, tasks like Common Manipulation and Spatial Relationship show consistently higher scores across all models, suggesting that generating physically plausible object interactions and spatial layouts is easier for current architectures than translating fine-grained visual-semantic reasoning (color, texture, numerical ordering, text reading) into robot actions.

The Manipulation Gap. The embodiment-specific scores in Table 2 show that "models consistently score higher on coarse-grained locomotion tasks (Quadruped, Humanoid) than on fine-grained manipulation." For Wan 2.6: Humanoid 0.667, Quadruped 0.723 vs. Single Arm 0.666, Dual Arm 0.680. The gap is much more pronounced in weaker models: CogVideoX-5B scores 0.496 on Humanoid but only 0.338 on Single Arm — a gap of 0.158 points (32%). The paper attributes this to pre-training data distributions: human-centric video data provides stronger motion priors for humanoid embodiments, while the distinct kinematics of industrial manipulators (fixed base, articulated joints, parallel grippers) have few analogs in the training distribution.

These difficulty-conditioned patterns are evident from the table itself — no additional analysis is required to see that the "Visual Reasoning" column has systematically lower numbers across all models than the "Spatial Relationship" column, or that the "Quadruped" column has systematically higher numbers than the "Single Arm" column. The paper's contribution is in identifying and naming these patterns as diagnostic signals for where video generation models need improvement.

Iterative Scaling Unlocks Physical Capabilities

The paper identifies a compelling pattern in model evolution: model iteration correlates strongly with improved physical reasoning, not just improved visual quality. The Wan series provides the clearest evidence:

• Wan 2.1 (Rank 14, Avg 0.399) → Wan 2.2_5B (Rank 16, 0.380) → Wan 2.5 (Rank 3, 0.570) → Wan 2.6 (Rank 1, 0.607).

The jump from Wan 2.1 to Wan 2.5 represents a gain of 0.171 points (43%), and Wan 2.6 adds another 0.037 points. Similarly, Seedance evolves from 1.0 (Rank 6, 0.551) to 1.5 Pro (Rank 2, 0.584).

Critically, these improvements are not uniform across tasks. The Wan series shows the largest gains on tasks that the paper identifies as "execution-oriented": Wan 2.6 scores 0.656 on Spatial Relationship compared to Wan 2.1's 0.268 — a gain of 0.388 points (145%). On Visual Reasoning, the gain is smaller: from 0.204 (Wan 2.1) to 0.531 (Wan 2.6) — a gain of 0.327 points (160% relative, but still leaving a substantial absolute gap). This suggests that scaling and iterative refinement improve physical execution capabilities faster than cognitive reasoning capabilities.

The paper interprets this as evidence that "scaling laws and iterative optimization are not just improving visual quality but are actively refining the model's understanding of physics, distinct motion patterns, and control logic" (Section 5.2.1). This is supported by the pattern, but alternative explanations are possible: the improvements could reflect better training data quality, better architecture choices, or simply better optimization of the visual quality objective, which happens to correlate with physical plausibility scores. The paper does not isolate the causal mechanism.

Robotics-Specific Training Helps But Cannot Compensate for Scale

The performance of robotics-specific models reveals a nuanced tradeoff. Cosmos 2.5 (Rank 9, Avg 0.464) outperforms 11 open-source models including significantly larger general-purpose systems (e.g., HunyuanVideo at Rank 21, 0.303), despite being purpose-built for physical AI simulation. The paper interprets this as evidence that "training with physical data yields stable gains in robotic tasks" (Section 5.2.1).

However, Cosmos 2.5 still trails all top-7 commercial models by substantial margins (0.464 vs. Wan 2.6's 0.607). This suggests that domain-specific data cannot fully compensate for the "World Knowledge" provided by large-scale pretraining — a finding the paper calls the "Specialization Dilemma."

At the low end, models fine-tuned on very narrow robot data perform dramatically worse: Vidar (Rank 24, 0.206) and UnifoLM-WMA-0 (Rank 25, 0.123) rank at the absolute bottom — below even the weakest general-purpose open-source models. Vidar scores 0.073 on Common Manipulation, 0.054 on Long-Horizon Planning, and 0.050 on Visual Reasoning. These are near-floor scores, indicating that the model produces essentially task-irrelevant outputs on most evaluation dimensions. The paper concludes that "domain-specific data is valuable for control precision, [but] it cannot fully compensate for the deficit in 'World Knowledge' provided by large-scale pretraining" — and that the tradeoff between proprietary robot data and generalizable representations "remains a pivotal challenge for future research."

Human Evaluation Validation

Table 3 reports the human preference study results on a ten-model subset. The Spearman rank correlation between human scores and RBench scores is ρ = 0.96, with a two-sided p < 10⁻³. The per-model rank differences (∆r column) are minimal: 8 of 10 models have rank differences of 0 or ±1. Only CogVideoX-5B shows a rank difference of +1 (ranked 10th by humans, 9th by RBench) and LTX-Video shows a rank difference of -1 (ranked 10th by RBench, 9th by humans).

The absolute scores are also closely aligned, with human scores and RBench scores differing by at most 0.098 points (for LTX-Video: 0.246 human vs. 0.344 RBench). The Bland-Altman analysis (Appendix D, Figure 15) shows a bias of 0.002 and 95% limits of agreement of [-0.108, 0.112] after linear leave-one-out calibration, indicating close absolute agreement beyond mere rank correlation.

This validation is critical because the paper's automated metrics rely on MLLM-based evaluation (GPT-5 and Qwen3-VL), which could in principle diverge from human judgment in systematic ways. The high correlation demonstrates that the VQA-style prompts and scoring rubrics capture human-perceivable quality differences in robotic video generation.

RoVid-X Finetuning Validation

Table 4 reports the effectiveness of RoVid-X by finetuning two base models (Wan2.1_14B and Wan2.2_5B) on 200k randomly sampled instances. The key results:

For Wan2.1_14B (base → finetuned):

• Total Score: 0.399 → 0.446 (+0.047, 11.8% relative improvement)
• By task: Common Manipulation 0.344 → 0.376 (+0.032), Long-Horizon 0.335 → 0.389 (+0.054), Multi-Entity 0.282 → 0.295 (+0.013), Spatial 0.268 → 0.314 (+0.046), Visual Reasoning 0.205 → 0.298 (+0.093)
• By embodiment: Single Arm 0.464 → 0.526 (+0.062), Dual Arm 0.497 → 0.546 (+0.049), Quadruped 0.595 → 0.639 (+0.044), Humanoid 0.599 → 0.628 (+0.029)

For Wan2.2_5B (base → finetuned):

• Total Score: 0.380 → 0.439 (+0.059, 15.5% relative improvement)
• By task: Common Manipulation 0.331 → 0.373 (+0.042), Long-Horizon 0.318 → 0.387 (+0.069), Multi-Entity 0.142 → 0.221 (+0.079), Spatial 0.313 → 0.403 (+0.090), Visual Reasoning 0.234 → 0.284 (+0.050)
• By embodiment: Single Arm 0.436 → 0.514 (+0.078), Dual Arm 0.448 → 0.503 (+0.055), Quadruped 0.590 → 0.628 (+0.038), Humanoid 0.607 → 0.641 (+0.034)

Several patterns are notable. First, improvements are consistent across all task and embodiment categories — there is no dimension where finetuning hurts performance. Second, the largest relative gains tend to occur on the hardest tasks: Visual Reasoning (+0.093 for Wan2.1_14B) and Spatial Relationship (+0.090 for Wan2.2_5B) show some of the largest absolute improvements. The Multi-Entity Collaboration task for Wan2.2_5B shows the largest relative improvement (0.142 → 0.221, +56%), though from a very low base. Third, the embodiment-level improvements are larger for the embodiment types that general models struggle with most (Single Arm: +0.078 for Wan2.2_5B) compared to those where performance is already higher (Humanoid: +0.034).

The paper interprets these results as evidence that RoVid-X's diverse robot embodiment coverage and physical property annotations provide training signal that general video pretraining misses. The improvements on manipulation-heavy tasks (Single Arm, Dual Arm) and cognitive tasks (Visual Reasoning) specifically suggest that the dataset's structured captions and physical annotations address the failure modes identified by RBench.

---

Ablation Studies and Robustness Checks

The paper does not conduct traditional ablation studies in the sense of systematically removing components and measuring performance degradation. The closest equivalents are:

MLLM evaluator choice (GPT-5 vs. Qwen3-VL): The paper reports full evaluation results using both GPT-5 (Tables 6-14 in Appendix G) and Qwen3-VL (Tables 15-23). This serves as a robustness check on the automated metrics — if both MLLMs produce similar rankings, the metrics are less likely to be artifacts of a particular model's biases. While the paper does not explicitly compare the two sets of rankings, the fact that both are reported suggests they are broadly consistent. The availability of both open-source (Qwen3-VL) and closed-source (GPT-5) evaluation enables reproducibility without API dependence.

Task-specific vs. generic TAC criteria: The paper's TAC metric uses different evaluation criteria for each of the five task families (Section 3.2.1, Appendix B). This is an implicit ablation: rather than using a single generic "task adherence" VQA prompt for all tasks, the paper tailors the evaluation to each task's specific requirements (event completion rates for Long-Horizon Planning, question chains for Visual Reasoning, spatial relation accuracy for Spatial Relationship, etc.). The paper does not report what happens with generic criteria, but the design choice itself reflects an implicit finding from development that task-specific evaluation is necessary for discriminative assessment.

Motion amplitude penalty thresholds: The threshold values for the motion amplitude penalty ($t = 0.1$, $t_{\text{low}} = 0.05$) and the adaptive smoothness thresholds ($\tau_s(m)$ values of 0.01, 0.015, 0.025, 0.03) are described as being "determined through grid search on a validation split to ensure reproducibility" (Appendix B.5). However, the paper does not report the grid search results, the validation split used, or the sensitivity of final rankings to these thresholds. This is a methodological gap — the reported scores depend on these thresholds, and without sensitivity analysis, the reader cannot assess how robust the rankings are to different threshold choices.

Finetuning with different dataset sizes: The finetuning experiments use 200k samples from RoVid-X (~5% of the full dataset) due to "computational constraints" (Section 5.4). The paper does not report results with different dataset sizes (e.g., 50k, 100k, 400k, 1M), so the scaling behavior of dataset size vs. performance improvement is unknown. This is a significant limitation: the paper's central claim about RoVid-X's value rests on these finetuning results, but we don't know whether performance would continue to improve with larger subsets of the 4M dataset or whether the gains saturate at 200k.

Choice of base models for finetuning: The paper finetunes only two base models (Wan2.1_14B and Wan2.2_5B), both from the same model family (Wan) and both using the same architecture (Diffusion Transformer). Results on other architectures (e.g., HunyuanVideo's architecture, CogVideoX's expert transformer) are not reported. This limits the generalizability claim — we don't know whether RoVid-X provides similar benefits for models with fundamentally different architectures.

Two MLLM evaluators as implicit robustness: The paper uses both GPT-5 and Qwen3-VL for evaluation. The detailed per-task and per-embodiment scores from both evaluators are reported in Appendix G (Tables 6-23). While the paper does not compute the correlation between the two evaluators' scores or discuss differences, the availability of both sets of results provides a natural robustness check. A reader comparing Table 6 (Common Manipulation with GPT) to Table 15 (Common Manipulation with Qwen) can observe that the absolute scores differ (GPT scores tend to be lower) but the relative rankings appear broadly consistent.

Human evaluation on a subset: The human evaluation study (Section 5.3, Table 3) uses only 10 of the 25 evaluated models. The paper doesn't explain why these 10 were chosen or whether they are representative of the full model distribution. The 10-model subset spans the full performance range (from Wan 2.5 at 0.573 human score to CogVideoX-5B at 0.333), which is appropriate for correlation analysis, but the Spearman correlation of 0.96 on this subset may not generalize perfectly to the full 25-model set.

No ablation on the physical property annotations: The paper does not report an experiment where models are finetuned on RoVid-X without the physical property annotations (optical flow, depth maps, super-resolution) to isolate their contribution. This is a significant gap: the paper claims that these annotations are valuable for training, but provides no direct evidence that they improve performance beyond what the raw video + caption data would provide. The reported finetuning improvements could be entirely attributable to the scale and diversity of the video data, with the annotations contributing nothing — without an ablation, we cannot distinguish these hypotheses.

No comparison to alternative dataset construction strategies: The paper does not compare its four-stage pipeline (GPT-5 filtering → quality scoring → MLLM captioning → physical annotation) to simpler alternatives, such as using only raw video with automatically generated captions without quality filtering or physical annotations. This makes it impossible to assess whether the pipeline's complexity is justified by performance gains.

---

Critical Assessment

Claim 1: RBench provides a comprehensive benchmark with fine-grained metrics that reveal deficiencies in current video generation models for robotic tasks.

What was demonstrated: The paper evaluated 25 models across 5 tasks and 4 embodiments using 5 automated metrics, producing a rich set of diagnostic results (Table 2, Appendix G) that reveal systematic performance patterns — the Media-Simulation Gap, the Cognitive Gap, the Manipulation Gap, and the Specialization Dilemma. The metrics correlate strongly with human judgment (ρ = 0.96, Table 3).

What was not demonstrated: The paper does not establish that RBench's metrics are more discriminative or informative than simpler alternatives. There is no comparison to using a single generic VQA prompt ("Does this video correctly show the robot performing the task?") vs. the task-specific, multi-metric approach. The 0.96 Spearman correlation with human judgments is impressive, but it is computed on only 10 models and does not compare against a simpler baseline evaluation protocol (e.g., using only CLIP score or only generic MLLM evaluation). Without such a comparison, the paper's claim that its fine-grained metrics are necessary (rather than merely sufficient) for good evaluation is unvalidated.

Genuine weaknesses:

• The benchmark contains only 650 samples, with 50 per task category and 100 per embodiment. Split across the 9 per-task metrics reported in Appendix G (which involve additional sub-scores like AES, TCS, OCS, RCS, etc.), each fine-grained sub-metric is computed on a very small number of samples. This raises questions about the statistical reliability of per-task and per-embodiment scores, especially for the 5th-ranked through 15th-ranked models where score differences are small (0.01–0.03).
• The paper does not report confidence intervals, standard deviations, or any measure of variance for the scores in Table 2. With 3 generations per sample, there is inherent variance from stochastic generation, and without reporting this variance, the reader cannot assess whether a 0.02 difference in average score is statistically meaningful.
• The 650 evaluation samples are curated through a process that involves human annotator verification, but the paper does not report inter-annotator agreement on prompt quality or sample selection. If annotators disagreed on what constitutes a "realistic" prompt or a "high-quality" reference image, the evaluation set itself may encode annotator biases.

Claim 2: Commercial closed-source models significantly outperform open-source models on robotic video generation, revealing a "substantial capability gap."

What was demonstrated: Table 2 shows that the top 7 models are all commercial closed-source, with the best open-source model (Wan2.2_A14B, Rank 8) scoring 0.100 points below the top commercial model (Wan 2.6, Rank 1). The open-source tier spans a wide range (0.256 to 0.507), with multiple open-source models scoring below 0.40.

What was not demonstrated: The comparison is not controlled for model scale, training data, or architecture. The paper does not report parameter counts, training data size, or training compute for any commercial model, making it impossible to determine whether the performance gap reflects better technology or simply more compute. Wan 2.6 could be dramatically larger than Wan2.2_A14B — without scale information, the "commercial vs. open-source" gap could equally be described as a "scale gap" or "compute gap."

Genuine weaknesses:

• The paper evaluates only one version of each model (its default configuration). Commercial APIs may use different generation parameters than open-source defaults (e.g., more inference-time compute, different guidance scales, different noise schedules). The performance gap could partially reflect API-side optimizations rather than model capability differences.
• The most capable open-source model (Wan2.2_A14B) is from the same family as the top commercial model (Wan 2.6) — they are both Wan models. The gap between Wan 2.6 and Wan2.2_A14B may reflect the difference between Wan's commercial and open-source offerings specifically, not a general commercial/open-source gap across model families.

Claim 3: RoVid-X is an effective training dataset that improves video generation model performance on robotic tasks, validating both the dataset and the data collection pipeline.

What was demonstrated: Finetuning two Wan models on 200k RoVid-X samples produces consistent improvements across all task categories and embodiments (Table 4). Wan2.1_14B improves from 0.399 to 0.446, and Wan2.2_5B improves from 0.380 to 0.439.

What was not demonstrated:

• The finetuning uses only 5% of the full dataset (200k out of 4M). The paper does not establish that RoVid-X's claimed primary advantage — its scale (4M clips) — is responsible for the improvements. The gains could saturate at 200k, meaning the dataset's size beyond 200k is unused and potentially unnecessary. Alternatively, the gains could continue to scale with dataset size, in which case the reported improvements are a lower bound. Without scaling curve data, the paper provides no evidence either way.
• The paper does not compare RoVid-X finetuning to finetuning on alternative datasets of comparable size. Would 200k samples from Open X-Embodiment or Agibot World produce similar or better improvements? Without this comparison, the claim that RoVid-X is specifically valuable (as opposed to "more robot video data" generically) is unsubstantiated.
• The physical property annotations (optical flow, depth maps, super-resolution) are claimed as key differentiators of RoVid-X (Table 1 highlights these as unique features), but no ablation shows they contribute to the finetuning gains. It is possible that the raw video + caption data alone would produce identical improvements, in which case the annotation pipeline is unnecessary complexity.
• The finetuning uses MSE loss exclusively. This is a standard finetuning approach, but the paper does not explore whether alternative finetuning strategies (e.g., incorporating the optical flow or depth annotations as auxiliary losses) would yield larger gains. The reported improvements may represent a lower bound on what the full dataset + annotation-aware training could achieve, but conversely, the annotations may provide no benefit at all under the simple MSE-loss finetuning regime used.

Genuine weaknesses:

• Only two base models are tested, both from the same architecture family (Wan Diffusion Transformer). The improvements may not transfer to other architectures (e.g., HunyuanVideo's 3D VAE + DiT, CogVideoX's expert transformer, LTX's Video-VAE). The paper cannot claim general effectiveness of RoVid-X based on two models from one family.
• The finetuned models are not compared to the top-performing commercial models on RBench in Table 4. The finetuned Wan2.1_14B (0.446) and Wan2.2_5B (0.439) still rank below Cosmos 2.5 (0.464) and far below the commercial tier (0.534+). The improvements are meaningful but do not close the capability gap — RoVid-X finetuning makes weak models less weak, not competitive with state-of-the-art.
• The 200k random sample may not represent the full dataset's diversity. If the random sample skews toward certain robot types or task categories, the reported improvements may not reflect what full-dataset finetuning would achieve. The paper does not characterize the random sample's distribution relative to the full dataset.

Claim 4: The compute-optimal or adaptive difficulty-aware evaluation reveals that different model families fail in different ways, enabling targeted improvement.

What was demonstrated: The paper identifies several diagnostic patterns — the Cognitive Gap (models worse at Visual Reasoning than Spatial Relationship), the Manipulation Gap (models worse at Single Arm than Quadruped/Humanoid), and the Specialization Dilemma (robotics-specific training helps but cannot replace scale). These patterns are visible in Table 2's per-task and per-embodiment breakdowns.

What was not demonstrated: The paper does not show that these diagnostic insights enable targeted improvement — the finetuning experiments (Table 4) use the full RoVid-X dataset, not a targeted subset designed to address specific gaps. A stronger validation would be: if RBench reveals that a model is weak on Visual Reasoning, then finetuning specifically on Visual-Reasoning-heavy data from RoVid-X should improve Visual Reasoning scores disproportionately. The paper does not run such an experiment.

Genuine weaknesses:

• The diagnostic patterns, while interesting, are primarily descriptive. The paper identifies that models struggle with fine-grained manipulation, but does not investigate why — is it a training data issue, an architecture issue, or a fundamental limitation of current video generation paradigms? The benchmark reveals symptoms but not causes.
• The paper attributes performance patterns to specific causes (e.g., "models pre-trained on human-centric video data have stronger motion priors for humanoid embodiments") without experimental evidence. This is plausible speculation, but the paper does not train models with and without human-centric pre-training to test this hypothesis. The diagnostic patterns are correlational, not causal.

Missing Experiments That Would Have Strengthened the Paper

• Scaling curves for RoVid-X finetuning: training on 50k, 100k, 200k, 400k, 1M, and 4M samples to characterize how performance scales with dataset size. This would validate the dataset's scale advantage and help practitioners decide how much data to use.
• Ablation of physical property annotations: finetuning with video+captions only vs. video+captions+optical flow vs. video+captions+depth vs. full annotations, to quantify each annotation's contribution.
• Cross-architecture finetuning: evaluating RoVid-X on HunyuanVideo, CogVideoX, or LTX base models to establish generalizability.
• Comparison to alternative datasets: finetuning on equivalently-sized subsets of Open X-Embodiment, DROID, or Agibot World to establish RoVid-X's relative value.
• Statistical significance reporting: confidence intervals or standard deviations for the scores in Table 2 and Table 4, to enable assessment of whether small score differences are reliable.
• Per-category sample sizes and reliability: reporting the number of samples contributing to each per-task and per-embodiment score, and analyzing score stability as a function of sample size.
• Sensitivity analysis on metric thresholds: varying the motion amplitude penalty thresholds ($t$, $t_{\text{low}}$) and adaptive smoothness thresholds ($\tau_s(m)$) to assess whether rankings are robust to these hyperparameter choices.
• Generic vs. task-specific evaluation ablation: comparing RBench's task-specific TAC metrics to a single generic task-adherence prompt, to validate the paper's claim that fine-grained, task-specific evaluation is necessary.
• Closed-loop validation: evaluating whether RBench scores predict downstream task performance (e.g., policy success rate when training on generated videos), which would validate the benchmark's practical relevance beyond correlation with human aesthetic judgments.

6. Limitations and Trade-offs

The Difficulty Estimation Bottleneck: Evaluating RBench Scores Requires Expensive MLLM Inference

The assumption or constraint. RBench's automated evaluation metrics rely on two multimodal large language models — GPT-5 (closed-source) and Qwen3-VL (open-source) — to compute Physical-Semantic Plausibility, Task-Adherence Consistency, and Robot-Subject Stability. Each evaluation sample requires multiple MLLM calls: temporal grid analysis for PSS, task-specific VQA checklists for TAC, contrastive frame-pair analysis for RSS, and in the case of Visual Reasoning, an additional question-chain generation step. The paper does not report the total inference cost per evaluation sample, the token consumption per MLLM call, or the wall-clock time required to evaluate a single model.

The consequence. For practitioners who want to use RBench to evaluate their own models, the MLLM inference cost is a non-trivial barrier. If evaluating one model on all 650 samples costs hundreds of dollars in API fees (for GPT-5) or requires substantial GPU resources (for Qwen3-VL), iterative development cycles — where researchers evaluate models repeatedly during training or hyperparameter tuning — become prohibitively expensive. This creates a practical tension: RBench is positioned as a standardized benchmark for the community, but its reliance on expensive MLLM evaluation means it cannot be used as a frequent, lightweight diagnostic tool. The paper's own evaluation of 25 models with 3 generations per sample (1,950 generated videos per model, or 48,750 total generated videos) represents a computational investment whose magnitude is not disclosed.

What evidence exists in the paper. The paper provides no cost analysis — no token counts, no API pricing estimates, no GPU-hour requirements. The evaluation protocol (Section 5.1, Appendix B) describes what each MLLM call does but not what it costs. The availability of Qwen3-VL as an open-source alternative is noted but its hardware requirements are not specified. The paper acknowledges that evaluation uses "a temporal grid of uniformly sampled key frames" rather than full video processing, which limits token consumption relative to dense video evaluation, but does not quantify the savings.

Mitigation status. The paper does not address this limitation directly. Using an open-source MLLM (Qwen3-VL) partially mitigates the API cost concern, but the computational cost of running even an open-source model at evaluation scale remains unmeasured. The paper does not propose a cheaper proxy metric, a distilled evaluator, or a reduced evaluation subset that could serve as a lightweight development-time alternative to full RBench evaluation.

---

RoVid-X Finetuning Validation Is Performed on Only 5% of the Dataset, Leaving the Scale Claim Unverified

The assumption or constraint. The paper's central claim about RoVid-X is that its unprecedented scale — 4 million annotated video clips — addresses a critical data scarcity bottleneck in robotic video generation. However, the finetuning experiments that validate the dataset's effectiveness (Section 5.4, Table 4) use only 200k randomly sampled instances, or approximately 5% of the full dataset. The paper cites "computational constraints" as the reason for this limitation, but does not characterize how the 200k subset relates to the full 4M dataset in terms of distribution, diversity, or coverage.

The consequence. The headline result — that RoVid-X improves model performance — does not actually validate the dataset's primary claimed advantage (its scale). The improvements observed from 200k samples could saturate at that point, meaning the remaining 3.8M clips provide no additional benefit. Conversely, if performance continues to scale with dataset size, the 5% results represent a substantial underestimate of the full dataset's value. Neither hypothesis is tested. A practitioner deciding whether to download, store, and train on the full 4M-clip dataset receives no guidance on whether the marginal benefit of the additional 3.8M clips justifies the storage and compute costs. Moreover, the 200k random sample could be unrepresentative — if it disproportionately preserves the easier-to-learn or more common robot types and tasks, the reported improvements may not reflect what the full dataset would provide, and may overestimate how well a 200k subset generalizes to the long-tail diversity of the full dataset.

What evidence exists in the paper. Table 4 reports improvements from +0.047 (Wan2.1_14B) to +0.059 (Wan2.2_5B) in total score after finetuning on 200k samples. The paper provides no scaling curves — no results at 50k, 100k, 400k, 1M, or 4M samples. The distribution of the 200k random sample relative to the full dataset is not characterized (Section 5.4 mentions only that it was "randomly sampled" and the constraint was "computational").

Mitigation status. The paper acknowledges the computational constraint explicitly but treats the 200k experiments as sufficient evidence. The "Future Work" section does not mention conducting full-dataset finetuning experiments. The paper does not propose a strategy for practitioners to estimate the value of the full dataset from the 5% results, such as extrapolation from subsample scaling trends.

---

The Benchmark Covers a Narrow Task Domain (Structured Robot Manipulation) with No Evidence of Generalization to Other Embodied Domains

The assumption or constraint. RBench's evaluation is constructed around five task categories and four robot embodiment types, all in the domain of structured robot manipulation and locomotion in laboratory or quasi-laboratory settings. The tasks involve discrete, instruction-following behaviors: pick-and-place, multi-step planning sequences, spatial reasoning about object positions, and visual attribute matching. The embodiments cover wheeled/tracked manipulators, humanoids, and quadrupeds — but not aerial robots, soft robots, swarm systems, or mobile manipulators in unstructured outdoor environments. The paper implicitly assumes that performance on RBench is a meaningful proxy for video generation capability in the broader embodied AI landscape.

The consequence. A model that performs well on RBench may not generalize to embodied video generation tasks with fundamentally different characteristics: continuous control tasks without discrete sub-goals (e.g., fluid pouring, deformable object manipulation), interaction with non-rigid or granular materials, tasks requiring long-range navigation through cluttered environments, multi-agent scenarios beyond the "Primary Entity + Secondary Entity" dyad, or tasks in outdoor environments with natural terrain, variable lighting, and weather. This is not merely a "not tested on X" observation — it matters because the paper positions RBench as a general benchmark for "robot-oriented video generation" and "embodied intelligence," and the benchmark's task and embodiment design implicitly defines what "embodied" means. A model that scores 0.607 on RBench but cannot generate a convincing video of a drone navigating a forest or a robot manipulator kneading dough has not actually demonstrated general physical understanding — it has demonstrated proficiency on the specific subclass of rigid-body, discrete-task, indoor manipulation that RBench samples.

What evidence exists in the paper. The paper categorizes its task domains (Section 3.1, Appendix A) and embodiment types, making the scope explicit. However, the paper does not discuss the boundaries of this scope, does not acknowledge that these categories represent a subset of embodied AI challenges, and does not caution readers that RBench scores may not transfer to other embodied domains. The evaluation samples are drawn from public datasets and online sources (Section 3.1), which are themselves biased toward laboratory manipulation settings due to the historical priorities of the robot learning community. This sampling bias is not discussed as a limitation of what RBench can measure.

Mitigation status. No mitigation is attempted. The "Future Work" section mentions plans for "closed-loop control experiments in both simulation environments and on real-world hardware" and "more automated and physically grounded evaluation metrics," but does not mention expanding the benchmark's task or embodiment coverage. The paper's framing — "a comprehensive benchmark... for robotic video generation" — does not include the necessary qualifiers about domain scope.

---

The FLOPs-Matched, Latency, and Resolution Comparisons Across Models Are Uncontrolled

The assumption or constraint. The paper evaluates 25 models at their default operating points — each model generates videos at its intended resolution, frame rate, and duration without any attempt to equalize compute, latency, or output quality. The video specifications vary dramatically across models (Section 5.1, Appendix C): Wan2.1_14B generates 832×480 at 16 fps over 5 seconds (81 frames total), while LTX-2 generates 1536×1024 at 24 fps (roughly 4× the pixels per frame, 1.5× the frame rate), and CogVideoX-5B generates 720×480 at 8 fps over 6 seconds. The paper also notes that some models (e.g., DreamGen) generate at 768×432 while others generate at 1280×720. The evaluation metrics normalize certain computations (e.g., motion amplitude is normalized by video diagonal; Appendix B.4) but do not control for the information content available at different resolutions or frame rates.

The consequence. The model rankings in Table 2 confound model capability with generation budget. A model that generates 120 high-resolution frames at 24 fps has more temporal and spatial information to work with than a model generating 81 lower-resolution frames at 16 fps — even if the underlying video generation capability is identical, the higher-resolution, higher-framerate output provides more opportunity to demonstrate smooth motion, stable morphology, and task completion. The Motion Smoothness metric, for example, is computed from per-frame quality scores; more frames provide finer temporal granularity and may produce different smoothness scores independent of actual motion quality. A practitioner comparing their own model to the published rankings cannot determine whether a score difference reflects genuine capability differences or merely different generation parameters. The paper also does not discuss inference latency: commercial API models may use more inference-time compute or model ensembling that contributes to their performance advantage, but this remains invisible in the comparison.

What evidence exists in the paper. Appendix C specifies the generation parameters for each model. The paper does not discuss how these differences might affect the evaluation, does not attempt FLOPs estimation or latency measurement, and does not perform any controlled experiment (e.g., generating all models at a fixed resolution and frame rate, or comparing models at equal total pixel counts). The normalization steps in the metrics (e.g., motion normalization by diagonal) are documented but their adequacy for handling resolution differences is not validated.

Mitigation status. The paper does not acknowledge this as a limitation. The evaluation protocol states that each model is run with "official default configurations to ensure consistency with the model's preset settings" (Section 5.1), which treats the generation configuration as part of the model's identity rather than as a confound to be controlled. This is a reasonable choice for a benchmark that aims to evaluate models "as deployed," but it means the rankings should not be interpreted as measuring an abstract "video generation capability" separate from implementation choices.

---

The Human Evaluation Validates Only a 10-Model Subset, and No Baseline Evaluation Protocol Is Compared

The assumption or constraint. The paper validates its automated metrics against human judgments using a study with 30 participants evaluating 10 out of 25 models (Section 5.3, Table 3, Appendix D). The paper reports a Spearman correlation of ρ = 0.96 between human scores and RBench scores on this subset. However, the paper does not compare its multi-metric, MLLM-based evaluation protocol against simpler alternatives — for example, a single generic VQA prompt ("Does this video correctly show the robot performing the task?") or a standard perceptual metric (CLIP score, FVD) would almost certainly correlate with human judgments to some degree, and the paper provides no evidence that RBench's complex, task-specific, multi-metric design produces substantially better correlation than these simpler baselines.

The consequence. The ρ = 0.96 correlation is impressive in absolute terms, but its informativeness depends on the counterfactual. If a single generic MLLM prompt achieves ρ = 0.90 and RBench achieves ρ = 0.96, the marginal value of the complex evaluation protocol is modest relative to its computational cost. If a standard perceptual metric achieves ρ = 0.80, the value is larger but the benchmark may still be over-engineered for the gain. Without this comparison, the paper's claim that its fine-grained, task-specific metrics are necessary (rather than merely sufficient) for good evaluation remains unvalidated. Additionally, the human evaluation covers only 40% of the models in the benchmark (10 of 25); the paper does not justify this selection or establish that the 10-model subset is representative. Models at the extremes of the ranking (Rank 1–2 and Rank 24–25) are included, but the middle of the distribution — where score differences are small and discriminative power matters most — may be under-sampled. The 30-participant sample is adequate for aggregate correlation but small for detecting systematic biases in specific task categories or embodiment types.

What evidence exists in the paper. Table 3 reports the 10-model comparison with per-model human scores, RBench scores, ranks, and rank differences. The Spearman correlation is reported as ρ = 0.96 (p < 10⁻³). The Bland-Altman analysis in Appendix D (Figure 15) reports a bias of 0.002 and limits of agreement of [-0.108, 0.112] after linear leave-one-out calibration. The paper does not report any comparison to simpler evaluation baselines. The rationale for selecting these 10 models is not provided.

Mitigation status. The paper does not acknowledge the absence of baseline evaluation protocol comparisons. It does not discuss the representativeness of the 10-model subset or the statistical power of the 30-participant study for detecting per-category biases. The "Future Work" section mentions developing "more automated and physically grounded evaluation metrics" but not validating the current metrics against simpler alternatives.

---

The Benchmark Has No Demonstration of Practical Utility: RBench Scores Are Not Shown to Predict Downstream Task Performance

The assumption or constraint. RBench evaluates video generation models on how well their outputs satisfy task descriptions and exhibit physical plausibility, but the paper does not connect these evaluation scores to any downstream metric of practical interest for embodied AI. The paper's Introduction and Related Work sections motivate the benchmark by arguing that video generation models are used to synthesize robot training data, predict future states for planning, and serve as world simulators. However, the paper never tests whether a model with a higher RBench score actually produces better robot policies when its generated videos are used for training, or whether RBench scores correlate with the physical accuracy of a downstream visuomotor policy trained on the generated data.

The consequence. A practitioner choosing between two video generation models for a robot learning pipeline cannot use RBench scores to make an informed decision. Model A (RBench score 0.60) might produce visually convincing videos that look physically plausible to human evaluators and to MLLM judges but contain subtle contact dynamics errors that catastrophically degrade policy learning. Model B (RBench score 0.45) might produce less visually polished videos that nonetheless contain more physically accurate action trajectories, leading to better policy performance. Without a closed-loop validation — train a policy on generated videos, deploy on real hardware, measure success rate — the benchmark's claim to measure "the fidelity and utility of video-based world models" (Section 1) is an aspiration, not a demonstrated fact. The high human correlation (ρ = 0.96) validates that RBench aligns with human aesthetic and task judgments, but human judgment of video quality may not align with the requirements of training physically competent robot policies. This is the gap between "this video looks correct" and "a policy trained on this video would work in the real world" — a gap the paper opens but does not bridge.

What evidence exists in the paper. The paper includes no policy learning experiments, no sim-to-real transfer tests, and no correlation analysis between RBench scores and any downstream task metric. The "Future Work" section mentions plans to "employ Inverse Dynamics Models (IDM) to recover executable actions from generated videos, enabling closed-loop control experiments," but treats this as future work rather than as validation of the current benchmark.

Mitigation status. The paper does not attempt to close the loop between evaluation and utility. This is a deliberate scope choice — the paper focuses on establishing the benchmark and dataset — but it means that the paper's core premise (that better RBench scores imply better performance in embodied AI applications) remains a hypothesis. The RoVid-X finetuning experiments (Table 4) show that training on robot video data improves RBench scores, but do not show that the improved scores translate to better policy learning or simulation fidelity.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes evaluation as a first-class infrastructure problem for embodied video generation, shifting the field from a pretraining-centric view toward a measurement-and-improvement cycle that parallels what standardized benchmarks enabled in computer vision and NLP. Before RBench, the dominant evaluation paradigm treated video generation quality as synonymous with perceptual fidelity — sharp frames, smooth motion, semantic alignment with text prompts. This paper demonstrates that perceptual quality and physical task competence are orthogonal dimensions that can actively conflict, and that evaluating only one dimension produces systematically misleading conclusions about model capability. The Sora series is the canonical example: widely celebrated for creative video generation, yet ranking 17th and 22nd on RBench because cinematic smoothness does not imply physical correctness. This finding is not merely "Sora is bad at robotics" — it reveals that consumer video generation objectives and embodied AI requirements are not points on a continuum but distinct optimization targets that can pull models in opposite directions.

The broader reframing is that video generation for embodied AI is not a harder version of general video generation — it is a qualitatively different task with distinct failure modes, evaluation criteria, and training data requirements. The paper's five-metric taxonomy (physical-semantic plausibility, task-adherence consistency, motion amplitude, robot-subject stability, motion smoothness) operationalizes this distinction. Prior work treated "physical plausibility" as a monolithic concept tested through isolated physics puzzles (does the ball bounce?). RBench decomposes it into specific, robot-relevant sub-failures: floating/penetration, spontaneous emergence/disappearance, non-contact attachment, morphological drift, task omission, and incorrect action ordering. This decomposition is what makes the benchmark diagnostic rather than merely ranking — it tells you why a model failed, not just that it failed.

The paper also resolves a latent contradiction in how the field talks about video generation for robotics. Optimists point to models like Cosmos 2.5 and DreamGen as evidence that video generation can serve as world simulators; pessimists point to failures like Vidar and UnifoLM ranking at the bottom of RBench as evidence that these models fundamentally lack physical understanding. RBench provides a unified framework that reconciles both views: robotics-specific training yields stable gains (Cosmos 2.5 at Rank 9 outperforms many larger general models), but cannot compensate for the World Knowledge deficit from large-scale pretraining (it still trails all top-7 commercial models). The "Specialization Dilemma" the paper identifies — domain data helps but cannot replace scale — converts a binary debate into a quantified tradeoff with actionable implications: invest in both diverse pretraining and domain-specific physical data, with the optimal ratio depending on the target embodiment distribution.

The most lasting conceptual contribution may be the synergistic ecosystem framing: the argument that evaluation benchmarks and training datasets are coupled problems that must be solved together. This is more than a paper with two contributions. It articulates a specific thesis about how progress happens in embodied AI — that without measurement tools, data collection is undirected; without data, measurement reveals problems that cannot be fixed. The finetuning experiments (Table 4) provide initial evidence for this thesis by showing that RoVid-X-trained models improve on the dimensions RBench measures, but the thesis is better understood as a research program than a proven result. It redirects community energy from "build a better video model" (a modeling problem) to "build a better measurement-and-improvement cycle" (a systems problem), which is a fundamentally different and potentially more productive framing.

Follow-Up Research This Work Enables

Scaling laws for robotic video generation data. The paper claims RoVid-X's primary advantage is scale (4M clips vs. the next largest at 1.4M), but the finetuning experiments use only 200k samples (~5%). A direct follow-up would characterize the scaling relationship: train Wan2.1_14B or equivalent models on 50k, 100k, 200k, 400k, 1M, 2M, and 4M RoVid-X samples, evaluate on RBench, and fit a power law relating dataset size to TC and VQ scores. This would answer whether the marginal value of additional data saturates (making the full dataset unnecessary for most practitioners), continues to scale log-linearly (validating the investment in 4M clips), or shows emergent improvements at specific scale thresholds (suggesting that certain physical understanding capabilities require minimum data volumes to emerge). The finetuning cost at 200k samples is already reported as "computationally constrained" — the 4M-sample experiment would require estimating the compute budget needed and potentially developing a smaller-scale proxy (e.g., scaling with a subset of task categories) that extrapolates to full-dataset performance.

Ablation of physical property annotations on downstream policy learning. Table 1 highlights optical flow, depth maps, and super-resolution as unique RoVid-X features, but Table 4's finetuning experiments provide no evidence that these annotations contribute to performance improvements. A critical follow-up would train models on RoVid-X with and without each annotation type: (a) video + captions only, (b) video + captions + optical flow, (c) video + captions + depth, (d) video + captions + super-resolution, (e) full RoVid-X. Evaluating on RBench would measure whether the annotations improve video quality; more importantly, using the generated videos to train inverse dynamics models (IDMs) and then evaluating the resulting policies on real hardware would measure whether annotation richness translates to better downstream task performance. A negative result — annotations don't improve policy learning — would be highly informative, suggesting that the community's investment in physical property annotation may be misallocated relative to simply scaling raw video data.

Cross-architecture generalization of RoVid-X benefits. The finetuning experiments test only Wan-family Diffusion Transformer models. An extension would replicate the 200k-sample finetuning protocol on architecturally diverse models: HunyuanVideo (3D VAE + full-attention DiT), CogVideoX (3D VAE + expert transformer), LTX-2 (Video-VAE + efficient DiT), and Cosmos 2.5 (flow-based architecture). If improvements are consistent across architectures, RoVid-X's value is architecture-agnostic and the dataset can be recommended as universal training data. If improvements are architecture-specific, the dataset's design (resolution, caption format, annotation types) may be implicitly optimized for Wan-style models, and alternative dataset formulations might be needed for other architectures. This experiment would also test the paper's implicit claim that diverse robot embodiment data transfers across model families, since the Wan-specific finetuning results could reflect architecture-data synergy rather than general data value.

Human-to-robot motion prior transfer as a controlled experiment. The paper hypothesizes that human-centric video pretraining creates motion priors that transfer better to humanoid robots than to single-arm manipulators, and that this explains the Manipulation Gap in Table 2 (systematically higher Humanoid than Single Arm scores). A direct test would train two video generation models from scratch: one on a dataset of only human manipulation videos, one on a dataset with equal human and robot manipulation data, and a third on robot-only data (a narrowed RoVid-X). Evaluating all three on RBench across embodiments would quantify the transfer gradient — how much does human video pretraining help or hurt each embodiment type? A finding that human-only pretraining degrades single-arm manipulation performance relative to no pretraining would be strong evidence for negative transfer from human motion priors, suggesting that embodiment-balanced pretraining is necessary rather than merely beneficial. The paper's current evidence is correlational (models pre-trained on human data show the Manipulation Gap); a causal experiment would substantially strengthen the claim.

Closing the loop: RBench scores as predictors of policy learning success. The paper's most significant unvalidated premise is that higher RBench scores imply better downstream utility for embodied AI. A direct test would sample models across the RBench ranking spectrum (e.g., Wan 2.6 at Rank 1, Cosmos 2.5 at Rank 9, Wan2.1_14B at Rank 14, CogVideoX-5B at Rank 23), use each to generate synthetic trajectory videos for a fixed set of manipulation tasks, train identical inverse dynamics models and policies on each model's outputs, and evaluate the resulting policies on real hardware or high-fidelity simulation. The key question: does the RBench ranking order predict policy success rate order? A strong correlation would validate RBench's practical relevance; a weak or inverted correlation would reveal that the benchmark measures dimensions of video quality that are uncorrelated with the action-level information that policies extract, fundamentally challenging the paper's premise that better-looking robot videos produce better robot policies.

Dynamic difficulty-adaptive evaluation with targeted data selection. The paper identifies systematic difficulty patterns — models struggle more on Visual Reasoning and Single Arm manipulation than on Spatial Relationship and Humanoid locomotion — but does not demonstrate that these diagnostics enable targeted model improvement. A follow-up would use RBench to identify a specific model's weakest task category (e.g., CogVideoX-5B scores 0.098 on Multi-Entity Collaboration vs. 0.212 on Long-Horizon Planning), sample a RoVid-X subset enriched for that weak category, finetune the model on the targeted subset, and measure whether the weak-category score improves disproportionately compared to a random-sampling baseline. This would close the loop from diagnosis to treatment, demonstrating that RBench is not merely a ranking tool but an actionable development instrument. A negative result — targeted data doesn't produce disproportionate gains — would suggest that the benchmark's diagnostic categories don't carve the capability space at joints that correspond to independent, improvable sub-skills.

Practical Applications and Downstream Use Cases

Data filtering for robot learning pipelines. Teams building visuomotor policies from large-scale robot video datasets face a quality control problem: not all collected trajectories are equally useful for policy learning, and manually filtering thousands of hours of video is infeasible. RBench's automated metrics — particularly Physical-Semantic Plausibility and Task-Adherence Consistency — can be repurposed as data quality filters: run each collected trajectory video through the same VQA-style evaluation, discard or down-weight trajectories with low PSS or TAC scores (indicating teleoperation errors, incomplete tasks, or physical anomalies), and train policies only on the filtered subset. The paper's demonstration that these metrics correlate with human judgment at ρ = 0.96 (Table 3) provides confidence that the automated filtering would align with what a human annotator would select, at a fraction of the cost. The 4M-clip scale of RoVid-X demonstrates that automated evaluation at this volume is practical — the same infrastructure that evaluated 25 models on RBench can filter production-scale robot datasets.

Model selection for video-based policy augmentation. When augmenting a robot learning dataset with synthetically generated videos, practitioners must choose which video generation model to use. RBench provides a principled selection criterion: rather than choosing the model with the best perceptual quality or the largest parameter count, choose the model with the highest RBench score on the specific task category and embodiment type that matches the target deployment. For a team deploying a single-arm manipulator for common pick-and-place tasks, Table 2 shows that Wan 2.6 scores 0.666 on Single Arm and 0.546 on Common Manipulation — while Veo 3 scores 0.634 on Single Arm but 0.521 on Common Manipulation. The tradeoff depends on which dimension matters more for the specific deployment. Without RBench, this comparison would rely on generic video quality metrics that don't distinguish between task types, leading to suboptimal model selection.

Curriculum design for embodied video model training. The difficulty-conditioned performance patterns in Table 2 — models struggle more on Visual Reasoning than Spatial Relationship, more on Single Arm than Quadruped — can inform training data curricula. Rather than training uniformly on all RoVid-X data, practitioners can stage training: start with easier embodiment-task combinations (Quadruped locomotion, Humanoid Spatial Relationship), where even weak base models achieve non-trivial scores, then progressively introduce harder combinations (Single Arm Visual Reasoning, Dual Arm Multi-Entity Collaboration). The benchmark's per-task and per-embodiment scores provide a quantitative difficulty ordering that wasn't previously available. The finetuning improvements in Table 4 (largest absolute gains on Visual Reasoning: +0.093 and +0.050 for the two tested models) suggest that training on harder tasks is where the most progress remains to be made, supporting a curriculum that allocates more training time to difficult combinations.