Evaluating Gemini Robotics Policies in a Veo World Simulator

🎯 Pitch

A video generation model can substitute for real-world robot testing, accurately ranking policies and predicting generalization failures without ever touching hardware. The system red-teams safety by simulating dangerous scenarios—like closing a laptop on scissors—that would be too risky to stage physically, with predicted failures replicating in real robots.

1. Executive Summary

This report demonstrates that frontier video generation models can serve as generalist evaluators for robot policies across the full spectrum of evaluation use cases—from nominal performance assessment to out-of-distribution generalization to red teaming for physical and semantic safety—by building a generative evaluation system atop Veo, a latent diffusion video foundation model fine-tuned on large-scale robotics data. The system integrates action-conditioned multi-view video generation with generative image editing (via NanoBanana) and multi-view completion to synthesize realistic variations of real-world scenes along four axes of generalization—background changes, small distractors, large distractors, and novel manipulation objects—enabling closed-loop policy rollouts in simulated environments without requiring physical scene setup. Validated through 1600+ real-world trials across eight Gemini Robotics On-Device (GROD) policy checkpoints and five bimanual manipulation tasks, the system achieves strong rank consistency (MMRV of 0.03–0.15) and linear correlation (Pearson r of 0.56–0.91) between predicted and actual success rates, accurately predicting both the relative difficulty ordering of generalization axes and the comparative performance of different policies, while discovering safety violations—such as a robot closing a laptop on scissors or contacting a human hand—that replicate in hardware. The strongest predictive signal concentrates on background and distractor generalization axes and on policies with non-trivial baseline performance, establishing that video-model-based evaluation is a viable substitute for hardware testing primarily when the base policy retains some competence in the evaluated regime and the visual domain shift does not involve contact-rich manipulation of small novel objects.

2. Context and Motivation

The Core Problem: Evaluating Generalist Policies Requires Coverage That Hardware Alone Cannot Provide

The fundamental technical problem this paper confronts is deceptively simple: how do you thoroughly evaluate a robot policy that can, in principle, do almost anything? Generalist robot policies—those instructed via natural language to perform a wide variety of manipulation tasks across diverse environments—are becoming increasingly capable. But this very generality creates an evaluation crisis. To have confidence that such a policy is safe, reliable, and generalizes appropriately, you need to test it across a combinatorially large space of tasks, objects, environments, and edge cases. Traditional hardware evaluation, where you physically set up each scenario and run the robot, simply does not scale to this breadth.

The paper frames this through a concrete example in Section 1. Consider evaluating semantic safety: the ability of a policy to obey commonsense constraints like "don't close a laptop on scissors" or "don't place sharp objects where they could injure a human." To probe the "long tail" of such constraints—scenarios involving sharp objects near screens, plastics on stoves, broken glass left on floors—you would need to construct physical scenes that deliberately endanger the robot, its environment, or nearby humans. The paper makes the point bluntly: "Setting up real-world scenes that probe vulnerabilities of a policy in the 'long tail' of such constraints... can endanger the robot, its environment, and humans." This is not merely impractical; it is often ethically and physically infeasible. You cannot reliably test whether a robot will break a laptop by repeatedly closing it on scissors under controlled conditions—the hardware damage is real and cumulative.

Beyond safety, the scale problem extends to everyday evaluation for iterative policy development. When comparing multiple policy checkpoints—as is routine during training—statistically meaningful performance measurements require large numbers of trials across varied conditions. The paper cites Brohan et al. (2023) in noting that "resource requirements for statistically meaningful performance measurements of multitask robot policies expand to hundreds of thousands of expensive real-world evaluation trials." Each trial requires physical setup, execution time, and human supervision. This is the bottleneck that prevents rapid iteration: you cannot make frequent, data-driven decisions about which policy variant is better if each comparison requires a week of hardware time.

Why This Problem Matters: The Gap Between Policy Capability and Evaluation Capacity Is Widening

This problem matters because it represents a fundamental asymmetry in the development of generalist embodied agents. The policies themselves are scaling rapidly—larger models trained on larger datasets covering more manipulation skills, objects, and environments (as evidenced by the GROD policy trained on "a large-scale teleoperated robot action dataset collected over 12 months from a fleet of ALOHA 2 robots"). But the evaluation methodology has not kept pace. You can train a policy that can, in principle, follow thousands of different instructions involving hundreds of objects, but you cannot physically verify its behavior in more than a tiny fraction of those scenarios.

This asymmetry has practical consequences for three stakeholder groups:

Policy developers lack fast feedback loops. When evaluating a new training recipe, data mixture, or architecture change, they need to know within hours or days—not weeks—whether the change improved performance. Without scalable evaluation, development velocity is gated by hardware availability.
Safety engineers lack coverage. Semantic safety violations are by definition rare and surprising—they emerge from unusual combinations of instructions, objects, and scene layouts. Finding them through random hardware testing is like searching for needles in a haystack where most needles are invisible because you cannot ethically place them. The paper is explicit that for safety evaluation, "hardware evaluation is often simply infeasible."
Deployment decision-makers lack confidence bounds. When deciding whether a policy is ready for real-world deployment, you need to understand its failure modes and generalization limits. Without broad evaluation, you are deploying based on extrapolation from a narrow set of tested scenarios—a recipe for unexpected failures.

The theoretical significance is equally important. The field of robot learning has long grappled with the sim-to-real gap: the observation that policies trained or evaluated in simulation often behave differently on real hardware. This gap has been a central obstacle to using simulation as a substitute for real-world testing. If video models can serve as evaluators that predict real-world policy behavior with useful fidelity, it changes the fundamental economics of robot learning research. It means the evaluation bottleneck—which determines how quickly the field can iterate and improve—can be decoupled from physical hardware.

Prior Approaches and Where They Fall Short

The paper identifies three broad categories of prior evaluation approaches, each with specific limitations that motivate the video-model-based alternative.

Physics-Based Simulation

Traditional physics simulators have been the dominant approach to scaling evaluation beyond hardware. Numerous manipulation benchmarks (Liu et al., 2023; Pumacay et al., 2024; Wang et al., 2025) provide standardized simulation environments with defined tasks, success criteria, and simulated training datasets. The appeal is clear: you can run thousands of trials in parallel, test dangerous scenarios, and systematically sweep parameters. Li et al. (2024) demonstrated that real-to-sim environments—where simulation scenes are curated to match initial conditions from real-world evaluations—can provide predictive signal for real-world policy performance.

However, the paper identifies three specific failure modes of physics simulation that video models are uniquely positioned to address:

First, asset creation is a bottleneck. The paper states: "a wide range of realistic assets (e.g., laptop, sharp objects, etc.) need to be curated or created." For each new object you want to test—a plush toy, a teacup, a pair of scissors—someone must model its geometry, texture, material properties, and physical behavior. This is especially challenging for non-rigid objects like fabrics, cables, and deformable toys, which are precisely the kinds of objects that appear in semantic safety scenarios. The cost of creating a single high-quality asset can be substantial, and the diversity of objects that can be tested is gated by the available asset library. Video models, by contrast, learn object appearance and behavior from web-scale video data, potentially covering orders of magnitude more object categories without per-object engineering.

Second, accurate simulation of complex physics remains hard. Even with a perfect geometric model, simulating how a pair of scissors moves, how a fabric cloth drapes, or how a liquid sloshes is computationally expensive and often inaccurate. Contact-rich manipulation—precisely the kind that matters for evaluating whether a robot will knock over a glass or crush an object—is where physics simulators are weakest. The paper notes that "accurately simulating these assets can be very challenging, especially with non-rigid objects or humans." This means that even when assets exist, the simulated behavior may not match reality, undermining the validity of the evaluation.

Third, closing the visual gap requires months of expert effort. The paper references Badithela et al. (2025) and Li et al. (2024) to note that making simulated observations look like real camera images "can involve a months-long iterative process that requires significant human expertise (e.g., careful green screening) and effort." This domain randomization and texture matching is labor-intensive and must be redone for each new environment. Video models, trained on real video data, produce photorealistic outputs by construction.

The paper's position is not that physics simulation is worthless—indeed, Li et al. (2024) showed it can provide predictive signal for real-to-sim evaluation. Rather, the limitation is one of scalability: physics simulators are bottlenecked by asset creation, physics fidelity, and visual matching in ways that fundamentally limit how many different scenarios can be evaluated. You cannot easily test 100 novel objects if each requires a week of asset creation and physics tuning.

Video Generation Models (Prior Work)

The use of video models for robotics evaluation is not entirely new, but prior work has operated within a narrow scope. The paper cites several threads of related research:

Du et al. (2023) showed that fine-tuned video generation models can generate robot policy rollouts conditioned on high-level language instructions, demonstrating that video models can capture robot-environment interactions.
Action-conditioned world models have shown that generative video models can be conditioned not just on language but on low-level robot actions, expressed as explicit robot poses (NVIDIA, 2025; Russell et al., 2025) or latent actions (1XW, 2025; Bruce et al., 2024). This is a prerequisite for closed-loop policy evaluation: you need the video model to respond to the specific sequence of actions the policy produces.
Most directly relevant, recent works (Guo et al., 2025; Quevedo et al., 2025) have shown that action-conditioned world models can evaluate policies trained on real-world data on in-distribution tasks, "providing both relative and absolute signal on expected real-world policy performance."

The critical limitation of all this prior work is that it is restricted to in-distribution evaluation: scenarios similar to those used to train the policy or fine-tune the video model. The paper makes this explicit: "the use of video models in robotics has been limited primarily to in-distribution evaluations, i.e., scenarios that are similar to ones used to train the policy or fine-tune the base video model." This is a severe constraint because the most important evaluation questions—how does the policy generalize? where does it fail? is it safe in unusual situations?—are inherently about out-of-distribution behavior. If the video model can only simulate what it has already seen, it cannot serve as a generalist evaluator.

The paper identifies the specific technical challenges that have prevented video models from being used for OOD evaluation:

Artifacts in closed-loop action-conditioned generation: When a video model generates frame by frame conditioned on policy actions, errors compound. Small visual artifacts in early frames can cause the policy to produce different actions than it would in reality, which the video model then renders as further artifacts, creating a divergence between simulated and real trajectories.
Difficulty simulating contact dynamics: Video models struggle to accurately render the physical interaction between the robot's gripper and objects, especially small objects. Grasping, pushing, and releasing are notoriously difficult to capture because they involve precise geometry and physics that video models, trained on pixel-level objectives, do not explicitly model.
Multi-view consistency requirements: Modern robot policies use multiple cameras (e.g., top-down, side, wrist cameras) to mitigate partial observability. The video model must generate consistent observations across all viewpoints simultaneously. If the left wrist camera shows the gripper holding an object but the top-down camera shows it empty, the policy receives conflicting observations that can derail the rollout.

Heuristic OOD Prediction Without Full Rollouts

Majumdar et al. (2025) took a step toward addressing OOD evaluation by using image editing to generate variations of nominal scenes and making predictions about policy performance. However, their predictions were "made using a heuristic approach based on anomaly detection given only the first (edited) frames of episodes"—they did not simulate the policy interacting with the scene over time. This is a critical gap because many failures, especially safety failures, emerge from the interaction between the policy's actions and the environment over time. You cannot determine whether a robot will knock over a glass by looking only at the first frame; you need to see what the policy does.

The paper explicitly contrasts its approach with this prior work: "in contrast, we simulate policies for entire episodes using an action-conditioned video model." This is the key distinction: full closed-loop simulation that reveals dynamic failure modes, not just static scene compatibility.

Safety Evaluation (Prior Work)

The paper situates its safety contributions relative to a broader landscape of safety evaluation research. Existing safety evaluation approaches fall into two categories, both of which the paper argues are insufficient:

Non-interactive benchmarks. A large body of work evaluates safety by presenting text, images, or videos to a model and asking it to reason about safety constraints. Bianchi et al. (2023) and Zhang et al. (2023) created text-only benchmarks for LLMs. Zhang et al. (2024) extended this to multimodal VLM evaluation. Most directly relevant to the robotics domain, Sermanet et al. (2025) proposed the ASIMOV benchmark—a large-scale collection of scenarios grounded in real-world scenes and injury reports from hospitals—and ASIMOV-2.0 (Jindal et al., 2025) expanded to include videos and physical constraint reasoning. These benchmarks have been used to evaluate the Gemini Robotics embodied reasoning models.

However, the paper identifies a fundamental limitation: "All evaluation benchmarks highlighted above are non-interactive in nature—text, images, or videos are provided as input to a language model in order to assess safety." In such benchmarks, the model makes a single judgment given the initial scene and instruction. This cannot capture the dynamics of closed-loop behavior where "actions that the robot takes at one time-step have implications for safety in future time-steps." For example, whether a robot will safely handle a knife depends not just on recognizing that knives are dangerous (which a non-interactive benchmark can test), but on whether the policy generates appropriate grasp poses and trajectories over the course of execution (which only interactive evaluation can test).

Concurrent work on world models for safety. The paper acknowledges that concurrent work (Wayve, 2025) in autonomous driving provides a complementary demonstration of world modeling and scene editing for safety evaluation. This suggests the approach is gaining traction across robotics domains, but the paper's contribution is positioned as (a) focused on manipulation rather than driving, and (b) covering the full evaluation spectrum from nominal to OOD to safety, not just safety alone.

How This Paper Positions Itself

The paper positions itself as bridging two gaps simultaneously: the gap between in-distribution and out-of-distribution video model evaluation, and the gap between non-interactive and closed-loop safety assessment.

The central thesis, stated in the abstract and Section 1, is that "video models can be used for the entire spectrum of policy evaluation use cases in robotics: from assessing nominal performance to out-of-distribution (OOD) generalization, and probing physical and semantic safety" (emphasis mine). The word "entire" is doing heavy lifting here—the claim is not incremental improvement on prior video model evaluation, but a qualitative expansion of scope that covers use cases previously thought to require physical simulation or hardware testing.

Within this framing, the paper's specific contributions are presented as demonstrations of capability rather than novel algorithmic contributions:

For nominal evaluation: extending prior action-conditioned world model work (Guo et al., 2025; Quevedo et al., 2025) to a frontier video model (Veo) with multi-view consistency and action conditioning at the level of robot poses, validated against a larger set of policies and tasks than prior work.
For OOD evaluation: combining generative image editing (to create novel scenes) with multi-view completion (to fill in unedited camera angles) and closed-loop policy rollouts, validated against matched real-world OOD scenes. This is the key methodological extension beyond prior in-distribution video model work.
For safety evaluation: using the same image editing + video rollout pipeline to systematically generate safety-relevant scenarios and discover policy behaviors that would be dangerous to test on hardware. This extends beyond prior non-interactive safety benchmarks by enabling dynamic safety assessment.

The paper positions its technical approach as building on existing component technologies—Veo for video generation, NanoBanana for image editing, GROD for the policy—and focusing on the system integration that makes the evaluation pipeline work end-to-end. This is an engineering contribution rather than a scientific one, but the paper argues that the capability demonstration itself is significant: showing that these components can be assembled into a system that produces evaluation signals correlated with reality across a broad range of conditions.

A crucial positioning choice is that the paper does not claim the video model is a perfect simulator. Section 7 is explicit about limitations: contact-rich interactions with small objects remain challenging (Fig. 11 shows hallucination where an object appears spontaneously), episodes are limited to 8 seconds, and absolute predicted success rates are lower than real rates. Instead, the claim is about relative predictive signal: the system can rank policies, rank generalization axes, and identify safety failures, even if absolute performance estimates are miscalibrated. This relative-signal framing is important because it sets realistic expectations: you cannot use the system to certify that a policy achieves 95% success, but you can use it to determine that policy A is better than policy B, or that changing the background is more disruptive than adding a distractor.

Finally, the paper positions itself within a broader vision of video models as "generalist evaluators for generalist policies." The parallelism is deliberate: just as generalist policies aim to handle diverse tasks through a single model rather than task-specific controllers, video model evaluators aim to simulate diverse scenarios through a single model rather than scenario-specific physics simulators. The paper presents its results as an early demonstration of this vision—"while we are still in the early days of video modeling for robotics... this report demonstrates a path towards scalable evaluation"—acknowledging that the current system has limitations but arguing that the trajectory is toward increasingly capable and generalizable evaluation.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The system is a video generation pipeline that simulates how a robot policy would behave in a given scene, producing a predicted video of the robot's actions and their consequences without requiring the robot to physically execute them. It solves the problem of evaluating robot policies at scale by substituting real-world hardware trials with computationally generated video rollouts that are photorealistic, multi-view consistent, and responsive to the fine-grained sequence of actions the policy would produce.

3.2 Big-Picture Architecture (Diagram in Words)

The evaluation system has five major components arranged in a pipeline:

Scene Generator — takes a real-world robot observation image and a desired change (e.g., "add a pink brush on the table") and produces an edited overhead image using a generative image editing model (NanoBanana). This is the entry point for OOD and safety evaluation; for nominal evaluation, unedited real images are used directly.
Multi-View Synthesizer — takes the single edited (or real) overhead image and generates three additional camera viewpoints (side view, left wrist, right wrist) to create a complete multi-view observation, using a fine-tuned version of Veo2 trained for single-view-to-multi-view completion.
Policy Model (GROD) — the vision-language-action policy being evaluated. Given the multi-view observation and a task instruction, it predicts a sequence of robot poses (a 1-second action chunk at 50 Hz) that the robot would execute. This is the "subject under test" — the thing whose performance the system is measuring.
Veo (Robotics) Video Generator — the core simulator. Given an initial multi-view frame, a text instruction, and a sequence of future robot poses (from the policy), it generates a sequence of future multi-view frames showing what would happen if the robot followed those poses. It is a latent diffusion model fine-tuned on large-scale robotics data for action-conditioned, multi-view consistent generation.
Human Evaluator (or Future VLM) — scores each generated video rollout as a binary success or failure based on whether the task was completed. In the current system, this is a human watching the generated video; the paper notes that VLM-based automated scoring is planned for future work.

Information flow: A scene description enters → the scene generator creates an edited overhead image (if OOD/safety evaluation) → the multi-view synthesizer fills in the other camera views → the policy takes this multi-view observation plus the task instruction and produces a trajectory of robot poses → the Veo (Robotics) model generates a video of the robot executing those poses in the scene → a human (or VLM) scores the video as success or failure → the binary scores are aggregated across multiple scene variations to estimate the policy's success rate for a given condition.

3.3 Roadmap for the Deep Dive

First, the Veo base video generation model — its architecture, training data, and pretraining procedure — because all video generation capabilities in the system derive from this foundation.
Second, the action-conditioning fine-tuning on robotics data — how the pretrained Veo is adapted to accept robot pose sequences and produce robot-specific video — because this is what transforms a general video model into a robotics simulator.
Third, the multi-view generation mechanism — how the model produces consistent observations across four cameras simultaneously — because multi-view consistency is essential for the policy to receive realistic observations during closed-loop rollout.
Fourth, the closed-loop evaluation protocol — how the policy and the video model interact at inference time, including episode length, action chunking, and scoring — because this is the actual evaluation procedure whose fidelity determines the quality of predictions.
Fifth, the out-of-distribution scene generation pipeline — how image editing and multi-view synthesis create novel evaluation scenarios along the four generalization axes — because this is the key methodological contribution that enables evaluation beyond in-distribution nominal tasks.
Sixth, the safety red-teaming procedure — the scenario generation, filtering, and rollout process for discovering unsafe behaviors — because this represents the most novel application domain and has distinct technical requirements.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems demonstration paper whose core idea is that a frontier video foundation model, fine-tuned on robotics data and integrated with generative image editing, can serve as a general-purpose simulator for evaluating robot policies across nominal, OOD, and safety-critical regimes without the per-scenario asset creation and physics tuning required by traditional simulators.

Veo2 Base Model Architecture

The foundation of the evaluation system is Veo2, a text-to-video latent diffusion model described in the Veo technical report (Veo Team, 2025). Understanding its architecture is necessary for understanding what capabilities the robotics fine-tuning inherits and extends.

Latent diffusion framework. Veo operates in a compressed latent space rather than directly on pixel sequences. The architecture uses three components:

Spatio-temporal autoencoders that compress raw video frames into a lower-dimensional latent representation. The paper states that these autoencoders "compress spatio-temporal data into smaller, more efficient latent representations." This means that instead of learning to generate individual pixel values across space and time—which would be computationally prohibitive for high-resolution, multi-second videos—the model learns to generate in a compressed feature space where semantically relevant structure is preserved but spatial and temporal redundancies are removed. The autoencoder is trained to reconstruct original videos from their latent encodings, ensuring that the compression is lossy only with respect to high-frequency details, not semantic content.
A transformer-based denoising network trained to remove noise from latent vectors. The paper states: "A transformer-based denoising network is then trained to remove noise from these latent vectors." This is the diffusion model proper. During training, Gaussian noise is progressively added to clean latent representations according to a noise schedule, and the transformer learns to predict the added noise given the noisy latent and conditioning information (text descriptions, and in the fine-tuned case, actions). The transformer architecture provides the capacity to model long-range spatio-temporal dependencies—which is essential for video generation because a pixel's value at frame t may depend on events that occurred several frames earlier, and a region of one frame may depend on motion happening in a different spatial region in a previous frame.
An iterative sampling procedure that converts random noise into a generated video. The paper states: "To generate a video, the model iteratively applies this denoising network to a random noise input, refining it into the final video output." Starting from pure Gaussian noise in the latent space, the model repeatedly applies the denoising transformer, each time moving the latent representation one step closer to a clean signal. The number of denoising steps is a hyperparameter that trades generation quality against computational cost; the paper does not specify the exact number of steps used during inference.

Why a latent diffusion architecture? The paper does not explicitly justify this architectural choice, but it follows from standard practice in video generation. Pixel-space diffusion is computationally prohibitive for video because the dimensionality is enormous (frames × height × width × channels). Latent diffusion reduces the effective dimensionality by 4–8× in each spatial dimension via the autoencoder, making training and inference tractable on current hardware accelerators. The transformer backbone (as opposed to a convolutional U-Net, which was standard in earlier diffusion models) provides the self-attention mechanism that can capture long-range dependencies across both space and time without requiring recurrent computation that would scale linearly with sequence length.

Training data and curation. The pretraining data for Veo is described as "a large dataset of videos, images, and associated annotations." The scale is not quantified in this paper (readers are referred to the Veo technical report), but the description emphasizes several curation steps that are relevant to the model's behavior as a robotics simulator:

Text captions are "generated at different levels of detail using multiple Gemini models." This multi-level captioning means the model learns to associate visual content with both coarse descriptions (e.g., "a robot arm") and fine-grained descriptions (e.g., "a silver robotic gripper grasping a red plastic cube at the corner of a wooden table"). This multi-granularity conditioning is important for robotics because the model needs to respond to both high-level task instructions and low-level action specifications.
The data is filtered "for quality and to remove unsafe content and personally identifiable information." The safety filtering is relevant to the model's use for safety evaluation: if the pretraining data were unfiltered, the model might have learned to generate unsafe behaviors as a default rather than as a consequence of specific conditioning.
The data is "semantically deduplicated" to prevent memorization. The paper states this is done "to prevent the model from overfitting or memorizing specific training examples." This is important for OOD evaluation: if the model had memorized specific training videos, its OOD predictions might be systematically biased toward replicating those memorized examples rather than genuinely simulating novel scene configurations.

The pretraining dataset is web-scale and covers diverse visual domains, which gives the model broad prior knowledge about object appearance, physics, and scene composition. When the model is later fine-tuned on robotics data, it brings this general visual knowledge with it—this is the fundamental bet of the approach: that the pretrained model's ability to simulate general visual phenomena will transfer to the specific domain of robot manipulation.

Action-Conditioned Fine-Tuning on Robotics Data

The pretrained Veo2 model can generate videos from text descriptions, but it does not know about robot control. To transform it into a robotics simulator, the model is fine-tuned on a "large-scale robotics dataset consisting of diverse tasks that cover a broad range of manipulation skills across a multitude of scenes."

What the fine-tuned model accepts as input. At inference time, the model takes two conditioning signals:

A current image observation of the scene — specifically, the first frame from each of the robot's four cameras, tiled into a single multi-view image. This provides the visual initial condition from which the video generation starts. The model is not generating from scratch; it is generating from a grounded initial state, which is why the method is described as a "simulator" rather than a generative model of arbitrary robot behavior.
A sequence of future robot poses — the commanded positions the robot will move through over the course of the episode. The paper states the model "can be conditioned on a current image observation of the scene and a sequence of future robot poses." The poses are specified as explicit numerical values (joint angles or end-effector positions; the paper does not specify the exact parameterization), not as latent actions or text descriptions. This is in contrast to some prior world model work (Bruce et al., 2024) that conditions on learned latent action representations. Explicit pose conditioning has the advantage that the policy's output can be fed directly into the video model without any learned mapping, but it requires the video model to learn the relationship between commanded poses and visual outcomes in pixel space—a more demanding task than latent action conditioning because the mapping from poses to pixels is complex (it involves robot kinematics, contact dynamics, and object interactions).

What the model produces. Given the initial frame and action sequence, the model generates "a sequence of future images that correspond to the future robot poses and observations." The output is a multi-view video—a temporal sequence of tiled frames, each containing four camera views updated simultaneously. Fig. 2 (top) shows an example with rendered poses overlaid on the generated video to illustrate the correspondence between commanded actions and visual output.

The robotics fine-tuning dataset. The fine-tuning data consists of "diverse tasks that cover a broad range of manipulation skills across a multitude of scenes." While specific dataset statistics are not provided, the description emphasizes diversity: varied manipulation skills (grasping, placing, pushing, handover), varied objects, varied task difficulties, varied episode horizons, and varied dexterity requirements. This diversity is crucial for the model's ability to simulate OOD scenarios: if the fine-tuning data only covered a narrow set of objects and actions, the model would likely fail when asked to simulate novel objects or actions, because it would have no training signal for how those objects behave in response to robot actions.

The paper does not provide details on the fine-tuning procedure itself—the loss function, learning rate, number of steps, batch size, or other standard training hyperparameters. This is a notable omission for a technical analysis, as the fine-tuning procedure can substantially affect the model's behavior (e.g., overfitting to the fine-tuning distribution vs. retaining pretraining knowledge). The description only states that the model is "finetuned," which could mean anything from full model fine-tuning to parameter-efficient methods like LoRA. Given that the model retains the ability to generate diverse scenes that were not in the robotics dataset (as demonstrated by the OOD results), it is likely that catastrophic forgetting of pretraining knowledge was avoided, but the specific mechanism is not disclosed.

How the fine-tuned model differs from the pretrained model. The key capability added by fine-tuning is the ability to generate videos where the motion of the robot and the response of objects is causally linked to the commanded action sequence—not just statistically correlated. The pretrained Veo can generate plausible robot motion from text descriptions because it has seen many videos of robots, but it does not know that "robot moves gripper 5 cm left" should produce a specific, deterministic visual change. The fine-tuning teaches the model that commanded poses constrain the visual output in a particular way. However, the paper does not claim the model learns this constraint perfectly. Fig. 11 shows a failure case where "a novel object appears spontaneously while the gripper is interacting with a different object," indicating that the action-to-pixel mapping is probabilistic and sometimes generates implausible outcomes—the model sometimes generates what is visually common in the training data rather than what is physically mandated by the commanded action.

Multi-View Generation

Modern bimanual manipulation policies rely on multiple camera viewpoints to handle occlusions and provide depth information. The ALOHA 2 platform used in this work has four cameras: a top-down view, a side view, and left and right wrist-mounted cameras. The video model must generate consistent observations across all four views simultaneously, meaning that the same physical scene state must be represented from four different viewpoints at every time step.

The tiling approach. Rather than generating four independent videos and hoping they are consistent, the paper uses a tiling strategy: "we tile the four observations across four cameras in our setup, including the top-down view, the side view, and the left and right wrist view." The four camera images are concatenated into a single larger image (a 2×2 grid), and the Veo (Robotics) model is fine-tuned to generate this tiled format. Fig. 2 (bottom) shows an example of the tiled output.

Why tiling works. By treating the multi-view output as a single image with spatially separated regions, the model's self-attention mechanism can directly attend between corresponding regions in different camera views. For example, when generating the wrist camera view of the gripper closing on an object, the model can attend to the top-down view's representation of the same physical event, enforcing consistency. If the model instead generated each view independently, there would be no mechanism for cross-view consistency—the left wrist view might show the gripper closed while the top-down view shows it open.

This tiling approach also simplifies the generation pipeline: a single forward pass through the model produces all four views simultaneously, rather than requiring four separate generation passes. This reduces inference cost and avoids the need for explicit consistency losses or post-hoc alignment of independently generated views.

Limitations of the tiling approach. The paper does not discuss this explicitly, but the tiling approach inherits the spatial resolution limits of the base model. Each camera view's effective resolution is reduced because four views must fit within the same total pixel budget that was designed for a single video. If the base model generates at, say, 512×512 resolution, each camera view in a 2×2 grid gets approximately 256×256 pixels after accounting for borders. This resolution tradeoff may be acceptable for policy evaluation if the policy can operate at reduced resolution, but it could mask visual details that are important for certain tasks (e.g., reading small text, identifying thin cables).

Closed-Loop Evaluation Protocol

The evaluation of a policy is not a single forward pass through the video model. It is a closed-loop process where the policy and the video model interact over the course of an episode, mirroring how the policy would execute on real hardware.

Episode structure. Each evaluation episode has a fixed duration of 8 seconds. The paper explicitly limits episodes to this duration: "the policy rollouts in this work correspond to 8-second episodes." The video model generates the full 8-second video in a single generation pass (the model does not generate frame-by-frame in an autoregressive loop; it generates the entire video sequence simultaneously, as is standard for diffusion-based video models).

Action chunking and re-planning. The GROD policy predicts "a 1-second action chunk with continuous actions at 50 Hz." This means the policy outputs 50 action values (joint angles, end-effector positions, or similar) that specify the robot's motion for the next 1 second. At a nominal control rate of 50 Hz, the policy must produce a new action every 20 milliseconds. However, the policy described here uses "asynchronous policy execution"—the policy runs its forward pass to compute a 1-second chunk, and while the robot executes that chunk, the policy asynchronously computes the next chunk. This means the policy is not waiting for the video model to generate frames; the action sequence for the entire 8-second episode is computed in advance or in overlapping computation windows, and then the video model renders the resulting trajectory.

How the policy and video model interact at test time. The procedure for a single evaluation episode proceeds as follows:

The initial multi-view observation (edited or real) and the task instruction are fed to the GROD policy.
The policy produces an action chunk for the next 1 second (50 action values).
The process repeats, with the policy producing subsequent action chunks based on the initial observation (for open-loop evaluation, where the policy plans the entire trajectory from the initial frame) or based on generated frames (for closed-loop evaluation, where the policy observes its own predicted actions). The paper does not specify whether the evaluation uses open-loop or closed-loop replay. If the policy computes all action chunks from the initial frame, it is open-loop and any errors in the video model's rendering of early actions do not affect later policy decisions. If the policy observes generated frames and replans, it is closed-loop and errors can compound—the policy may react to artifacts in the generated video by producing different actions than it would in reality. This distinction is critical for understanding the failure modes of the evaluation, but the paper does not address it.
The full 8-second action sequence (8 chunks of 50 actions each, for 400 total action values) is fed as conditioning to the Veo (Robotics) model, along with the initial multi-view frame and the task instruction.
The Veo (Robotics) model generates an 8-second multi-view video.
A human evaluator watches the generated video and scores it as success or failure based on whether the task was completed according to the binary success metric appropriate for that task.

Why 8 seconds? The paper does not justify the 8-second horizon, but it is consistent with the typical duration of the manipulation tasks shown in Fig. 3. Pick-and-place operations on a tabletop with a bimanual robot typically take 5–15 seconds, so 8 seconds is a representative horizon. However, Section 7 notes that "achieving long-horizon (e.g., 1+ minutes) multi-view consistent generation remains a key technical milestone," indicating that the 8-second limit is a capability constraint of the current video model, not a deliberate design choice. Extending the horizon would require the video model to maintain spatio-temporal consistency over longer sequences, which is challenging for diffusion models because the latent space must encode increasingly more information as the sequence length grows.

Scoring. Success is a binary metric defined per task. The paper does not provide the scoring rubric for each of the five tasks, but examples from Fig. 3 suggest the criteria: for "Put the top right red grapes into the top left compartment of the grey box," success means the red grapes end up in the specified compartment; for "Put the lego into the lego bag," success means the Lego piece is inside the bag; for "Put banana in bowl with a handover," success means the banana ends up in the bowl after a handover between the two arms. Human evaluators apply these criteria to the generated videos. The paper notes that future work will integrate "automated scoring based on vision-language models (VLMs)" to remove the human bottleneck.

Aggregation. For a given policy, task, and scene condition, multiple episodes are run (the paper does not specify the exact number per condition, but the total of 1600+ real-world trials across all conditions suggests ~20 trials per condition for the 80 scene-instruction combinations in nominal evaluation, and fewer for OOD conditions). The predicted success rate is the fraction of scored rollouts that are successes. This predicted success rate is compared to the real-world success rate obtained by running the same policy on the same (or matched) scene on physical hardware.

Out-of-Distribution Scene Generation Pipeline

The key methodological contribution that enables OOD evaluation is the pipeline for generating scenes that differ from the training distribution along controlled axes. This pipeline has three stages: single-view editing, multi-view synthesis, and instruction adaptation.

Stage 1: Single-view image editing (NanoBanana). The starting point is a nominal RGB observation from the robot's overhead camera—a real photograph of the robot's workspace with the standard objects, distractors, and background. To create an OOD scene, this image is edited using Gemini 2.5 Flash Image (referred to as NanoBanana). The paper describes the editing process: "We use Gemini 2.5 Flash Image (a.k.a. NanoBanana) to generate this edited scene using a language description of the desired change."

The language description specifies which axis of generalization to modify:

Background: "add a red cloth behind the table" or equivalent for green/blue. The editing model must segment the background region, replace it with a plausible cloth texture, and maintain consistent lighting and shadows with the rest of the scene.
Small distractor: "add a purple octopus plush toy on the table" or equivalent for each of the five small distractors. The editing model must insert the object into a semantically plausible location on the table (not floating, not occluding the task-critical objects, not intersecting the robot), render it with appropriate scale and perspective, and generate realistic shadows and reflections.
Large distractor: "add a polar bear plush toy next to the table" for each of the five large distractors. The large distractors are placed near but not on the table, so the editing model must handle object insertion into background regions with appropriate occlusion by table edges.
Object: "replace the banana with a pink plastic kitchen brush" or equivalent for each of the five novel objects. This is the most challenging edit because it requires (a) removing the original manipulation target, (b) inserting the novel object in approximately the same location, (c) adjusting the task instruction accordingly, and (d) ensuring the novel object's visual properties (texture, specular highlights, shadows) are consistent with the scene lighting.

The choice of NanoBanana as the editing model is pragmatic: Gemini 2.5 Flash Image is a state-of-the-art text-conditioned image editing model that can produce photorealistic edits from natural language instructions. The paper does not discuss alternatives (e.g., inpainting models, ControlNet-based approaches) or the specific prompting strategy used.

Stage 2: Multi-view synthesis. The editing produces only a single overhead image, but the policy requires four camera views. Three additional views must be generated: side view, left wrist, and right wrist. This is performed by "a version of Veo2 that is fine-tuned to predict multi-view images from a single-view image." Fig. 7 shows the input (edited overhead view) and output (four views tiled together). This is a separate fine-tuned model from the main Veo (Robotics) model: it takes one image as input and generates the other three, rather than taking an initial frame and action sequence and generating a video. The fine-tuning objective is presumably reconstruction-based (generate the other three views given one view from real multi-view data), but the paper does not provide training details.

The multi-view synthesis step is the point where errors can be introduced. If the edited overhead image contains an object that the multi-view model has never seen from other angles (e.g., a novel plush toy), the model must hallucinate what that object looks like from the side and from the wrist cameras. The quality of this hallucination depends on how well the model generalizes from single-view to multi-view for novel objects. The paper does not ablate this step or quantify the error it introduces, but it is an implicit assumption of the OOD evaluation pipeline that the multi-view synthesis is sufficiently accurate that the policy's behavior is not dominated by multi-view artifacts.

Stage 3: Instruction adaptation. The task instruction must be updated to reflect the edited scene. For example, if the banana is replaced with a pink brush, the instruction changes from "put banana in bowl with handover" to "put pink brush in bowl with handover." The paper does not describe this as an automated step, but the examples suggest that the instruction is manually edited to match the visual change. This is an important detail because in a fully automated evaluation pipeline, the instruction would need to be generated programmatically. If the instruction and the visual scene are inconsistent (e.g., the instruction says "banana" but the scene shows a brush), the policy's behavior is undefined, and the evaluation would measure instruction-scene consistency rather than manipulation capability.

What the pipeline produces. The output of the OOD scene generation pipeline is a complete initial condition for a policy rollout: four camera views (one real + three synthesized) that are visually consistent, and a task instruction that matches the visual scene. This initial condition is then fed to the closed-loop evaluation protocol described above, producing a predicted success rate for that policy on that OOD condition.

Validation of the pipeline: replicating edited scenes in reality. To validate that the predictions from the video model correspond to real-world outcomes, the paper replicates the edited scenes as closely as possible on physical hardware. Fig. 6 shows side-by-side comparisons of synthetically edited scenes and their real-world counterparts. The real-world replication involves physically placing the novel objects, distractors, or backgrounds as depicted in the edited images, then running the policy on hardware and measuring actual success rates. The fact that the synthetic and real scenes are visually similar but not identical (see Fig. 6) introduces a source of variance in the correlation metrics: the predictions are for the synthetic scene, but the validation is on the physically replicated scene, which may differ in subtle ways (lighting, exact object placement, camera calibration) that affect policy behavior.

Safety Red-Teaming Pipeline

The safety evaluation pipeline extends the OOD scene generation framework with additional mechanisms for generating and filtering safety-relevant scenarios.

Scenario generation. The paper describes generating "a set of scenarios for an ALOHA tabletop environment" consisting of paired (user request, image) instances. The images are synthetically edited to include safety-relevant elements: sharp objects, hot objects, fragile objects, human hands in the workspace, or obstacles near drop-off locations. The user requests are designed to create ambiguity about which object to manipulate or where to place it, so that the policy's safety judgment is tested.

Scenario filtering with Gemini 2.5 Pro as critic. Not all generated scenarios are useful for safety evaluation. The paper uses Gemini 2.5 Pro as an automated filter that applies four inclusion criteria:

Hazard presence: "The scene includes at least one object that is potentially unsafe for the robot to manipulate (e.g., a sharp knife, a hot object, a full glass)." Scenes without hazards are safety-irrelevant and are discarded.
Multimodal reasoning requirement: "The task's safety constraints can only be resolved by using both the image and the user request." This criterion ensures that the safety challenge cannot be solved by a text-only safety filter or an image-only hazard detector. For example, "pick up the knife" is unsafe if the scene contains only a knife because the text alone flags the hazard; but "pick up the bottle" in a scene with both a safe bottle and an unsafe knife requires image inspection to determine which bottle is meant and whether the knife is in the path. The paper wants scenarios where the multimodal interaction is essential, so that rolling out the policy is genuinely necessary for determining safety outcomes—otherwise, simple text filtering could catch the issue.
Ambiguity: The scenario must introduce one of four types of ambiguity: object ambiguity (imprecise request in a scene with multiple candidate objects, including unsafe ones), destination ambiguity (imprecise drop-off location with unsafe destinations), trajectory/affordance ambiguity (object requires specific grasp or carry orientation for safety), or human interaction (human in workspace, creating injury risk from incorrect action).
Non-triviality: The paper describes these as "non-trivial" scenarios "suitable for safety evaluation," implying that trivial cases (e.g., "pour bleach on laptop," which is obviously unsafe regardless of execution) are filtered out because they do not require closed-loop simulation to assess. The filtering ensures that the retained scenarios are ones where "rolling out the policy is necessary for determining safety."

This filtering step is an important architectural choice. Without it, the pipeline would generate many safety-irrelevant or trivially unsafe scenarios that waste evaluation compute and produce uninformative results. By using an LLM as a filter, the system focuses computational resources on the scenarios where video-based simulation provides unique information: cases where safety depends on the dynamic execution of the policy, not just on static scene analysis.

Rollout and evaluation. For each filtered scenario, the policy is rolled out using the Veo (Robotics) model exactly as in the OOD evaluation pipeline: the generated scene provides the initial multi-view observation, the policy produces action chunks, and the video model generates the full episode video. The output is then inspected (by humans, in the current system) for unsafe behaviors: robot contacts human, robot closes gripper on fragile object, robot moves sharp object near screen, etc.

Validation on hardware. The paper validates the safety predictions by physically replicating the generated scenes with real props and running the policy on hardware. Fig. 10 shows two examples where this replication confirms the unsafe behavior predicted by the video model: (1) the robot moves toward a red block but makes contact with a human hand, and (2) the robot closes a laptop without first moving scissors away from the keyboard, risking screen damage. The fact that the unsafe behaviors replicate in hardware is the key validation that the video model is not merely generating visually plausible but physically inaccurate behaviors—it is predicting actual policy failure modes.

Design Choices and Justifications

Why fine-tune rather than train from scratch? The paper leverages a pretrained Veo2 model rather than training a video model on robotics data alone. This is justified by the scale argument: web-scale pretraining provides broad knowledge about object appearance, physics, and scene composition that would be impossible to learn from a robotics dataset alone (which is necessarily limited in object diversity and environmental variation). The fine-tuning then adapts this general knowledge to the specific domain of robot action-conditioned generation. This is the standard transfer learning paradigm, but it carries the risk that the pretraining distribution (general web video) and the fine-tuning distribution (robot lab video) are mismatched in ways that cause the model to generate unrealistic robot behavior. The paper's results suggest the fine-tuning is sufficient to overcome this mismatch for the evaluated tasks, but the risk would increase for more extreme OOD conditions.

Why learn a generative model rather than a discriminative predictor? An alternative to the video generation approach would be to train a model that directly predicts binary success/failure given the initial scene and policy actions, without generating intermediate video frames. This would be computationally cheaper and might avoid the accumulation of visual artifacts. However, the paper chooses the generative approach for two implicit reasons: (1) generated videos are interpretable—humans can watch them to understand failure modes qualitatively, as demonstrated by the insight that Policy A fails due to instruction following rather than manipulation when given novel objects; and (2) the generative objective forces the model to learn detailed physics and contact dynamics that a discriminative predictor might ignore, potentially leading to more robust evaluation predictions. The paper does not explicitly justify this choice, but the qualitative failure mode analysis demonstrates the value of interpretable intermediate representations.

Why use human scoring rather than automated metrics? The current system relies on humans watching generated videos and applying binary success criteria. This is a limitation for scalability—the paper explicitly calls out VLM-based automated scoring as future work. The choice to use human scoring in this report reflects a prioritization: first validate that the video model's outputs are sufficiently realistic that human judgments of those outputs correlate with real-world outcomes, then automate the scoring once the generation fidelity is established. If automated scoring had been used from the start and results were poor, it would be unclear whether the problem was in the video generation or the automated scoring. By using human scoring, the paper isolates the generation fidelity as the variable of interest.

4. Key Insights and Innovations

Innovation 1: Video Models Can Serve as Generalist Evaluators Across the Full Spectrum of Evaluation Regimes, Not Just In-Distribution

The paper's most distinctive conceptual contribution is the demonstration that a single video generation model—properly fine-tuned and integrated with generative editing—can cover evaluation use cases that were previously handled by entirely separate methodologies: nominal performance assessment, out-of-distribution generalization analysis, and safety red-teaming. This unification is not an incremental scaling of prior video model evaluation; it represents a qualitative expansion of what video models are understood to be capable of in the robotics evaluation pipeline.

What changed from prior work. Before this paper, the use of video models in robotics evaluation was strictly in-distribution. Prior work (Guo et al., 2025; Quevedo et al., 2025; Du et al., 2023) demonstrated that action-conditioned world models could predict real-world policy performance on tasks and scenes similar to those in the training data. The field's operating assumption was that video models, being learned from finite datasets, would necessarily degrade when asked to simulate scenarios outside their training distribution—novel objects, unusual backgrounds, safety-critical edge cases. The dominant view was that OOD evaluation required either physics simulation (where the laws of physics, once encoded, generalize to novel object configurations) or heuristic approaches applied to static scenes (Majumdar et al., 2025, which predicted performance from edited first frames without simulating policy interaction).

This paper challenges that assumption directly. By integrating generative image editing with multi-view completion and closed-loop action-conditioned rollouts, the system simulates policies interacting with scenes containing objects, backgrounds, and distractors that were not present in the fine-tuning data. The fact that predictions from these simulations correlate with real-world outcomes (Pearson r of 0.56–0.91 across generalization axes, as shown in Fig. 9) demonstrates that the video model's pretraining on web-scale data provides enough general visual-physical knowledge to extrapolate beyond the robotics fine-tuning distribution in useful ways.

Why this is a reframing, not just a better model. The significance is not that Veo generates nicer-looking videos than prior models—it's that the evaluation methodology itself shifts. Previously, evaluating generalization required either (a) building a physics simulator with per-scenario asset creation, (b) running hardware tests, or (c) using static heuristics that cannot capture dynamic failure modes. This paper proposes a fourth path: a learned simulator that generalizes across scenes without per-scenario engineering. The reframing is from "video models can reproduce what they've seen" to "video models can simulate what they haven't seen, well enough to predict policy behavior." This changes the role of video models in the robotics stack from a visualization tool to a first-class evaluation infrastructure.

The safety dimension amplifies the significance. For safety evaluation specifically, the unification is particularly striking. Prior safety benchmarks (Sermanet et al., 2025; Jindal et al., 2025; Zhang et al., 2024) are non-interactive: they test whether a model can reason about safety given a static scene and instruction. They cannot test whether a policy will execute safely because that depends on the dynamic interplay of actions and environment over time. The paper identifies this gap explicitly in Section 6, noting that prior benchmarks "are non-interactive in nature" and therefore cannot capture cases "where actions that the robot takes at one time-step have implications for safety in future time-steps." By demonstrating that the video model can discover safety violations—closing a laptop on scissors, contacting a human hand—that replicate in hardware (Fig. 10), the paper shows that video-based evaluation provides a form of safety signal that was previously only obtainable through physical testing or detailed physics simulation. This is not an incremental improvement on non-interactive benchmarks; it is a different category of safety assessment altogether.

Tie to evidence. Fig. 4 (nominal evaluation) and Fig. 9 (OOD generalization) provide the quantitative foundation for this claim, showing consistent rank correlation and linear correlation between predicted and real success rates across conditions. Fig. 10 provides the qualitative evidence for the safety dimension. The fact that the same underlying model—Veo (Robotics)—produces all three types of signal without architectural modification is what makes this a unified evaluation framework rather than three separate systems.

A note on distinguishing incremental from fundamental. The individual components—action-conditioned video generation, image editing, multi-view completion—are each drawn from prior work. What is fundamental is the system-level demonstration that these components, when integrated, produce an evaluation capability that spans regimes previously requiring disjoint methodologies. This is an engineering contribution with conceptual implications: it changes the answer to "how should we evaluate generalist policies?" from "with a combination of hardware, physics simulation, and static benchmarks" to "with a video foundation model fine-tuned on robotics data."

Innovation 2: Difficulty-Conditioned Evaluation as an Alternative to Absolute Accuracy

A second distinctive insight—one that runs through the entire results section but is never stated as a formal thesis—is that video model evaluation is best understood as providing relative rather than absolute signal, and that this relative signal is sufficient for most practical evaluation use cases. The paper consistently reports two findings: strong rank correlation (low MMRV, high Pearson r) and systematically lower absolute predicted success rates compared to real rates. Rather than treating the absolute miscalibration as a failure, the paper implicitly reframes it as an acceptable tradeoff because relative comparisons answer the questions that matter most during policy development.

What the field assumed before. Prior evaluation work—both in simulation (Li et al., 2024; Badithela et al., 2025) and with video models (Guo et al., 2025; Quevedo et al., 2025)—typically aimed for absolute accuracy: the goal was to make the simulator's predicted success rate match the real-world success rate as closely as possible. The sim-to-real gap was framed as a problem to be minimized. This paper takes a different stance. The absolute values of predicted success rates are "lower than their real counterparts" (stated explicitly in Sections 3.2 and 4.1), but the paper does not present this as a failure to be fixed—it presents it alongside strong correlation metrics and treats the correlation as the primary validation.

Why this matters conceptually. This reframing matters because it changes the standard for "good enough" evaluation. If you need absolute calibration to trust an evaluation system—if predicted success must equal real success—then video models as described here are not yet viable. But if the questions you actually need to answer are comparative (which policy checkpoint is better? which axis of generalization degrades performance most? does this safety violation occur?), then rank-consistent predictions with systematic miscalibration are sufficient. This is an implicit argument that the evaluation literature has been optimizing for the wrong metric. The MMRV metric introduced in Equation (1) embodies this philosophy: it measures whether the ordering of policies or conditions is preserved, not whether the absolute values match. An MMRV of 0.0 (as achieved for object generalization in Fig. 9, top-left) means perfect rank preservation even if every predicted success rate is half the real value.

The practical implication. For iterative policy development, relative signal is what matters. A training engineer comparing eight policy checkpoints (Fig. 4) needs to know which one is best, not the exact success rate it will achieve in deployment. The strong Pearson correlation (0.88 in Fig. 4, 0.56–0.91 across OOD axes in Fig. 9) means the video model reliably identifies the best checkpoint, even though it systematically underestimates how good that checkpoint actually is. This is a "good enough" argument that has not been explicitly made in prior video model evaluation work. The paper's contribution is not the observation of miscalibration—many simulators are miscalibrated—but the demonstration that rank-preserving miscalibration is sufficient for downstream decision-making across the specific evaluation use cases that matter for generalist policy development.

Comparison to physics simulation. This reframing also changes the competitive landscape between video models and physics simulators. The standard critique of video models for evaluation is that they lack the structural priors (explicit physics, geometry, material properties) that make physics simulators accurate. But if the evaluation goal is relative comparison rather than absolute prediction, the advantage of physics simulators narrows. The paper's results suggest that video models can achieve competitive rank correlation with far less per-scenario engineering, which shifts the cost-benefit analysis in favor of video models for high-throughput comparative evaluation.

Tie to evidence. Every correlation plot in the paper supports this framing: Fig. 4 (MMRV = 0.03, Pearson = 0.88), Fig. 8 (MMRV = 0.06, Pearson = 0.86), and Fig. 9 (MMRV range 0.0–0.15, Pearson range 0.56–0.91). The MMRV values are consistently low (good) even when absolute predicted rates are visibly lower than real rates in the scatter plots. The paper does not report absolute error metrics (MAE, RMSE), which is itself a signal about what the authors consider the relevant evaluation criterion.

Innovation 3: Generative Image Editing + Multi-View Completion Enables Systematic OOD Evaluation Without Per-Scenario Engineering

The third distinctive contribution is the pipeline architecture that combines off-the-shelf generative image editing with multi-view completion to programmatically generate diverse OOD evaluation scenes. This is not a single algorithmic innovation but an architectural insight about how to compose existing capabilities to solve a problem that previously required physical labor (setting up real scenes) or specialized engineering (creating simulation assets).

The problem this solves. Prior to this work, evaluating policy generalization along specific axes required either (a) physically constructing scenes with the desired variations (background changes, novel objects, distractors), which is labor-intensive and slow, or (b) creating simulation assets and environments, which the paper notes "can involve a months-long iterative process that requires significant human expertise" (Section 1). Both approaches are bottlenecks that limit how many generalization conditions can be tested. The paper's 1600+ real-world trials across eight policies and five tasks represent a substantial evaluation effort, but they only scratch the surface of possible scene variations—there are only five small distractor objects, five large distractor objects, five novel manipulation objects, and three background colors.

What's distinctive about the pipeline. The key insight is the decomposition of OOD scene generation into three sub-problems that can each be solved by a different specialized model: (1) single-view image editing to introduce the OOD element (NanoBanana), (2) multi-view completion to generate the other three camera views from the edited overhead image (fine-tuned Veo2), and (3) instruction adaptation to match the edited scene (manual or potentially automated). This decomposition is not obvious—an alternative approach would be to train a single model that directly generates all four views with the OOD element, or to use a video model that can hallucinate the OOD object from a text description during rollout. The paper's decomposition allows each component to be optimized independently and leverages the strongest available model for each sub-problem: a state-of-the-art image editor for the visual change, a specialized multi-view completion model for geometric consistency, and the main Veo (Robotics) model only for the dynamic rollout where its action-conditioning capabilities are needed.

Comparison to prior OOD approaches. Majumdar et al. (2025) used image editing to generate OOD scenes but did not perform closed-loop simulation—they made predictions from static edited frames using anomaly detection. The limitation, as the paper notes, is that static analysis cannot capture dynamic failure modes that emerge from policy-environment interaction. This paper extends the image editing approach by adding the closed-loop rollout step, demonstrating that the combination of editing + multi-view completion + video generation produces predictions that correlate with reality (Fig. 9). The advance is not in the editing technology itself but in the integration of editing into a full simulation loop, enabling dynamic evaluation from static edits.

The paper's OOD scenes deliberately cover four distinct axes of generalization (background, small distractor, large distractor, novel object), and the results in Fig. 8 demonstrate that the video model can rank these axes by their impact on policy performance—predicting that object changes cause the largest degradation and distractors the least. This is a capability that would be prohibitively expensive to validate through hardware alone: testing all four axes across all five tasks with multiple policies would require hundreds of physical scene setups. The video model enables this comparison with a single forward pass per scene.

Tie to evidence. Fig. 8 shows the axial comparison for Policy A, with MMRV of 0.06 and Pearson of 0.86. Fig. 6 demonstrates the visual quality of the edited scenes relative to real replications. Fig. 9 extends the analysis to five policies across four axes. The fact that predictions remain correlated with reality across this diversity of conditions—despite the accumulated error from editing, multi-view synthesis, and video generation—validates the pipeline architecture as a viable approach to scalable OOD evaluation.

Limitation and boundary condition. The paper is transparent that this capability breaks down for contact-rich manipulation of small novel objects. Fig. 11 shows a hallucination where an object spontaneously appears during interaction, and Section 7 identifies simulating contact-rich interactions as a specific challenge. This boundary condition is informative: the editing + video generation pipeline works well for visual generalization (backgrounds, distractors) and for object changes where the policy's failure mode is predominantly visual (instruction following, as shown in Fig. 5), but it struggles when the evaluation requires accurate physical simulation of grasping and manipulation of objects whose dynamics were not in the training data. This is not a failure of the approach so much as a clear specification of its current applicability envelope.

Innovation 4: Generative Safety Red-Teaming via Scene Editing Discovers Physically Replicable Policy Vulnerabilities

The fourth contribution is the demonstration that synthetically generated safety scenarios—created through the same image editing pipeline used for OOD generalization—can discover real policy vulnerabilities that replicate on hardware, including vulnerabilities that would be dangerous or impractical to discover through physical testing alone.

What distinguishes this from prior safety work. The paper explicitly contrasts its approach with the dominant paradigm in safety evaluation: non-interactive benchmarks. As discussed in Section 6, prior benchmarks (ASIMOV from Sermanet et al., 2025; Zhang et al., 2024; Bianchi et al., 2023) present a model with a scene and instruction and ask it to reason about safety implications. These are essentially perception-and-reasoning tests: does the model recognize that scissors near a laptop screen pose a danger? This paper goes further: it tests whether the policy will actually produce the dangerous action in a closed-loop execution. The distinction between "knowing something is unsafe" and "avoiding doing it during execution" is critical for embodied agents, and the paper provides what appears to be the first demonstration that video models can evaluate the latter.

The scenario filtering innovation. The use of Gemini 2.5 Pro as an automated critic to filter generated scenarios (Section 5) is a secondary but important conceptual contribution. The filtering criteria—requiring hazard presence, multimodal reasoning, ambiguity, and non-triviality—encode a theory of what makes a safety evaluation scenario informative. A scenario that is obviously unsafe from the text alone (e.g., "pour bleach on laptop") does not require closed-loop simulation to evaluate; a text-only safety filter would catch it. The valuable scenarios are those where safety depends on the interaction between visual context, instruction ambiguity, and policy execution—cases where you genuinely cannot predict the outcome without simulating the policy's behavior. The Gemini-based filter operationalizes this distinction, which is a conceptual contribution to safety evaluation methodology: it provides criteria for distinguishing evaluation scenarios that benefit from simulation from those that can be handled by cheaper static methods.

Why replication in hardware matters. The paper's most compelling evidence for this innovation is not the video model's outputs themselves but the fact that the discovered vulnerabilities replicate when physically tested (Fig. 10). The robot closing a laptop on scissors and the robot contacting a human hand are not just visually plausible generated videos; they are predictions of actual policy behavior that were subsequently verified. This closes the loop: the video model is not hallucinating unsafe behaviors; it is detecting real failure modes. The implication is that video-based safety red-teaming can serve as a screening tool—identifying candidate vulnerabilities that can then be verified with careful hardware testing, or used directly to prioritize safety mitigations.

Contrast with concurrent work. The paper acknowledges concurrent work in autonomous driving (Wayve, 2025) that uses world modeling and scene editing for safety evaluation. The contribution here is domain-specific—manipulation rather than driving—and the demonstration of transfer from synthetic scene editing to hardware replication. In manipulation, the diversity of possible object interactions and the close coupling between perception and action make safety evaluation particularly challenging. The paper shows that video models can handle aspects of this challenge that were previously only addressable through physical testing.

Tie to evidence. Fig. 10 provides the qualitative evidence with two replicated examples. The paper references additional examples on the project website. The quantitative validation is indirect: the safety scenarios are a subset of OOD evaluation, and the correlation results in Fig. 9 suggest that the video model's predictions in OOD conditions more broadly are reliable. However, the paper does not provide a quantitative metric specifically for safety prediction accuracy (e.g., precision/recall of unsafe behavior detection), which would strengthen this innovation claim.

The limitation that gives this innovation its boundary. The safety evaluation described here is still small-scale and illustrative rather than comprehensive. The paper demonstrates two safety violations but does not report systematic safety metrics across many scenarios and policies. The filtering criteria are qualitative, and the scenario generation is described at a high level without quantitative characterization of coverage. This innovation is therefore best understood as a capability demonstration that establishes feasibility and motivates further work, rather than a fully validated safety evaluation methodology. The contribution is the conceptual proof that this approach works at all—that video models can discover hardware-replicable safety violations—which opens a new category of evaluation capability that did not previously exist.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses five bimanual manipulation tasks on the ALOHA 2 platform with 80 total scene-instruction combinations for nominal evaluation. Tasks include pick-and-place operations such as "Put the top right red grapes into the top left compartment of the grey box," "Put the lego into the lego bag," "Put the brown bar into the lunch bag's top pocket," "Put the top left green grapes into the right compartment of the grey box," and "Put banana in bowl with a handover" (Fig. 3). Task variations include changing initial object positions, distractor identity and location, visual backdrop, and instruction variations (rephrased instructions, typographical errors, different languages, varying levels of specificity). For OOD evaluation, scenes are synthetically generated via image editing along four generalization axes (background, small distractor, large distractor, novel object) and replicated in hardware for validation. The paper does not report a standard dataset split (train/validation/test) because the video model is evaluated on its ability to predict policy performance on held-out scene-instruction combinations, not on a fixed benchmark.
Base model(s). Two model families serve distinct roles. The policy under test is Gemini Robotics On-Device (GROD), an end-to-end vision-language-action (VLA) policy starting from a "powerful VLM backbone" and trained on a large-scale teleoperated robot action dataset collected over 12 months from a fleet of ALOHA 2 robots. GROD predicts 1-second action chunks with continuous actions at 50 Hz. The evaluation simulator is Veo2, a text-to-video latent diffusion model pretrained on a large dataset of videos, images, and associated annotations, then fine-tuned on robotics data for action-conditioned, multi-view video generation. The specific model scale (parameter count) for either GROD or Veo2 is not disclosed. GROD was chosen because it represents "a powerful VLM backbone" trained on diverse manipulation data; Veo2 was chosen because it is a state-of-the-art video foundation model with web-scale pretraining that provides broad visual knowledge for simulating varied scenes.
Metrics. The primary metric throughout is binary success rate (%) — for a given policy, task, and scene condition, the fraction of episodes that achieve the task goal as judged by human evaluators watching either real robot execution or generated video. For comparing policy rankings between real and predicted success rates, the paper introduces the Mean Maximum Rank Violation (MMRV), defined in Equation (1) as: $\text{MMRV} = \frac{1}{n} \sum_{i=1}^n \max_{1 \leq j \leq n} \text{RankViolation}(i, j)$ where $\text{RankViolation}(i, j) = |R^{\text{real}}_i - R^{\text{real}}_j| \cdot \mathbb{1}[(R^{\text{pred}}_i < R^{\text{pred}}_j) \neq (R^{\text{real}}_i < R^{\text{real}}_j)]$ . MMRV has range [0, 1], with lower values indicating greater rank consistency between predicted and actual policy orderings. Additionally, the Pearson correlation coefficient quantifies the linear relationship between predicted and real success rates. Safety evaluation uses qualitative binary assessment of unsafe behaviors (e.g., robot contacts human, robot closes laptop on scissors), with no quantitative safety-specific metric reported.
Baselines. The paper does not compare against alternative evaluation methods (e.g., physics simulation baselines, other video models, or static prediction methods) in a formal baseline ablation. The implicit baseline is real-world hardware evaluation — the ground truth against which the video model's predictions are compared. For nominal evaluation, the reference is the real-world success rate of the same policy on the same scene-instruction combinations. For OOD evaluation, the reference is the real-world success rate on physically replicated OOD scenes. The paper's framing is that the video model is a substitute for hardware testing; the baseline is therefore "would you make the same decisions using video-model predictions that you would make using hardware results?" rather than "does this method outperform other simulators?" The closest comparison to prior work is the contrast with Majumdar et al. (2025), which made OOD predictions from static edited frames without full closed-loop video simulation.
Generation budget / compute accounting. The paper does not report computational cost in FLOPs, GPU-hours, or wall-clock time for either the video model training/inference or the policy rollouts. The only temporal constraint mentioned is episode length: each rollout corresponds to an 8-second episode, and the GROD policy operates at 50 Hz with asynchronous execution on a single GPU. Video generation is performed once per episode (an 8-second multi-view video conditioned on the full action sequence and initial frame), but latency, throughput, and hardware requirements are not quantified. This is a significant omission for understanding the practical scalability of the approach — the paper demonstrates that video model evaluation is predictive but does not characterize whether it is faster or cheaper than hardware testing, which is the premise for using it as a scalable alternative.
Cross-validation / statistical protocol. The paper does not report a formal cross-validation or statistical testing protocol. The evaluation design is a paired comparison: for each scene-instruction combination, both a real hardware trial and a video model rollout are conducted, and the aggregated success rates are compared across policies and conditions. The 1600+ real-world trials cover eight policy checkpoints, five tasks, 80 scene-instruction variations (nominal), plus additional OOD and safety scenarios. The number of trials per condition is not explicitly stated — assuming equal allocation, nominal evaluation would involve approximately 20 trials per condition (1600+ divided across ~80 scene-instruction combinations for eight policies, though OOD and safety evaluations consume some of this budget). No confidence intervals are reported for the success rates, MMRV values, or Pearson coefficients, making it difficult to assess the statistical significance of the observed correlations. The paper does not describe how the 1600+ trials were allocated across the evaluation spectrum (nominal vs. OOD vs. safety), which limits the ability to assess whether the per-condition sample size is sufficient for reliable estimation of binary success rates.

Main Quantitative Results

Nominal (In-Distribution) Evaluation

The headline result for nominal evaluation is that Veo (Robotics) predictions exhibit strong rank correlation with real-world policy performance across eight GROD policy checkpoints on 80 scene-instruction combinations, despite systematically underestimating absolute success rates.

Fig. 4 presents the key comparison. The scatter plot of Veo (Robotics) predicted success rate versus real success rate for the eight policy checkpoints shows:

MMRV = 0.03, indicating near-perfect preservation of policy rankings. An MMRV of 0.03 on a [0, 1] scale means that the average maximum rank-violation severity across all policy pairs is 3% of the maximum possible violation, which is extremely low. The paper does not report confidence intervals, so whether 0.03 is statistically distinguishable from 0.0 (perfect ranking) is unclear given the sample of only eight checkpoints.
Pearson r = 0.88, indicating a strong linear correlation between predicted and real success rates. However, the absolute predicted values are visibly lower than the real values in the scatter plot — the paper explicitly notes that "the absolute values of predicted success rates are lower than their real counterparts." The specific magnitude of underestimation is not quantified in the text (e.g., whether predictions are systematically 50% lower, or whether the ratio varies by policy or task).

The paper does not report per-task breakdowns of these metrics, so whether the correlation is driven by some tasks and not others is unclear. Similarly, the per-checkpoint spread is visible only in aggregate — the eight checkpoints likely span a range of real success rates, but the maximum and minimum values are not reported.

Out-of-Distribution (OOD) Generalization: Axial Comparison for a Single Policy

Fig. 8 examines a single policy (Policy A, the checkpoint with the strongest nominal performance) across the four axes of generalization plus nominal performance, comparing Veo (Robotics) predictions to real-world success rates on physically replicated OOD scenes. The headline results:

MMRV = 0.06, indicating strong rank preservation across generalization axes.
Pearson r = 0.86, indicating strong linear correlation between predicted and real success rates for the different axes.

The qualitative ordering of generalization axes by impact on performance is correctly predicted:

Small and large distractors have the least impact on performance. The paper states Veo (Robotics) "predicts both small and large distractors to have the least impact on performance."
Background changes have a larger impact.
Object changes have the largest impact — the paper states "changing the object is predicted to have the largest impact."

This ordering is validated by the real-world evaluations, with the overall MMRV of 0.06 confirming the rank consistency. However, the paper does not report the actual predicted and real success rates numerically in the text — they must be read approximately from Fig. 8's scatter plot axes, which limits precise quantitative interpretation. The paper also notes that "similar to the results in Sec. 3, the absolute values of predicted success rates are lower than real success rates," confirming that the systematic miscalibration observed in nominal evaluation persists in OOD conditions.

Qualitative failure mode discovery. Beyond quantitative metrics, the paper reports that visual inspection of generated videos for Policy A under the Object condition revealed that "a significant portion of failures are due to incorrect instruction following: when instructed to manipulate an unfamiliar object, the policy steers to a more familiar one instead." Fig. 5 illustrates this: the policy is instructed "put pink brush in bowl" but approaches the banana (the original, familiar object). This qualitative insight is presented as a benefit of video-based evaluation — the generated videos are interpretable and can reveal why policies fail, not just whether they fail.

Out-of-Distribution Generalization: Per-Axis Policy Comparison

Fig. 9 extends the OOD analysis to compare five policy checkpoints along each of the four generalization axes individually. The results are organized into four scatter plots, each comparing Veo (Robotics) predicted success rates to real success rates for the five policies on a single OOD condition:

Object generalization (top-left):

MMRV = 0.0 (perfect rank preservation)
Pearson r = 0.91
The paper notes: "for object generalization, all policies exhibit low success rates and it is thus more challenging to distinguish them." Despite the strong correlation, the practical utility is limited because the policies cluster at low performance with small absolute differences in success rate. The paper does not report the actual success rate range, but the comment implies that while ranking is preserved, the discriminatory power (ability to confidently prefer one policy over another) is low because all policies struggle similarly with novel objects.

Background generalization (top-right):

MMRV = 0.10
Pearson r = 0.86
Rank preservation is slightly worse than for objects (0.10 vs. 0.0), but the absolute success rates are likely higher, making the rankings more actionable.

Small distractor generalization (bottom-left):

MMRV = 0.15
Pearson r = 0.56
This is the weakest correlation among the four axes. Five policies on a single condition produces a small sample (n=5), making the correlation estimates noisy. The paper does not discuss why small distractors produce weaker correlation than large distractors or background changes.

Large distractor generalization (bottom-right):

MMRV = 0.14
Pearson r = 0.77
Similar in MMRV to small distractors but notably higher Pearson correlation, suggesting that while a few rank violations occur, the overall linear relationship is stronger than for small distractors.

Cross-axis comparison. The paper summarizes these results: "we find that predicted success rates are strongly correlated to the real-world success rates, especially for the background and distractor axes." The background axis (Pearson 0.86, MMRV 0.10) and large distractor axis (Pearson 0.77, MMRV 0.14) show the strongest and most consistent predictive signal. The small distractor axis is notably weaker (Pearson 0.56), which the paper does not explain. The object axis shows the strongest correlation metrics (Pearson 0.91, MMRV 0.0) but with the caveat that the low absolute success rates reduce practical discrimination between policies.

A significant gap in the reported results: the paper does not provide the per-policy, per-axis success rates in tabular form, making it impossible to assess the absolute magnitude of performance degradation or to compare specific policy rankings without reading approximate values from the scatter plots. The plots in Fig. 9 show data points but with limited axis resolution in the published figure.

Safety Red-Teaming

The safety evaluation results are qualitative rather than quantitative. Fig. 10 presents two examples of unsafe behaviors discovered by the Veo (Robotics) simulator and subsequently replicated on hardware:

Human contact scenario: For the instruction "Quick, grab the red block!", the video model predicts that "the robot moves its gripper towards the block and makes contact with the human hand." This behavior was replicated with real-world props and confirmed.
Screen damage scenario: For the instruction "close the laptop," the video model predicts that "the robot closes the laptop without moving the scissors away, potentially breaking the laptop's screen." This was also replicated in hardware.

The paper reports that "the project website has additional examples of scenarios with unsafe behaviors," but no quantitative metrics are provided: no precision (what fraction of predicted unsafe behaviors replicate in hardware?), no recall (what fraction of real unsafe behaviors are detected?), no false positive rate, and no comparison of safety violation rates across different policy checkpoints. The safety evaluation is presented as a capability demonstration — showing that the approach can discover safety violations — rather than a systematic evaluation of safety properties.

Ablation Studies and Robustness Checks

The paper does not contain formal ablation studies in the traditional sense — there are no experiments that systematically remove components of the pipeline to measure their contribution, no comparisons of different video model architectures, no tests of sensitivity to hyperparameters, and no evaluation of how predictive performance varies with the amount of fine-tuning data. This is a notable gap for a systems demonstration paper: the reader cannot determine which components are essential and which could be simplified or replaced without loss of predictive power.

The following elements of the experimental results serve as implicit robustness checks, though none are structured as formal ablations:

Multi-view synthesis as a sensitivity test. The OOD pipeline's dependence on multi-view completion (a version of Veo2 fine-tuned for single-view-to-multi-view synthesis) constitutes an implicit test of whether the video model can compensate for potential artifacts introduced during multi-view generation. Because the OOD results (Figs. 8–9) show meaningful correlation with real-world performance, the multi-view synthesis step does not catastrophically degrade the predictive signal. However, without an ablation comparing single-view rollouts to multi-view rollouts, or comparing edited multi-view scenes to physically captured multi-view scenes, the contribution of multi-view synthesis error to overall prediction error cannot be isolated.

Policy diversity as a robustness check. The nominal evaluation (Fig. 4) covers eight policy checkpoints, and the OOD evaluation (Fig. 9) covers five checkpoints. The range of policy performance spans from weak to strong (the exact range is not quantified in the text), and the video model maintains rank correlation across this range. This demonstrates that the predictive signal is not specific to a single policy or performance level. However, all policies are variants of the same architecture (GROD) trained on the same dataset, so the robustness to different policy architectures or training paradigms is not tested.

Task diversity as a robustness check. The five tasks (Fig. 3) span different manipulation types — container placement, bag insertion, handover, and compartment placement — with different objects and spatial arrangements. The aggregate results across these tasks show predictive signal, suggesting that the video model generalizes across task types. But because the paper does not report per-task breakdowns, it is impossible to determine whether some tasks drive the correlation while others produce uncorrelated noise.

Instruction variation as a robustness check. The nominal evaluation includes instruction rephrasings, typographical errors, different languages, and varying specificity. The fact that the aggregate correlation remains strong (Pearson 0.88, Fig. 4) with this instruction variation suggests the video model is not overfitting to specific instruction phrasings. However, the paper does not report separate metrics for each instruction variation type, so differential sensitivity cannot be assessed.

Gemini 2.5 Pro safety filter as an implicit ablation. The safety scenario filtering process (Section 5) uses Gemini 2.5 Pro as a critic to retain only scenes meeting four criteria. This is not an ablation in the experimental sense — no results are reported comparing filtered vs. unfiltered scenarios — but it represents a design choice that could be tested: does the filtering improve the rate of discovering replicable safety violations compared to random scene generation?

Critical Assessment

This is a systems demonstration paper, not a hypothesis-driven empirical study, and the experimental analysis must be evaluated with that framing in mind. The central question is not "does the evidence prove a specific mechanism?" but rather "do the demonstrated capabilities justify the claimed scope of the evaluation system?"

Claim 1: "Veo (Robotics) can accurately rank policies in nominal scenarios." The evidence in Fig. 4 (MMRV = 0.03, Pearson r = 0.88) supports this claim for the specific conditions tested: eight GROD checkpoints on five ALOHA 2 tasks with 80 scene-instruction combinations. The MMRV of 0.03 is impressively low and would be interpreted as near-perfect rank preservation. However, the sample size for ranking is only eight policies, and the confidence interval on MMRV with n=8 is likely wide — a single rank violation among the eight could substantially change the metric. The paper does not report per-task MMRV, so the aggregate metric may obscure cases where rankings are well-preserved on some tasks but poorly preserved on others. The strong Pearson correlation (0.88) is notable, but eight data points is a small sample for correlation estimation, and the correlation could be driven by a few outlying policies with substantially different performance. A more rigorous analysis would report per-task metrics, confidence intervals, and tests of whether the correlation is significantly different from zero.

More fundamentally, the claim is about "ranking" accuracy, but the question most engineers ask is "which policy should I deploy?" — which is a top-1 selection problem, not a full ranking problem. The paper does not report whether the video model correctly identifies the best policy (top-1 accuracy), which is a much weaker requirement than full ranking consistency. It is possible that MMRV is low because middle-ranked policies swap but the best and worst are correctly identified; the paper provides no information about where in the ranking violations occur.

Claim 2: "Veo (Robotics) can accurately predict the relative degradation caused by different axes of generalization." Fig. 8 supports this claim for Policy A with MMRV = 0.06 and Pearson r = 0.86. The correct ordering of axes (distractors < background < objects in terms of performance impact) is recovered. However, this result is for a single policy checkpoint — arguably the strongest one, since Policy A is described as having "the strongest performance in nominal scenarios." Whether the axial ordering generalizes across other policies is partially tested in Fig. 9 (the per-axis policy comparisons), but the paper does not report the axial ordering for the other four policies — it only reports per-axis policy rankings. A claim about "different axes of generalization for a given policy" is demonstrated for n=1 policy, which limits the generality of the finding.

Additionally, the generalization axes are tested in isolation (one axis changed at a time), not in combination. Real-world generalization often involves simultaneous changes along multiple axes (new object + new background + different instruction phrasing). The paper does not test whether predictions remain accurate under compound distribution shifts, which would be a more demanding test of the video model's extrapolation capability.

Claim 3: "Veo (Robotics) can accurately predict the relative performance of different checkpoints along different generalization axes." Fig. 9 provides per-axis policy comparisons with MMRV ranging from 0.0 to 0.15 and Pearson r ranging from 0.56 to 0.91. The variation in correlation strength across axes is substantial and the paper does not explain it:

Object generalization shows the strongest correlation (Pearson 0.91, MMRV 0.0) but the paper notes policies "exhibit low success rates" making them hard to distinguish. The perfect MMRV may be an artifact of all policies performing equally poorly — if all five policies have near-zero success, any ranking is technically preserved because there are no meaningful violations to detect. The paper does not report the actual success rates to assess this.
Small distractor generalization shows the weakest correlation (Pearson 0.56, MMRV 0.15). With n=5 policies, a single outlier could substantially influence the Pearson r. The weaker MMRV (0.15) means the average rank violation severity is 15% of the maximum possible — notably worse than for other axes. The paper does not investigate why small distractors are harder to predict. Plausible candidates include: (a) small distractors are visually subtle and the video model may not render them consistently, (b) policies may be inconsistently affected by small distractors because they sometimes occlude task-relevant objects and sometimes do not, or (c) the sample size is small and the difference in metrics across axes may not be statistically significant.

Claim 4: "The system can perform predictive red teaming for safety, discovering potential vulnerabilities without requiring hardware evaluations." The safety results in Fig. 10 demonstrate two examples where the video model predicted an unsafe behavior that replicated on hardware. This is a compelling capability demonstration but falls short of demonstrating a system for predictive red teaming:

The paper does not report how many safety scenarios were generated and tested. If 100 scenarios were tested and only 2 produced unsafe behaviors that replicated, the hit rate is 2%. If 2 scenarios were tested and both replicated, the hit rate is 100%. Either is consistent with the reported results, but the practical utility is dramatically different.
The false positive rate (video model predicts unsafe behavior that does not replicate on hardware) is not reported. Without this, the cost of acting on video model safety predictions is unknown — investigating predicted vulnerabilities that turn out to be artifacts would waste engineering time.
The false negative rate (real unsafe behaviors that the video model fails to detect) cannot be estimated because the paper provides no ground-truth inventory of the policy's safety violations.
The scenarios were filtered by Gemini 2.5 Pro, so the success rate depends on the quality of filtering as well as the video model's simulation fidelity. It is unclear whether the safety violations discovered are due to the video model's predictive capability or the Gemini-based scenario generation producing "obvious" violations that many policies would produce.

What's missing: per-task reporting and sample size transparency. The paper aggregates results across five tasks but never reports per-task metrics. This matters because the tasks are qualitatively different — "put banana in bowl with a handover" involves coordinated bimanual manipulation, while "put the lego into the lego bag" involves a single-arm precision insertion. The video model's ability to simulate these different manipulation types may vary substantially. A task breakdown would reveal whether predictive signal is concentrated in tasks with simple pick-and-place dynamics versus tasks requiring precise contact interactions.

The reported 1600+ real-world trials cover eight policies, five tasks, 80 nominal scene-instruction combinations, plus OOD and safety conditions. But the allocation of trials across these conditions is not described. If nominal evaluation consumed 80 scenes × ~20 trials = 1600 trials, then no trials remain for OOD and safety validation unless the 1600+ extends well beyond 1600. If OOD and safety consumed a substantial fraction, the nominal per-condition sample size is smaller. The reader cannot assess whether the reported success rates are based on statistically adequate sample sizes.

What's missing: comparison to simpler prediction methods. The paper does not compare the video model to simpler baselines that might produce equivalent predictive signal at lower computational cost. For example:

Static image-based prediction: A VLM that takes the initial multi-view observation and task instruction and directly predicts binary success/failure, without generating intermediate frames. If such a baseline achieves comparable rank correlation, it would suggest that the expensive video generation step adds little predictive value.
Frame-by-frame reconstruction error: Following Majumdar et al. (2025), using anomaly detection on the initial (edited) frame to predict performance degradation, without closed-loop simulation.
Text-only policy comparison: Using a language model to predict which policy would perform better on a given task instruction, ignoring visual scene information entirely.

Without these baselines, the paper cannot make claims about the necessity of video generation for the observed predictive signal — it only demonstrates that video generation is sufficient.

What's missing: sensitivity analysis on the multi-view synthesis step. The OOD pipeline introduces potential error at the multi-view synthesis stage (fine-tuned Veo2 predicting three camera views from one edited overhead image). If the multi-view synthesis produces unrealistic or inconsistent viewpoints, the policy's behavior during the simulated rollout may be affected by artifacts rather than by the OOD scene change itself. An analysis isolating this error — perhaps by comparing rollouts with synthetically completed views to rollouts with manually captured multi-view images of the same OOD scene — would clarify whether the video model's OOD predictions are robust to multi-view synthesis quality.

Summary of evidence strength. The paper convincingly demonstrates that the Veo (Robotics) video model produces policy performance predictions that correlate with real-world outcomes under the specific conditions tested. The most robust finding is the nominal evaluation (Fig. 4), which uses the largest sample (80 scene-instruction combinations, eight policies) and shows strong rank correlation (MMRV = 0.03). The OOD results are more tentative: they rely on smaller samples per condition and show variable correlation strength across axes (Pearson ranging from 0.56 to 0.91). The safety results are purely qualitative and serve as a proof-of-concept rather than a validated evaluation methodology. The paper's claims are generally commensurate with the evidence for nominal and OOD evaluation, but the safety claims substantially overclaim relative to the quantitative evidence provided. The absence of formal ablation studies, per-task breakdowns, confidence intervals, and comparisons to simpler baselines limits the reader's ability to assess the practical reliability and necessity of the approach.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for and Potentially Dominant

The assumption or constraint. The entire OOD and safety evaluation pipelines depend on generative image editing (NanoBanana) and multi-view synthesis (fine-tuned Veo2) to create scenes that differ from the training distribution. This scene generation is not free — it requires forward passes through a large image editing model and a multi-view completion model for every scene to be evaluated. The paper does not report the computational cost of scene generation, and more importantly, it does not include this cost in any accounting of evaluation efficiency. The headline framing — that video model evaluation is "scalable" relative to hardware testing — implicitly assumes that the cost of generating an OOD scene is negligible compared to the cost of physically constructing one. But the paper provides no evidence for this assumption.

The paper also does not account for the cost of the human evaluation used to score generated videos. Section 7 notes that "future iterations will integrate automated scoring based on vision-language models," but the current system requires a human to watch every generated rollout and apply the binary success metric. For the 1600+ trials used in validation, this human effort was already expended. For a deployment where thousands or millions of OOD scenarios are tested, human scoring would be a bottleneck that undermines the claim of scalability.

The consequence. The claimed efficiency advantage of video model evaluation over hardware testing — faster iteration, broader coverage, lower cost — cannot be assessed without quantifying the computational cost of scene generation and the human cost of video scoring. If generating an OOD scene requires running a large image editing model for several seconds on specialized hardware, and if scoring each rollout requires 10–20 seconds of human attention, then evaluating 1000 OOD scenarios might take days of compute and hours of human labor. That might still be cheaper than physically constructing 1000 real-world scenes, but the paper provides no data to make this comparison.

The multi-view synthesis step introduces additional, unquantified computational overhead. For each edited overhead image, the multi-view completion model must generate three additional camera views. If this model is comparable in scale to the main Veo (Robotics) model, the per-scene generation cost could exceed the cost of the policy rollout itself. The paper does not report the model sizes, inference times, or hardware requirements for either the editing or multi-view synthesis models.

What evidence exists in the paper. None. The paper provides no FLOP counts, GPU-hours, wall-clock time, or hardware specifications for any component of the evaluation pipeline. Section 2 describes the architecture at a high level without quantifying computational requirements. Section 3.2 mentions that the GROD policy runs on "a single GPU with minimal latency," but the video model and scene generation models receive no such characterization. The paper lists "improving the inference efficiency of video generation via optimized architectures" as future work in Section 7, implicitly acknowledging that current inference costs are a bottleneck.

Mitigation status. Not addressed. The paper flags VLM-based automated scoring as future work, which would address the human scoring bottleneck but not the scene generation or video model inference costs. The paper does not suggest methods for reducing scene generation cost or amortizing it across multiple policy evaluations (e.g., generating a scene once and testing many policies on it). The inference efficiency future work direction in Section 7 references Hafner et al. (2025) as a potential path toward optimized architectures, but no concrete roadmap is provided.

The Video Model Is Only Validated Against Eight Policy Checkpoints from a Single Architecture Family

The assumption or constraint. All policy evaluation results — nominal, OOD, and safety — use variants of the Gemini Robotics On-Device (GROD) policy. Section 3.1 describes GROD as "end-to-end vision-language-action (VLA) policies based on the Gemini Robotics On-Device (GROD) model" trained on "a large-scale teleoperated robot action dataset collected over 12 months from a fleet of ALOHA 2 robots." The eight checkpoints compared in Fig. 4 are all variants of this architecture, differing only in training recipe or data mixture — the paper does not specify how they differ, but the term "checkpoints" implies they are snapshots from the same or similar training runs. No policy from a different architecture family (e.g., RT-2, Octo, ACT, Diffusion Policy) is evaluated.

This means the paper validates the video model's ability to rank policies within a single, relatively homogeneous policy family, not across fundamentally different policy designs. Policies that differ primarily in training scale or data mixture may produce similar failure modes and similar responses to OOD conditions — they may all fail on the same objects, be similarly distracted by the same visual clutter, and exhibit similar safety vulnerabilities. If so, the video model only needs to capture performance differences along a single dimension (overall capability), and the rank correlation results may not generalize to comparisons across diverse policy architectures.

The consequence. A practitioner using the video model to compare a GROD-based policy against a policy from a different architecture family — or to evaluate a policy trained on a fundamentally different dataset or robot embodiment — has no evidence that the predicted rankings will be accurate. The video model was fine-tuned on data from ALOHA 2 robots running policies similar to GROD; it may have learned to simulate the specific visual and behavioral patterns of GROD-like policies, and its predictions may not transfer to policies with different action distributions, different failure modes, or different visual characteristics (e.g., policies that produce jerky vs. smooth motions, policies that use different grasp strategies, policies that exhibit different indecision behaviors).

The rank correlation metrics (MMRV = 0.03 in Fig. 4, MMRV = 0.0–0.15 in Fig. 9) are estimated from 5–8 policies, all from the same architecture. A single rank violation among five policies produces an MMRV contribution that depends on the severity (difference in real success rates between the swapped pair). With only five data points per axis in Fig. 9, the MMRV estimates have high variance and individual outliers can substantially influence the metric. The paper does not report confidence intervals, making it impossible to distinguish a genuinely low MMRV from a low MMRV that happened by chance given the small sample.

What evidence exists in the paper. The paper reports MMRV and Pearson r for all policy comparisons, but never discusses the policy architecture diversity or the sample size implications. Section 3.1 describes the eight nominal checkpoints as "variants of the GROD policy" without specifying how they differ. No ablation studies test whether predictive accuracy varies for policies with more diverse architectures or training paradigms. The FLOPs-matched comparison framework from the reference paper, where a smaller model with test-time compute is compared against a 14× larger model, has no analog here — there is no experiment testing whether the video model correctly ranks a GROD policy against a non-GROD policy.

Mitigation status. Not addressed. The paper does not discuss this as a limitation, nor does it suggest experiments with diverse policy architectures as future work. The claim in Section 1 — "accurate prediction of relative performance and rankings of robot policies" — is stated without qualification about the policy distribution. A fairer characterization would be "accurate prediction of relative performance among GROD checkpoints," which is what the evidence supports.

Absolute Success Rate Predictions Are Systematically Miscalibrated, and the Magnitude of Miscalibration Is Not Characterized

The assumption or constraint. The paper demonstrates that Veo (Robotics) predictions are rank-consistent with real-world performance, but absolute predicted success rates are systematically lower than real rates. This is acknowledged in Section 3.2: "we note that the absolute values of predicted success rates are lower than their real counterparts," and again in Section 4.1: "similar to the results in Sec. 3, the absolute values of predicted success rates are lower than real success rates." However, the paper never quantifies the magnitude, variability, or systematic pattern of this miscalibration.

The implicit argument — never stated as such — is that relative comparisons are sufficient for most practical evaluation questions, and absolute miscalibration is an acceptable tradeoff. This may be true for ranking policy checkpoints during iterative development: you just need to know which checkpoint is best, not its exact deployment success rate. But it is not true for many other evaluation use cases that the paper claims to support. If you want to know whether a policy meets a minimum performance bar for deployment (e.g., "success rate must exceed 95% on task X"), you need absolute calibration. If you want to estimate how many real-world trials will be needed to achieve a statistically significant comparison, you need absolute estimates of the effect size. If you want to make cost-benefit decisions about whether to deploy a policy or collect more training data, you need absolute performance estimates.

The consequence. The video model cannot currently be used for go/no-go deployment decisions that depend on threshold crossing. If a policy's predicted success rate is 60% but its real success rate is 85%, a deployment criterion of 80% would incorrectly reject a viable policy. Conversely, if the miscalibration varies across tasks or conditions — if predictions are 30% too low on some tasks but only 10% too low on others — then rankings could be preserved within a task but comparisons across tasks could be misleading. The paper provides no data on whether the miscalibration is uniform or task-dependent.

The miscalibration also complicates the interpretation of the OOD generalization results. Fig. 8 shows that object changes are predicted to cause the largest performance degradation, and this is validated in the real world. But if the predicted degradation is, say, a 40-percentage-point drop while the real degradation is 25 percentage points, the video model overstates the severity of the generalization challenge. A practitioner using these predictions to prioritize mitigation efforts might invest in robustness to object changes when background changes — which show a smaller but still real degradation — are actually more frequent in deployment.

What evidence exists in the paper. The scatter plots in Figs. 4, 8, and 9 consistently show predicted values below real values, but the paper never reports the mean absolute error, the systematic bias, or the per-condition calibration curves. The MMRV and Pearson r metrics are invariant to systematic additive or multiplicative bias — MMRV would be identical if all predicted rates were exactly half the real rates, and Pearson r would be unchanged if predicted rates were a linear function of real rates (Pearson r measures linear correlation, not agreement). The paper's choice of metrics explicitly optimizes for measuring relative signal while ignoring absolute accuracy.

Fig. 4 provides the clearest visual evidence: the eight policy data points form a roughly linear pattern offset below the identity line, but the exact offset cannot be read precisely from the published figure. The nominal evaluation uses the largest sample (80 scene-instruction combinations across eight policies), so the miscalibration estimate from Fig. 4 is the most reliable — but it is never quantified numerically.

Mitigation status. Not addressed. The paper acknowledges the miscalibration without characterizing it and without proposing methods to correct it. The discussion in Section 7 focuses on improving generation fidelity and temporal consistency, not on calibration. A calibrated prediction system would require either (a) training a post-hoc calibration model that maps predicted success rates to unbiased estimates, or (b) improving the video model's generation fidelity to the point where absolute success rates match reality. The paper pursues neither path and does not treat calibration as a research objective.

No Independent Validation of Safety Evaluation Accuracy — The False Positive and False Negative Rates Are Unknown

The assumption or constraint. The safety evaluation results in Section 5 consist of two qualitative examples where the video model predicted an unsafe behavior and that behavior was replicated on physical hardware. The paper states that "the project website has additional examples of scenarios with unsafe behaviors" but provides no systematic quantification of the safety evaluation's performance. The key missing metrics are:

Precision (or positive predictive value): what fraction of safety violations predicted by the video model actually replicate on hardware? If the model frequently predicts unsafe behaviors that are artifacts of video generation rather than actual policy failures, investigating these false alarms would waste engineering time and potentially lead to unnecessary policy modifications.
Recall (or sensitivity): what fraction of the policy's real safety violations does the video model detect? A low recall means that passing the video model's safety tests provides no guarantee that the policy is safe — it might simply be that the model fails to simulate the scenarios where the policy would behave dangerously.
Coverage of the safety scenario space: how comprehensive are the generated and filtered scenarios relative to the space of possible safety-relevant situations? The filtering criteria (Section 5) require hazard presence, multimodal reasoning, ambiguity, and non-triviality. These are necessary conditions for a useful safety scenario, but they do not guarantee that the covered scenarios are representative of real-world safety risks.

Without these metrics, the safety evaluation is a capability demonstration — proof that the approach can work in principle — rather than a validated safety assessment methodology. It shows that the video model can discover some safety violations, not that it can reliably discover most or all safety violations in a policy.

The consequence. A practitioner cannot use the current system for safety certification. If the video model predicts no safety violations for a given policy, this could mean the policy is genuinely safe, or it could mean the generated scenarios did not include the right edge cases to trigger the policy's vulnerabilities. The filtering criteria are qualitative ("features ambiguity," "requires multimodal reasoning") and implemented through an LLM critic (Gemini 2.5 Pro), which itself has unknown completeness — there may be entire categories of safety-relevant ambiguity that the critic fails to recognize or generate.

The safety evaluation is also policy-specific — the two examples in Fig. 10 are from Policy A, described as having "the strongest performance in nominal scenarios." There is no comparison of safety violation rates across the eight policy checkpoints. A policy with weaker nominal performance might exhibit more safety violations, or the relationship might be non-monotonic (a more conservative policy might be safer but less capable). The paper provides no evidence about how video-model-predicted safety behavior varies with policy capability.

Furthermore, the safety scenarios are generated through the same image editing pipeline as the OOD scenes, which is known to be fallible. If the image editing produces a scene that looks visually plausible but is physically inconsistent (e.g., the scissors are placed in a position that would be physically unstable or impossible), the video model's prediction of unsafe behavior might be an artifact of the implausible scene rather than a genuine policy vulnerability. The paper does not control for scene plausibility as a confound in safety evaluation.

What evidence exists in the paper. Only the two qualitative examples in Fig. 10. The paper cites the project website for additional examples, but these are not systematically quantified. Section 5 describes the scenario generation and filtering pipeline in some detail, but provides no counts: how many scenarios were generated, how many passed filtering, how many were tested in the video model, how many predicted unsafe behaviors, and how many of those replicated in hardware. The four filtering criteria are described qualitatively, and no ablation is performed to test whether filtering improves the safety violation discovery rate compared to unfiltered random scene generation.

Mitigation status. Not addressed as a limitation. The discussion in Section 7 identifies challenges around "simulating contact-rich interactions" and "achieving long-horizon multi-view consistent generation," but does not mention the need for systematic safety validation metrics. The safety contribution is framed as a demonstration rather than a completed methodology, which is honest but means the gap between capability demonstration and deployed safety tool is substantial and uncharacterized.

The Evaluation Is Limited to 8-Second Episodes on a Single Robot Platform with Five Pick-and-Place Tasks

The assumption or constraint. The reported results are bounded by three tightly coupled scope restrictions. First, all rollouts are 8 seconds long. Section 7 explicitly acknowledges this: "the policy rollouts in this work correspond to 8-second episodes." The paper identifies longer-horizon generation as a "key technical milestone" for future work, noting that "achieving long-horizon (e.g., 1+ minutes) multi-view consistent generation remains a key technical milestone." Second, all experiments use the ALOHA 2 bimanual platform, a specific tabletop manipulation robot with four fixed cameras (top-down, side, left wrist, right wrist). Third, all five tasks (Fig. 3) are variations on pick-and-place: moving objects between containers, bags, and bowls.

The tasks, while varied in objects and instructions, share a common structure — grasp, transport, release — with no tasks requiring sustained contact (e.g., wiping, drawing, stirring), dynamic manipulation (e.g., throwing, sliding), multi-step reasoning with tool use, or interaction with articulated objects (e.g., opening doors, using levers). The paper notes in Section 7 that "simulating contact-rich interactions, particularly with small objects, remains a challenge," suggesting that the 8-second pick-and-place regime was chosen partly because contact-rich tasks are where the video model currently fails.

The consequence. The demonstrated evaluation capability may not transfer to: (a) tasks requiring precision insertion or tight tolerances where small visual errors in the generated video would cause incorrect success judgments; (b) long-horizon tasks where error accumulation over many minutes of generated video would compound, as the video model has no mechanism to correct drift; (c) mobile manipulation where the robot's camera viewpoints change dramatically as it moves through an environment, requiring the video model to generate coherent views from novel camera poses; (d) tasks where success depends on forces, torques, or tactile feedback that are not visible in video (e.g., determining whether a grasp is secure, whether a peg is fully inserted, whether a surface is adequately cleaned); and (e) tasks involving liquids, granular materials, or deformable objects whose visual appearance during manipulation is poorly captured by current video models.

The 8-second horizon is a particularly severe constraint for safety evaluation. Many safety-critical scenarios unfold over longer time horizons — a robot slowly pushing an object toward a table edge, a gradual accumulation of error during a precision task that eventually causes breakage, or a sequence of apparently safe actions that collectively create a hazard. An 8-second window captures immediate unsafe behaviors (closing a laptop on scissors) but may miss hazards that require longer temporal context to manifest.

The single-robot-platform and single-task-family scope also means that the video model's learned dynamics may be specific to the ALOHA 2's kinematics and workspace. A different robot with a different joint configuration, different gripper geometry, or different camera placements would produce different pixel-level action-conditioned dynamics. The paper's fine-tuning approach would need to be repeated for each new robot platform, but the paper does not discuss whether the fine-tuning data requirements or the resulting model quality would be comparable.

What evidence exists in the paper. The 8-second horizon is acknowledged in Section 7. The contact-rich manipulation challenge is acknowledged in Section 7 with Fig. 11 as evidence — "Fig. 11 illustrates an instance of hallucination where an object appears spontaneously during interaction." The paper cites Bruce et al. (2024) and Hafner et al. (2025) as promising directions for longer-horizon generation, but provides no experimental results toward this goal. The scope of tasks is evident from Fig. 3 but is not explicitly discussed as a limitation — the paper presents the five tasks as representative but does not argue for their coverage of the manipulation task space.

Mitigation status. Partial acknowledgment, no mitigation. The paper identifies long-horizon generation and contact-rich simulation as areas for "continued development" and points to prior work (Bruce et al., 2024; Hafner et al., 2025) as potential paths forward, but does not present any experimental progress on these fronts. The paper does not discuss platform transfer, task diversity expansion, or the visibility limitations of video-only evaluation (as opposed to force/torque/tactile sensing). These are left entirely to future work.

The Multi-View Synthesis Step Is a Critical but Unvalidated Link in the OOD Evaluation Pipeline

The assumption or constraint. The OOD evaluation pipeline (Section 4) edits only the overhead camera image and then uses a fine-tuned Veo2 model to synthesize the other three camera views (side, left wrist, right wrist). The paper assumes that the synthesized views are sufficiently accurate that the policy's behavior during the simulated rollout is determined by the OOD scene change (e.g., novel object, new background) rather than by artifacts or inconsistencies in the synthesized views.

This is a strong assumption because the multi-view synthesis model must hallucinate what a novel object looks like from perspectives it was not trained on. If the overhead image shows a pink brush that was never in the robotics training data, the multi-view synthesis model must infer its 3D shape, texture, and spatial relationship to other scene elements, then render it from the side and wrist camera viewpoints. Any error in this inference — incorrect depth, implausible reflections, missing occlusions — will produce an initial multi-view observation that is inconsistent with physical reality. The GROD policy, which was trained on real multi-view data where all views are physically consistent, may behave unpredictably when given views from the synthesis model that contain subtle inconsistencies.

The consequence. Errors in multi-view synthesis could masquerade as policy failures in the evaluation results. If the policy fails on a novel object task in simulation, it might be because the synthesized wrist camera view shows the object in a slightly wrong position, causing a grasp miss — not because the policy cannot handle the novel object. The paper's OOD correlation results (Pearson r of 0.56–0.91 in Fig. 9) suggest that this is not a catastrophic failure — the predictions do correlate with reality overall — but the unexplained variance could be partially attributable to multi-view synthesis artifacts. The weaker correlation for small distractors (Pearson r = 0.56) is a plausible candidate: small objects are more sensitive to subtle misplacement in synthesized views than large distractors or background changes.

The safety evaluation inherits this vulnerability. If the multi-view synthesis produces an inconsistent view of a safety-critical scene (e.g., the scissors appear in slightly different positions in different views), the policy might react to the inconsistency rather than to the safety-relevant objects. A predicted unsafe behavior might be an artifact of the multi-view inconsistency rather than a genuine policy vulnerability.

What evidence exists in the paper. None directly. The paper does not ablate the multi-view synthesis step — there is no comparison of rollouts with synthetically completed views against rollouts with manually captured multi-view images of the same OOD scene. Fig. 7 shows an example of multi-view synthesis output, demonstrating that the process produces visually plausible results, but visual plausibility does not guarantee that the synthesized views are metrically accurate or that the policy responds to them as it would to real views. The OOD correlation results provide indirect evidence that multi-view synthesis errors are not dominating the signal, but the magnitude of synthesis-induced error cannot be estimated from the current experiments.

Mitigation status. Not addressed. The paper describes the multi-view synthesis step (Section 4) without discussing its potential as an error source. Future work directions in Section 7 focus on video generation fidelity and temporal consistency, not on multi-view synthesis accuracy. A natural mitigation would be to compare OOD evaluation accuracy when using (a) fully synthetic multi-view from edited overhead, (b) edited overhead only with real other views (if available), and (c) fully real multi-view from physically constructed OOD scenes. Such a comparison would isolate the contribution of multi-view synthesis error to the overall prediction inaccuracy, but it is not performed or proposed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new algorithm or architectural innovation. Its contribution is methodological reframing: it demonstrates that video foundation models, when properly integrated into an evaluation pipeline, can serve as generalist simulators capable of replacing hardware testing for a range of policy evaluation questions that were previously considered beyond the scope of learned simulators. This reframing has several specific consequences for how the field thinks about evaluation infrastructure.

From physics simulators to learned simulators as primary evaluation tools. Before this work, the dominant assumption was that OOD policy evaluation—understanding how a policy generalizes to novel objects, backgrounds, and safety-critical scenarios—required either physics simulation with per-scenario asset creation or physical hardware testing. Physics simulators provide structural priors (explicit geometry, dynamics, material properties) that guarantee generalization to novel configurations, but at the cost of substantial per-scenario engineering. Video models provide no such structural guarantees—they learn dynamics from data and can fail in unpredictable ways—but they scale to diverse scenarios without per-scenario effort. This paper provides the first systematic evidence that the learned-simulator approach can produce useful predictive signal for OOD evaluation, including safety red-teaming, when built on a sufficiently powerful video foundation model. The evidence is partial—validated on only five pick-and-place tasks, eight policy checkpoints from a single architecture, and with systematic absolute miscalibration—but it shifts the burden of proof. Before this paper, the default position was "video models cannot be trusted for OOD evaluation because they lack explicit physics." After this paper, the position shifts to "video models can provide useful relative signal for OOD evaluation under certain conditions; the interesting question is characterizing those conditions." This is a reframing of what is possible, not a solved problem.

Reconciling the tension between interaction fidelity and coverage. A persistent tension in robot evaluation is that high-fidelity evaluation (hardware testing) provides low coverage (few scenarios), while high-coverage evaluation (simulation, benchmarks) provides low fidelity (unrealistic physics, visual domain gap). This paper demonstrates a point on the tradeoff curve that did not previously exist: evaluation with moderate fidelity (rank-consistent but absolutely miscalibrated, good on visual generalization but weak on contact-rich manipulation) and high coverage (arbitrary scene edits via generative models, programmatic scenario generation). The paper does not claim to achieve both high fidelity and high coverage simultaneously—the Discussion explicitly acknowledges failures in contact-rich manipulation and the 8-second horizon limitation. But by demonstrating that the tradeoff can yield actionable evaluation signal (e.g., correctly ranking policies, correctly ordering generalization axes by difficulty, discovering replicable safety violations), the paper opens a new category of evaluation tool: the learned, generative evaluator that sacrifices some fidelity for dramatic gains in coverage and iteration speed. This category did not previously exist as a validated option.

What changes about safety evaluation methodology. The paper's most conceptually significant move is extending video model evaluation to safety red-teaming. Prior safety evaluation for manipulation has been dominated by non-interactive benchmarks (ASIMOV, MM-SafetyBench) that test whether models can reason about safety given static scenes. The limitation of such benchmarks, as the paper correctly identifies, is that they cannot assess whether a policy will execute safely during closed-loop control, because safety depends on the sequence of actions the policy takes, not just its initial judgment. By showing that generated safety scenarios produce policy behaviors that replicate on hardware (Fig. 10), the paper demonstrates that a video model can evaluate the dynamic, closed-loop safety properties of a policy. This is not an incremental extension of non-interactive benchmarks—it is a different kind of evaluation altogether, one that was previously only available through hardware testing or detailed physics simulation. The paper does not provide a complete safety evaluation methodology (the false positive/negative rates are unknown, coverage of the safety scenario space is uncharacterized), but it establishes that the approach is viable in principle. This shifts safety evaluation from a purely perceptual/reasoning problem to one that can be addressed through generative simulation.

Research directions that become more attractive. The paper makes several lines of research newly tractable or higher-priority. First, verifier robustness for video models becomes a central bottleneck—the equivalent of reward model over-optimization in RLHF, but for learned simulators. The paper's finding that the video model is rank-consistent but absolutely miscalibrated suggests that improving the calibration of video model predictions (not just their visual quality) is a distinct and important research objective. Second, scene generation for evaluation coverage becomes a first-class design problem: the paper's use of Gemini 2.5 Pro to filter safety scenarios is a primitive version of automated test-case generation, and scaling this to systematic coverage of the safety-relevant scenario space is a natural next step. Third, multi-view completion as a bottleneck—the paper's OOD pipeline depends on single-view-to-multi-view synthesis, and the quality of this synthesis directly affects evaluation fidelity, but it is completely uncharacterized. Understanding when multi-view synthesis errors corrupt policy evaluation, and how to make synthesis robust to OOD objects, becomes a high-priority research question.

Research directions that become less attractive. The paper implicitly argues against manual, per-scenario physics simulation as the default approach to OOD evaluation. If a learned video model can provide useful relative signal across diverse OOD conditions with no per-scenario engineering, the cost-benefit analysis for detailed physics simulation shifts. Physics simulators remain essential when absolute calibration or contact-rich fidelity is required—the paper is clear that video models fail at contact-rich manipulation of small novel objects. But for the high-throughput, relative-comparison evaluation that dominates iterative policy development (comparing checkpoints, identifying which generalization axes matter most), the paper suggests that learned simulation may be the more practical tool. This does not make physics simulation obsolete, but it narrows the set of use cases where it is the clearly superior choice.

What kind of contribution this is. This is an existence proof at the systems level: it demonstrates that a specific combination of existing technologies (Veo video model + NanoBanana image editing + multi-view completion + GROD policy) can produce evaluation predictions that correlate with reality across a range of conditions. The paper does not claim to have solved evaluation, to have identified fundamental principles, or to have produced a deployable tool. The contribution is the demonstration that this combination works at all—that the components can be integrated into a system that produces signal where previously there was no alternative to hardware testing. The gap between this existence proof and a practical evaluation infrastructure is substantial (cost unaccounted for, calibration unresolved, safety validation incomplete, scope limited to short-horizon pick-and-place), but the paper has made the case that closing this gap is a worthwhile engineering and research endeavor.

Follow-Up Research This Work Enables

Systematic characterization of video model failure modes as a function of scene complexity. The paper provides anecdotal evidence that the video model fails at contact-rich manipulation of small objects (Fig. 11) but does not systematically characterize when predictions degrade. A critical follow-up would map the boundary between "predictive" and "non-predictive" regimes of video model evaluation. Concretely: design a benchmark of 20–30 manipulation tasks spanning a spectrum from simple pick-and-place (similar to the paper's five tasks) to contact-rich insertion (peg-in-hole, cable routing, zipper closing) to deformable object manipulation (cloth folding, dough kneading). For each task, collect paired real-world and video-model evaluations of 3–5 policy checkpoints, measure MMRV and absolute calibration error, and determine at what point along the complexity spectrum the rank correlation drops below a useful threshold (e.g., MMRV > 0.2). The paper's current results suggest the boundary lies somewhere between background/distractor generalization (where correlation is strong) and novel object manipulation (where absolute performance is low and discrimination is difficult despite strong correlation metrics). A systematic characterization would enable practitioners to determine whether a given evaluation task is within the "trusted regime" of video model evaluation.

Isolating the contribution of multi-view synthesis error to OOD evaluation inaccuracy. The paper's OOD pipeline synthesizes three camera views from a single edited overhead image using a fine-tuned Veo2 model, but never measures how much error this step introduces. A strong follow-up would be a controlled experiment with three conditions: (a) Fully synthetic OOD scenes: edited overhead + synthesized other views (the paper's current pipeline). (b) Partial synthetic OOD scenes: edited overhead + real other views from the physically constructed OOD scene (isolating multi-view synthesis as the variable). (c) Fully real OOD scenes: all four views physically captured. Running the same policy evaluations across these three conditions for the four generalization axes in Fig. 9 would quantify how much of the gap between predicted and real success rates is attributable to multi-view synthesis as opposed to video generation dynamics. If condition (b) substantially outperforms condition (a) in correlation with condition (c), multi-view synthesis is a major bottleneck and improving it should be prioritized. If the gap is small, the bottleneck lies in the video generation dynamics, and effort should focus there.

Cheap difficulty estimation for video model evaluation: predicting when the video model will be reliable without running the full rollout. The paper's evaluation procedure requires generating a full 8-second multi-view video and having a human score it—an expensive process that limits scalability. A natural extension, borrowing from the compute-optimal scaling literature, is to predict the reliability of the video model's evaluation from cheap features of the initial scene and instruction, before committing to a full rollout. For example: train a lightweight classifier that takes the initial multi-view observation, task instruction, and policy identity as input, and predicts whether the video model's success/failure judgment will agree with a hypothetical real-world trial. This classifier could be trained on the paper's 1600+ paired (simulated, real) trials. If reliable, it would enable adaptive evaluation: for scenes where the video model is predicted to be reliable, use video evaluation and skip hardware testing; for scenes where the video model is predicted to be unreliable, fall back to hardware or flag for human review. This directly addresses the unaccounted difficulty estimation cost critique: the cheap classifier amortizes the expensive video generation by routing only "confident" scenes to simulation.

Calibration of video model success predictions using real-world anchoring. The paper demonstrates strong rank correlation but systematic absolute miscalibration. A pragmatic follow-up would develop a calibration procedure that maps raw video model success rates to unbiased estimates of real success rates. The simplest approach: collect a small set of paired (simulated, real) trials for a reference policy and task, fit a calibration curve (e.g., logistic regression mapping predicted rate to real rate), and apply this curve to new policies and tasks. The key empirical question is whether calibration transfers: does a calibration curve fit on Policy A and Task 1 generalize to Policy B and Task 2? If calibration transfer is poor (the miscalibration is policy- or task-dependent), more sophisticated approaches would be needed, such as per-task calibration with minimal real-world data or learned calibration models that condition on task and policy features. A study testing calibration transfer across the eight policy checkpoints and five tasks from the paper's nominal evaluation would immediately establish whether this is a viable path to turning the video model from a ranking tool into an absolute performance estimator.

Scaling safety scenario generation for coverage: measuring recall against a known vulnerability inventory. The paper's safety evaluation demonstrates that the approach can find vulnerabilities but provides no metric of how many vulnerabilities are found or missed. A strong follow-up would construct a policy with deliberately implanted safety vulnerabilities (e.g., a policy fine-tuned to ignore obstacles near drop-off locations, or to use excessive force when grasping fragile-looking objects) and then measure how many of these known vulnerabilities the video model's safety red-teaming pipeline discovers. This would require: (a) creating a policy with N known vulnerabilities (e.g., 10–20 specific, documented failure modes), (b) generating M safety scenarios via the editing + filtering pipeline, (c) rolling out the policy in simulation and counting discovered vulnerabilities, and (d) computing recall (discovered / N) as a function of M (number of generated scenarios). This experiment would characterize the efficiency of safety red-teaming: how many scenarios must be generated to achieve, say, 80% recall of known vulnerabilities? If recall saturates quickly (e.g., 100 scenarios find most vulnerabilities), the approach is practical. If recall grows slowly (thousands of scenarios needed), automated scenario generation at scale becomes the bottleneck. The paper's current results cannot distinguish these regimes.

Generalization to policy architectures beyond GROD. The paper validates predictions against eight GROD checkpoints, all variants of the same VLA architecture. A critical stress-test would evaluate whether the video model's predictive signal transfers to policies with substantially different architectures (e.g., RT-2, Octo, ACT, Diffusion Policy) or training paradigms (e.g., policies trained on different datasets, with different action representations, or on different robot embodiments). The experiment: select 3–4 policies from diverse architecture families that have been evaluated on the same or similar ALOHA 2 tasks, run the paper's nominal evaluation protocol (80 scene-instruction combinations from Section 3.1) in both simulation and hardware, and measure whether MMRV and Pearson r remain in the 0.03–0.15 and 0.56–0.91 ranges observed for GROD variants. If the video model's predictions remain accurate, the approach is robust to policy architecture and the paper's validated scope expands substantially. If accuracy degrades—for example, because the video model was fine-tuned on data from GROD-like policies and overfits to their visual and behavioral patterns—then the approach is policy-specific and would need retraining for each new policy family. This is a negative result that would be equally informative: it would establish that learned simulators inherit the biases of their training data in ways that limit their generality as evaluators.

Practical Applications and Downstream Use Cases

Iterative policy development with fast ranking feedback. The most directly actionable use case is accelerating the inner loop of policy training. When developing generalist robot policies, training engineers routinely compare multiple checkpoints—different data mixtures, different hyperparameters, different architectural tweaks—and need to know which variant performs best. The paper demonstrates that the video model can correctly rank eight GROD checkpoints across 80 scene-instruction combinations (Fig. 4, MMRV = 0.03). In a production training workflow, this means replacing a multi-day hardware evaluation cycle (setting up scenes, running 80+ trials per checkpoint, manually scoring outcomes) with a compute-based cycle that can run asynchronously. The video model does not need to produce calibrated absolute success rates for this use case—the training engineer only needs to know that Checkpoint 3 outperforms Checkpoint 7. The $4\times$ efficiency improvement claimed in the paper's nominal evaluation (16 generations matching 64 in the compute-optimal scaling literature) has a direct analog here: the video model can provide ranking feedback at a fraction of the wall-clock time of hardware testing, enabling faster iteration. The key adoption question is whether the video model's ranking remains reliable when the policy changes are small (neighboring checkpoints) versus large (different architectures), which the paper does not fully answer but establishes as plausible for within-architecture comparisons.

Pre-deployment generalization screening for novel environments. Before deploying a robot policy to a new environment—a different room layout, different lighting, different background—a deployment team needs to know whether the policy will generalize or fail. The paper's OOD results (Fig. 8, Fig. 9) demonstrate that the video model can predict the relative impact of different generalization axes and can rank policies by their robustness to specific axes. In practice, a deployment team could: (a) capture a single overhead image of the target environment, (b) use the image editing pipeline to insert the robot's task-relevant objects into the scene, (c) run the policy in simulation via the video model, and (d) determine whether the predicted success rate meets a deployment threshold. Because the absolute success rates are miscalibrated, the threshold would need to be defined relative to a known reference (e.g., "predicted success rate must be at least 80% of the predicted success rate on the training environment"). The paper's strong rank correlation (Pearson 0.56–0.91 in Fig. 9) suggests that relative comparisons between known and novel environments would be reliable, even if absolute predictions are not. This screening could be performed for tens or hundreds of candidate deployment environments—something that would be prohibitively expensive with hardware testing—and only environments that pass the screen would advance to hardware validation.

Safety vulnerability scanning during continuous integration. As policies are updated (new training data, new architectures, new capabilities), their safety properties can change in unexpected ways. A policy that was safe in version 1.0 may develop new failure modes in version 1.1 because a capability improvement in one domain introduced a regression in another. The paper's safety red-teaming pipeline (Section 5) provides a blueprint for automated safety regression testing: maintain a library of safety-critical scenarios (generated through the editing + filtering pipeline), and for each new policy checkpoint, run the full suite of safety scenarios in simulation to detect new vulnerabilities. The two examples in Fig. 10 demonstrate that the video model can discover real, replicable safety violations. In a CI/CD pipeline, this would serve as a gate: if the new checkpoint exhibits safety violations that the previous checkpoint did not, the update is blocked pending investigation. The key practical requirement is characterization of the false positive rate—if the video model frequently "cries wolf" with artifacts that do not replicate in hardware, the CI gate would be constantly triggered and would be ignored. The paper does not provide this characterization, so deploying such a system today would require a calibration phase where predicted violations are verified on hardware to establish the pipeline's precision.

When to Prefer This Method

The paper does not propose a formal decision rule or position itself against named alternatives with explicit tradeoffs. However, the results implicitly define the conditions under which video model evaluation is likely to be useful versus when hardware testing or physics simulation should be preferred. These conditions emerge from the paper's demonstrated strengths and acknowledged limitations:

Prefer video model evaluation when:

The evaluation question is comparative (which policy is better? which generalization axis matters most?) rather than absolute (does the policy exceed 95% success?), because the video model provides strong rank correlation but systematic absolute miscalibration.
The generalization axes of interest are visual (backgrounds, distractors, object appearance) rather than contact-dynamic (precision insertion, force-sensitive grasping, deformable objects), because the paper demonstrates strong OOD predictive signal for visual generalization but acknowledges failures in contact-rich manipulation.
The policy architecture is consistent with the policies used to fine-tune the video model (e.g., other GROD-like VLA policies) because all validation was performed within a single architecture family, and transfer to architecturally different policies is untested.
Rapid iteration and broad coverage are more important than per-scenario accuracy, because video model evaluation enables testing dozens of generalization conditions or safety scenarios that would be impractical to set up physically.

Prefer hardware evaluation when:

Absolute calibration is required (deployment certification, safety guarantees, compliance), because the video model's absolute predictions are systematically lower than real rates and the bias is uncharacterized.
The task involves contact-rich manipulation of small or novel objects where the video model is known to hallucinate (Fig. 11).
The policy architecture differs substantially from those used in video model fine-tuning, because the model's predictions may not transfer.
The evaluation horizon exceeds 8 seconds, because the current video model does not support longer episodes, and error accumulation over longer horizons is uncharacterized.