Robot Learning from a Physical World Model

ArXiv: 2511.07416

🎯 Pitch

Video generators can hallucinate convincing robot demonstrations from a single image, but directly copying those pixel motions ignores physics and causes real-world grasping failures ~20% of the time. PhysWorld reconstructs an interactable physical simulator directly from the generated video, then trains a residual RL policy inside that proxy to ground visual plans into physically executable actions, eliminating such failures and lifting zero-shot success from 67% to 82% across ten manipulation tasks.

---

1. Executive Summary

PhysWorld introduces a framework that couples video generation with physical world modeling to enable zero-shot robotic manipulation from a single RGB-D image and a language command. Given a task prompt, the system generates a task-conditioned video (using Veo3), reconstructs a physically interactable scene from that video through geometry-aligned 4D reconstruction, textured mesh generation, and physical property estimation, and then learns object-centric residual reinforcement learning policies that track object poses within this reconstructed world model — transforming pixel-level visual guidance into physically executable actions without any real-world robot data collection. Across 10 diverse real-world manipulation tasks, PhysWorld achieves an average success rate of 82%, substantially outperforming the strongest zero-shot baseline (67%) by eliminating grasping failures (from 18% to 3%) and tracking failures (from 5% to 0%), establishing that physical feedback from a reconstructed world model is necessary for robust policy learning from generated videos only when the world model provides corrective feedback that compensates for compounding errors in grasping and planning.

2. Context and Motivation

The Core Problem: Translating Generated Videos into Physically Feasible Robot Actions

The fundamental question this paper tackles is: if video generation models can produce photorealistic demonstrations of tasks from a single image and a language prompt, how can we convert those visual demonstrations into physically accurate robotic actions without collecting real-world robot data? This matters because video generation models, trained on vast amounts of Internet data, exhibit remarkable generalization — they can synthesize plausible task executions across a wide variety of scenes, objects, and instructions. If these generated demonstrations could be reliably translated into robot actions, they would provide an essentially unlimited source of training data for robotic manipulation, circumventing the enormous cost and labor involved in collecting real-world robot demonstrations.

This gap between visual plausibility and physical feasibility is not incidental — it represents a fundamental tension. Video generators are trained to produce frames that look realistic to a human observer. They optimize for photorealism and semantic consistency, not for the physical constraints that govern how objects move, collide, and interact under forces. A generated video of "pour the tomato from the pan onto the plate" might look convincing while depicting object trajectories that are physically impossible — a pan tilting at an angle that would cause the tomato to slide prematurely, or an object motion that violates rigid-body dynamics. When a robot attempts to execute a policy derived directly from such pixel-level guidance, it will fail because the real world enforces physics whether the video was consistent with it or not.

The paper frames this explicitly:

"Video generation, despite its generalization power, only provides visual plausibility rather than physical accuracy for robotic tasks, whereas robots operating in the real world require physically accurate actions to interact with objects correctly."

This is the core tension that motivates the entire framework. The solution, as the paper argues, cannot come from making video generation more physically accurate — that would require training on data that doesn't exist at scale — but rather from introducing an intermediate representation that imposes physical consistency on the visual guidance.

Why This Problem Is Important: The Data Bottleneck in Robot Learning

The significance of this problem extends far beyond a technical curiosity about video generation. It touches on what is arguably the central bottleneck in modern robot learning: data scarcity. Robotic manipulation policies, particularly those trained via imitation learning or reinforcement learning, require large numbers of demonstrations or environment interactions. Collecting these on real hardware is expensive, time-consuming, and fragile — robots break, environments change, and human supervision is needed for reset and safety monitoring.

Several approaches have been developed to address this bottleneck:

• Teleoperation-based collection (e.g., [16]–[26] from the related work): Humans operate robots to collect demonstrations, but this is labor-intensive and limits scaling.
• Sim-to-real transfer: Policies are trained in simulation and transferred to reality (e.g., [50]–[62] from related work). This solves the data scaling problem but introduces a simulation-to-reality gap — policies trained in simplified simulators often fail when deployed in the messy, unpredictable real world.
• Real-to-sim-to-real: Real-world scenes are reconstructed in simulation, policies are trained there, and then deployed back to reality (e.g., [43], [44], [46], [47]). This is closer to the paper's approach but traditionally requires multi-view captures and dedicated reconstruction pipelines that don't work with monocular generated videos.

Video generation offers a radically different path: rather than collecting real demonstrations or manually constructing simulation environments, we can generate demonstrations on-demand from a single image and language prompt. The training data problem becomes a generation problem. But this promise remains unfulfilled as long as the generated demonstrations can't be reliably converted into actions.

The paper's approach — coupling video generation with physical world modeling — represents a synthesis that could, if successful, fundamentally change the economics of robot learning. If a single RGB-D image and a language command are sufficient to generate a physically grounded policy, the marginal cost of adding a new task drops to near zero. This is the practical significance.

There is also theoretical significance: the paper formalizes and provides evidence for the claim that physical feedback during learning is necessary when the source of behavioral guidance (video generation) does not itself respect physics. The ablation in Figure 4 comparing directly retargeting video motions ([8]) versus learning within a physical world model shows a 15-percentage-point gap in average success rate (67% vs. 82%). The failure mode analysis in Figure 6 attributes this specifically to reduced grasping and tracking failures, suggesting that the physical world model provides corrective feedback that compensates for errors that a purely visual pipeline has no mechanism to detect or correct. This is an empirical finding with implications for how we think about the relationship between perception, generation, and action.

Prior Approaches and Their Shortcomings

The paper situates itself against three broad categories of prior work, each of which partially addresses the problem but leaves a critical gap:

Approach 1: Learn Action Models from Generated Videos Using Real Robot Data

Several works ([1]–[5] in the references) train inverse dynamics models or visuomotor policies that take generated video frames as input and predict robot actions, using large real-world robot datasets for supervision. The idea is that if a model learns the mapping from pixels to actions from real data, it can then apply that mapping to generated pixels. This is conceptually appealing but suffers from a fundamental circularity: it requires exactly the kind of large-scale real-world data collection that video generation was supposed to replace. The paper states this as its primary criticism:

"Such methods generally rely on large-scale real-world demonstrations for alignment, while collecting them at scale is costly and labor-intensive."

If the promise of video generation for robotics is to eliminate the data bottleneck, methods that require real demonstrations trained on real robot data don't fulfill that promise. They use video generation as a data augmentation technique rather than as a replacement for real data.

A secondary concern, which the paper doesn't elaborate on but is implicit in the approach: these methods learn a correlation between visual patterns and actions from real data, but the action predictions have no physical grounding. If the generated video contains a physically implausible object trajectory, the action model will faithfully predict actions that attempt to reproduce it — and those actions will likely fail in the real world because physical reality won't cooperate.

Approach 2: Extract Actions by Directly Following Visual Cues from Generated Videos

A more direct approach, and the one that serves as the paper's primary baseline, is to skip the real-robot-data requirement entirely and extract actions by tracking visual motion representations from the generated video. Methods include:

• Optical flow tracking (AVDC, [6]): Estimate dense pixel-level motion from the generated video and command the robot to produce matching motions. The problem is that optical flow is inherently ambiguous — it represents 2D apparent motion, not 3D physical motion, so depth changes and occlusions produce flow patterns that are difficult to translate into end-effector commands.

• Sparse point tracking (Gen2Act, [7]): Track a sparse set of points through the video and use their trajectories as motion targets. This is more robust than dense flow but still suffers from tracking drift, especially under occlusion and motion blur — common artifacts in generated videos.

• Object pose tracking (RIGVid, [8]): Estimate object poses (3D position and orientation) from the generated video frames and then combine them with a grasping model and motion planner to execute the task. This is the strongest baseline the paper compares against (67% success rate in Figure 4) because object poses provide a more stable and semantically meaningful motion representation than flow or point tracks.

The paper's critique of these direct-retargeting methods is sharp and specific:

"Directly retargeting video motions to robots neglects underlying physical constraints, often leading to inaccurate manipulations."

The failure modes are instructive. Consider the task "put the book in the bookshelf." The generated video shows the book moving along a trajectory into the shelf. A direct-retargeting approach extracts that trajectory and commands the robot to follow it. But what if the trajectory in the video involves a slight clipping through the shelf edge — visually imperceptible but physically impossible? The motion planner will either reject the trajectory (causing a planning failure) or attempt to execute it (causing the book to collide with the shelf). In either case, the task fails, and there is no mechanism for recovery because there is no physical feedback to detect or correct the problem.

The failure mode analysis in Figure 6 quantifies these effects. Compared to PhysWorld, the direct-retargeting baseline (RIGVid, [8]) shows:

• Grasping failures: 18% vs. 3%. The grasping model, used in isolation, makes predictions that work in open-loop settings but fail when the object isn't exactly where expected or when the generated video's object pose estimate is slightly off. Without physical feedback, there's no way to adjust.
• Tracking failures: 5% vs. 0%. Object pose estimation from generated videos is imperfect, especially under motion blur or occlusion. Directly following these estimates means the robot follows inaccurate trajectories.
• Dynamics failures: 3% vs. 2%. Surprisingly, this category shows a smaller gap, suggesting that motion planning is relatively robust — but the upstream errors in grasping and tracking cascade into overall task failure.

The key insight from this comparison is that the failures are not primarily perceptual (the generated videos do show the right general motions) but rather interactive: the robot fails because it can't adapt its actions to the physical reality it encounters during execution.

Approach 3: Real-to-Sim-to-Real with Dedicated Multi-View Capture

A third family of approaches reconstructs a physical scene from real observations, imports it into a simulator, and trains policies there. This is the "real-to-sim-to-real" paradigm (e.g., [43], [44], [46], [47], [50]–[62]). By training in simulation with physical feedback and then transferring to reality, these methods address the physics problem that direct-retargeting misses.

However, the paper identifies a critical limitation that makes these approaches incompatible with generated videos:

"To obtain complete object and scene textured meshes or Gaussian splats, prior works require dedicated multi-view captures for reconstruction, making them difficult to apply to monocular generated videos."

Traditional reconstruction pipelines (Polycam, BundleSDF [65]) need multiple views of the scene — either from moving a camera around or from multiple calibrated cameras. Generated videos, by contrast, are monocular: they provide only a single viewpoint that may not even move much (imagine a video of a hand pouring from a pan — the camera might be static, looking at the scene from one angle). Objects and scene elements that are occluded in the video simply aren't observable, and standard multi-view reconstruction can't recover them.

This is a more subtle gap than the previous two. It's not just about the method of learning (direct retargeting vs. simulation-based learning) but about the compatibility of the available input data (a single monocular video) with the requirements of the reconstruction pipeline (multiple views). The paper's contribution here is recognizing that generated videos present a unique reconstruction challenge that existing real-to-sim pipelines aren't designed to handle.

How PhysWorld Positions Itself

PhysWorld positions itself at the intersection of two ideas that prior work has treated separately:

1. Video generation provides generalizable visual guidance. Following the insight of work like [6]–[8], PhysWorld uses generated videos as the source of task-relevant motion information. It does not require real robot demonstrations for alignment, distinguishing itself from [1]–[5]. It uses object poses as the tracking target (the most robust of the visual cue options), consistent with [8].

2. Physical simulation provides corrective feedback. Following the insight of real-to-sim-to-real work, PhysWorld introduces a proxy physical world model where policies can be trained with realistic physics. This distinguishes it from [6]–[8], which lack physical feedback entirely, and explains the large performance gap in Figure 4.

The synthesis — and the paper's primary technical contribution — is the coupling mechanism: the physical world model is built from the generated video itself. This is what makes the framework self-contained. It does not require a pre-built simulation environment, multi-view capture, or any human-provided 3D models. A single RGB-D image and a language command are sufficient.

The paper explicitly states this positioning:

"The core of PhysWorld lies in the synergy between physical world reconstruction and video generation: video generation provides pixel-level visual guidance for task execution, while the physical world model offers realistic feedback for learning from the generated visual guidance."

The key design tensions that emerge from this positioning are:

• Monocular reconstruction requires generative priors. Since the generated video only shows part of the scene (objects may be occluded, backgrounds may be hidden), PhysWorld uses image-to-3D generative models to hallucinate the unobserved geometry. This is a deliberate tradeoff: the completed geometry may not perfectly match reality (the paper acknowledges 7% reconstruction errors in Figure 6), but it's good enough to provide physical feedback that dramatically improves policy robustness over no feedback at all.

• Object-centric learning is more robust than embodiment-centric learning. Generated videos often hallucinate hands or have inconsistent hand kinematics (the paper reports embodiment-centric success rates of 30% and 10% on two tasks in Table II, compared to 90% and 80% for object-centric). PhysWorld focuses on tracking what the object does rather than what the embodiment does, which is both easier to estimate from video and more robust to physical feedback (object dynamics are simpler than contact-rich hand-object interactions).

• Residual RL combines the efficiency of classical methods with the robustness of learning. Direct grasping and motion planning ([8]) provides a reasonable baseline action, but it's brittle because it has no mechanism for error recovery. Pure RL from scratch is robust but slow to converge and requires careful reward design. Residual RL (Equation 5) sits in the middle: the policy only needs to learn corrections to the baseline actions, which is a much easier learning problem (Figure 7 shows convergence within a few hundred iterations compared to RL from scratch).

The paper's relationship to prior work is therefore one of selective synthesis with a clear technical gap-filling motivation: take the visual guidance pipeline from [8], add physical world modeling inspired by real-to-sim-to-real work but adapted for monocular generated videos via generative priors, and use residual RL to bridge the two. The result is a framework that inherits the generalization of video generation while adding the physical grounding that was missing — enabling zero-shot manipulation with substantially higher reliability than either approach alone.

3. Technical Approach

3.1 Reader Orientation

PhysWorld is a zero-shot robotic manipulation system that takes a single RGB-D image and a language command (e.g., "pour the tomato from the pan onto the plate") and outputs physically executable robot arm trajectories to accomplish that task. The system solves the problem of bridging generated video demonstrations — which are visually plausible but physically ungrounded — and real-world robot actions by reconstructing a physically interactable simulation environment from the generated video itself and then training a policy inside that environment with realistic physics feedback, eliminating entirely the need for real-world robot data collection.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five sequential stages, each feeding into the next:

1. Task-Conditioned Video Generation: Given an RGB-D image $I_0$ and a task prompt, an image-to-video model (Veo3) generates $T$ future frames $\{I_1, \ldots, I_T\}$ showing the task being completed.

2. Geometry-Aligned 4D Reconstruction: The generated video frames are converted into temporally consistent, metric-scale depth maps and point clouds $\{P_t\}_{t=0}^T$, using monocular depth estimation calibrated against the real depth sensor reading $D_0$.

3. Textured Mesh Generation: The first frame is decomposed into objects and background. Each object crop is fed into an image-to-3D generator to produce a canonical textured mesh. The background (with objects inpainted out) is converted to a mesh via height-map triangulation, using ray-casting to infer geometry in originally occluded regions.

4. Physical Scene Reconstruction and Alignment: The object and background meshes are assembled, assigned physical properties (mass, friction) via VLM queries, aligned to the world gravity axis, and collision-optimized to remove initial penetrations. The result is a physically interactable digital twin.

5. Object-Centric Residual RL: Object pose trajectories $\{x_t^o\}$ are extracted from the 4D reconstruction as learning targets. A reinforcement learning policy $\pi_\theta$ is trained inside the physical world model to track these object poses, outputting residual corrections $[\Delta p_t, \omega_t]$ on top of baseline grasp-and-plan actions. The trained policy is deployed zero-shot on the real robot.

Information flows strictly forward: image + prompt → video → 4D geometry → meshes → physical scene → RL policy → real-world actions. Each stage transforms the representation from less physically grounded to more physically grounded.

3.3 Roadmap for the Deep Dive

• First, the video generation step, covering which models are used, what constitutes a "usable" video, and how generation quality cascades into downstream performance (Table I).

• Second, the geometry-aligned 4D reconstruction, because it establishes the metric 3D structure and motion that all subsequent steps depend on — without calibrated depth, there are no object poses, no mesh registration, and no learning targets.

• Third, the textured mesh generation pipeline, which is the bridge from 2.5D point clouds to complete 3D geometry suitable for physics simulation — including the critical object-background decomposition and the generative priors for hallucinating unobserved geometry.

• Fourth, the physical scene reconstruction and alignment, which takes the meshes and makes them simulation-ready through property estimation, gravity alignment, and collision optimization — this is where the "physical" in physical world modeling is instantiated.

• Fifth, the object-centric residual RL formulation, covering the learning targets (object pose trajectories), the observation and action spaces, the residual policy structure, the reward functions, and the PPO training procedure — this is where visual guidance becomes executable robot commands.

3.4 Detailed, Sentence-Based Technical Breakdown

PhysWorld is primarily a systems integration and empirical analysis paper whose core technical insight is that coupling video generation with explicit physical world reconstruction — and training residual policies within that reconstructed world — provides the physical feedback necessary to convert generated visual demonstrations into robust real-world manipulation, without any real robot data. The paper does not propose fundamentally new video generation, depth estimation, 3D generation, or RL algorithms; rather, it shows how to assemble these components into a working pipeline and demonstrates that the physical world model is the critical missing piece that prior zero-shot video-to-action approaches lacked.

---

Video Generation

Model choice and rationale. PhysWorld supports any image-to-video model with text conditioning. The primary experiments use Veo3 [63] because it achieves the highest usable-video ratio (70%, Table I). Additional models evaluated include Tesseract [11] (36%), CogVideoX1.5-5B [9] (4%), and Cosmos-2B [10] (2%). These numbers are not incidental — they reveal a structural requirement of the PhysWorld pipeline: the generated video must be of sufficient quality that object poses can be robustly recovered from it. Low-quality generation (blurred frames, inconsistent object appearance, abrupt viewpoint changes) causes the downstream pose estimation to fail, which breaks the entire pipeline.

Definition of "usable" video. A video is "usable" if object poses can be recovered robustly from it — meaning the downstream FoundationPose [69] tracker does not lose track of the manipulated object throughout the sequence. The paper generates 10 videos per model-task combination and reports the fraction of videos meeting this criterion (Table I). The sharp drop from Veo3 (70%) to open-source models (CogVideoX1.5-5B at 4%, Cosmos-2B at 2%) indicates that the pipeline's feasibility is currently tied to the generation quality of large proprietary models. The paper notes that "robotic data fine-tuning (e.g., Tesseract) tends to outperform generic generators," suggesting a path toward improved open-source performance.

Input-output specification. The video generator receives:

• Input: An RGB image $I_0$ (the robot's first-person observation) and a language task command (e.g., "Pour the tomato from the pan into the white plate").
• Output: $T$ future frames $\{I_1, \ldots, I_T\}$ depicting task completion. The value of $T$ is not explicitly stated in the paper, but the pipeline operates on the full generated sequence.

Why video generation rather than direct action prediction. The paper's choice of video generation as the initial step is motivated by generalization. Video models trained on Internet-scale data have seen an enormous variety of objects, scenes, and manipulation patterns. They can therefore produce plausible demonstrations for tasks and objects that no robot has ever encountered. If the system instead tried to predict actions directly from the image and prompt (without the intermediate video), it would need task-specific training data that doesn't exist for arbitrary open-world tasks. The video serves as a universal intermediate representation that captures task semantics and approximate motion patterns, which the physical world model then refines into physically accurate actions.

Practical considerations. Only one successful video generation is needed per task execution — the system does not require multiple videos or video selection. The remaining 30% of Veo3 generations that are "unusable" represent a failure mode of the pipeline, but since generation is cheap relative to real-world data collection, generating a few videos and selecting a usable one is practical. The paper does not detail the selection procedure, but it is implied that pose estimation success serves as the automatic filtering criterion.

---

Geometry-Aligned 4D Reconstruction

This stage converts the generated RGB video into metric-scale 3D geometry over time — a 4D spatio-temporal representation that captures both the static scene structure and the dynamic object motions. Without this step, there would be no way to extract object pose trajectories, register meshes to the scene, or define learning targets for the policy.

Why standard monocular depth estimation is insufficient. Off-the-shelf depth estimators produce depth maps that are scale-ambiguous — they can tell you relative depth (this pixel is farther than that pixel) but not absolute metric depth (this point is 0.47 meters away). For robotic manipulation, metric scale is non-negotiable: the robot needs to know where objects are in real centimeters to plan grasps and motions. Furthermore, generated videos have no inherent scale information whatsoever — the video model outputs pixels, not meters.

MegaSaM initialization. The paper uses MegaSaM [64] as the depth estimation backbone because it produces temporally consistent depth estimates $\{D'_0, \ldots, D'_T\}$ for each frame. Temporal consistency means that depth values for the same physical point remain stable across frames (no flickering or abrupt jumps), which is essential for extracting smooth object trajectories. MegaSaM achieves this by processing the video as a sequence rather than estimating each frame independently. However, MegaSaM's estimates are not aligned with real-world metric scales — they live in an arbitrary coordinate system relative to the video.

Metric calibration via the real depth sensor. This is where the "D" in RGB-D becomes crucial. The robot's real depth camera provides a ground-truth metric depth map $D_0$ for the first frame. The paper solves for a global scale $\alpha$ and shift $\beta$ that best align MegaSaM's estimate $D'_0$ with the real depth $D_0$:

$\min_{\alpha, \beta} \sum_{p \in \Omega} w_p \left( \alpha D'_0(p) + \beta - D_0(p) \right)^2$

where $\Omega$ denotes the set of valid pixels (those with valid depth in both MegaSaM's output and the real sensor), $D'_0(p)$ is MegaSaM's estimated depth at pixel $p$, $D_0(p)$ is the real depth sensor reading at pixel $p$, and $w_p$ are Huber weights that downweight outlier pixels. The parameters $\alpha$ (a multiplicative scale factor) and $\beta$ (an additive shift) are the optimization variables.

What it computes: a robust linear regression that finds the best affine transformation mapping the ambiguous MegaSaM depth estimates to real metric depth. For each valid pixel, the squared difference between the calibrated estimate $\alpha D'_0(p) + \beta$ and the ground truth $D_0(p)$ is computed. The Huber weights $w_p$ reduce the influence of pixels where MegaSaM's estimate is grossly inconsistent with the sensor reading (e.g., on object boundaries where depth is ambiguous). The optimization finds $(\alpha, \beta)$ that minimize the weighted sum of squared errors.

Why this form: an affine model (scale + shift) is the simplest transformation that can correct both the overall magnitude of the depth estimates (scale) and any constant offset (shift) — the two dominant error modes in monocular depth estimation. Using robust regression with Huber weights, rather than ordinary least squares, prevents a few badly-estimated pixels from distorting the calibration. The alternative — using a more complex nonlinear calibration — would risk overfitting to the specific first-frame geometry and failing to generalize to subsequent frames.

Application to all frames. Once $(\alpha, \beta)$ are computed from the first frame, the same parameters are applied to all subsequent frames:

$D_t(p) = \alpha D'_t(p) + \beta \quad \text{for all } t = 0, \ldots, T$

This assumes that the scale ambiguity in MegaSaM is consistent across the entire video — a reasonable assumption since the video is generated from a single model with consistent internal geometry representations.

Un-projection to point clouds. With known camera parameters (intrinsic matrix from the real camera), the calibrated depth maps $\{D_t\}$ are un-projected into 3D point clouds $\{P_t\}_{t=0}^T$. For each pixel $p = (u, v)$ with depth $d = D_t(u, v)$, the corresponding 3D point is computed by back-projecting the ray through $(u, v)$ to distance $d$. This produces a dynamic point cloud sequence representing the 3D geometry of the scene as it evolves through the generated video — the "4D" representation (3D + time).

Why this two-stage approach (MegaSaM → calibration) rather than using the real depth sensor for all frames. The real depth sensor only provides $D_0$ — one frame. The generated video produces $T$ future frames for which no real depth exists. Using MegaSaM for temporal extension and calibrating against the single available real depth frame provides metric scale for the entire sequence without requiring a depth sensor that can see the future. This is a clever use of the fact that the first frame corresponds to the robot's current observation, where real sensor data is available.

---

Textured Mesh Generation

The 4D point clouds $\{P_t\}$ provide geometry and motion, but point clouds cannot be directly used in physics simulators. Physics engines (the paper uses an unspecified simulator, likely MuJoCo or Isaac Sim based on the robotics conventions) require meshes — watertight surface representations with defined interiors and exteriors — to compute collision detection, contact forces, and dynamics. Converting point clouds to meshes is the standard approach, but the generated monocular video presents a unique challenge: objects and background are only partially visible from the single camera viewpoint.

The occlusion problem. When an object sits on a table, the camera sees the front and top of the object, but not its back or bottom. The table surface underneath the object is completely occluded. A naive surface reconstruction from the visible point cloud would produce a mesh with holes where the occluded geometry should be — a mesh that objects would fall through in simulation. Traditional multi-view reconstruction pipelines avoid this by capturing the scene from many angles, but PhysWorld only has one view.

Object-background decomposition. The solution begins by separating the first frame $I_0$ into objects and background:

1. Object segmentation: Objects (the tomato, pan, plate, etc.) are identified and segmented from $I_0$. The paper does not specify the segmentation method (likely a combination of SAM-style segmentation and task-prompt grounding), but the result is a set of object masks and corresponding object crops $I_o$.

2. Background inpainting: Object pixels are removed from $I_0$, leaving holes where the objects were. These holes are filled using a masked image inpainting model [66] (ObjectClear), producing a completed background image $I_b$ that shows what the scene would look like without the objects — the table surface, walls, shelves, etc. The inpainting model uses object-effect attention to generate contextually appropriate completions (e.g., continuing the wood grain pattern of a table where a pan was removed).

This decomposition is critical because objects and backgrounds are handled by fundamentally different reconstruction strategies, reflecting their different roles in simulation: objects move (they are the dynamic entities), while the background is static and serves as the supporting surface.

Object mesh generation via image-to-3D. For each object crop $I_o$, an image-to-3D generator [67] produces a canonical textured mesh $M^o$. The generator (likely based on TRELLIS or a similar architecture) takes a single RGB image of an object and outputs a complete 3D mesh with texture — crucially, it hallucinates the unseen back, bottom, and sides of the object based on priors learned from large-scale 3D asset datasets. This is the "generative prior" that enables reconstruction from a single view: the model has seen enough pans, tomatoes, and plates during training to infer what the back of a pan should look like given an image of its front.

Why image-to-3D rather than multi-view reconstruction. Multi-view reconstruction would require the generated video to show the object from multiple angles — which it typically doesn't (the camera is static in most manipulation scenarios). Image-to-3D generation fills in the missing geometry using learned priors, trading perfect fidelity to the specific object instance for completeness. The paper acknowledges this tradeoff implicitly: the 7% reconstruction errors in Figure 6 are partly attributable to mismatches between the generated mesh and the real object geometry.

Background mesh generation with geometry inference. Background reconstruction is more complex because it requires both the inpainted image $I_b$ and the corresponding 3D geometry $P^b$ — including in the regions that were originally occluded by objects. The paper uses an object-on-ground assumption: objects rest on the background (table, shelf, floor), so the occluded regions are either: (a) planar supporting surfaces (the table under the pan), or (b) extend to infinity (the wall behind the shelf) bounded by scene limits.

Concretely, for each pixel that was originally occluded by an object:

1. Cast a camera ray through that pixel into the 3D scene.
2. Find the nearest intersection with either: (a) the supporting plane (estimated from visible background points using RANSAC plane fitting), or (b) the scene boundary (a bounding volume defining the extent of the reconstruction).
3. Assign the intersection point to $P^b$.

This fills in the occluded geometry with a physically plausible guess: the table continues under the object, the wall continues behind the shelf. With the completed $(I_b, P^b)$, the background mesh $M^b$ is generated via height-map triangulation: the point cloud is projected onto a 2D grid from the camera viewpoint, neighboring pixels are connected to form triangles, and $I_b$ is applied as the texture.

Scene assembly via registration. The object meshes $\{M^o\}$ and background mesh $M^b$ are assembled into a complete scene by aligning them to the observed point cloud $P_0$. For each object, its canonical mesh is rigidly transformed (rotated and translated) to match the object's observed position and orientation in $P_0$, using point cloud registration (likely ICP or feature-based alignment). The background mesh is similarly aligned. The result is a complete, textured 3D scene that matches the first frame of the video.

Why this decomposition rather than reconstructing everything monolithically. Separating objects from backgrounds allows independent handling of the two fundamentally different entities: objects need complete geometry (via generative priors) because they'll be grasped and moved; backgrounds only need plausible geometry in occluded regions (via planar extrapolation) because they're static. Reconstructing everything monolithically would either leave holes in the object meshes (if using the background strategy) or produce implausible hallucinated background geometry (if using the generative object strategy).

---

Physical Scene Reconstruction and Alignment

The assembled scene meshes $\{M^o, M^b\}$ represent 3D geometry with textures, but they are not yet a physical world model. Three transformations are needed to make them simulation-ready: physical properties must be assigned, the scene must be aligned with gravity, and initial collisions must be resolved.

Physical property estimation via VLM queries. Physics simulators require material properties for each object — mass, friction coefficients, restitution — to compute realistic dynamics. Rather than hard-coding these (which would limit generalization to novel objects) or estimating them from video (which is extremely difficult from monocular input), PhysWorld queries a vision-language model (VLM):

"We query a VLM with the object category to obtain typical physical parameters, and assign the predicted values to each object and the background for subsequent physical simulation."

The VLM (likely a model like Gemini or GPT-4V) receives the object category as text (e.g., "tomato," "ceramic plate," "metal pan") and returns estimates of typical mass, friction, and other parameters based on its commonsense knowledge. For example, a tomato might be assigned a mass of ~0.15 kg and moderate friction; a metal pan might be ~1.0 kg with low friction on its bottom surface. The paper cites [68] as inspiration for this approach.

Why VLM-based estimation rather than learning from interaction. Learning physical properties from video or interaction data would require substantially more data and computation — and would be specific to each object instance. VLMs, trained on Internet text, have absorbed vast amounts of implicit physical knowledge (text descriptions of objects include their typical weight, material, and behavior). This zero-shot property estimation inherits the VLM's generalization: it works for any object category the VLM knows about, even if no physical interaction data exists.

Gravity alignment. The reconstructed scene lives in the camera coordinate frame — a coordinate system where the axes correspond to the camera's orientation (x-right, y-down, z-forward). This is not aligned with the world gravity axis, which points downward (conventionally the negative z-axis in simulation). Physics simulators assume gravity acts along a known axis, so the scene must be rotated so that "down" in the camera frame aligns with the world gravity vector.

The paper estimates the ground plane normal $\mathbf{n}$ from the background point cloud $P^b$ using RANSAC plane fitting (identifying the largest planar region, which should be the table, floor, or other supporting surface). It then computes the rotation $R_{\text{grav}}$ that aligns $\mathbf{n}$ with the world up-axis $\mathbf{e}_z = [0, 0, 1]^\top$:

$R_{\text{grav}} = \exp([\mathbf{u}]_\times \theta), \quad \theta = \arccos(\mathbf{n}^\top \mathbf{e}_z), \quad \mathbf{u} = \frac{\mathbf{n} \times \mathbf{e}_z}{\|\mathbf{n} \times \mathbf{e}_z\|}$

where $\mathbf{n}$ is the estimated ground plane normal, $\mathbf{e}_z$ is the world up-axis unit vector, $\theta$ is the angle between them (computed via arccos of their dot product), $\mathbf{u}$ is the normalized axis of rotation (the cross product gives a vector perpendicular to both $\mathbf{n}$ and $\mathbf{e}_z$, which is the axis around which to rotate), $[\mathbf{u}]_\times$ is the skew-symmetric matrix of $\mathbf{u}$ (the matrix form of the cross product, such that $[\mathbf{u}]_\times \mathbf{v} = \mathbf{u} \times \mathbf{v}$ for any vector $\mathbf{v}$), and $\exp(\cdot)$ is the matrix exponential that converts the axis-angle representation $[\mathbf{u}]_\times \theta$ into a rotation matrix.

What it computes: the minimal rotation (shortest angular path) that brings the estimated ground plane into alignment with the horizontal plane of the simulation world frame. The angle $\theta$ is found from the dot product; the axis $\mathbf{u}$ is the normalized cross product of the normal and target. The rotation matrix is constructed via the exponential map from the Lie algebra $\mathfrak{so}(3)$ to the rotation group $SO(3)$. Applying $R_{\text{grav}}$ to all mesh vertices rotates the entire scene so that gravity points downward in the simulation.

Why this form: the axis-angle representation with the exponential map is the standard minimal parameterization of 3D rotations. It automatically handles numerical edge cases (when $\mathbf{n}$ is already aligned with $\mathbf{e}_z$, $\theta = 0$ and the exponential map returns the identity rotation). The alternative — computing a rotation matrix from Euler angles — would require handling gimbal lock and wouldn't guarantee the minimal rotation.

Collision optimization. After assembly, objects may have slight penetrations with the background or with each other — the registration to $P_0$ is imperfect, and different meshes may slightly overlap at their boundaries. In a physics simulator, interpenetrating objects generate large, unrealistic contact forces at the first simulation step, which can cause objects to "explode" apart violently.

To resolve this, the paper formulates a penetration-minimization optimization. The background mesh is voxelized into a signed distance field (SDF) $\phi_{\text{bg}}$, where $\phi_{\text{bg}}(\mathbf{x})$ is the signed distance from point $\mathbf{x}$ to the nearest background surface — positive values mean outside the background, negative values mean inside (penetrating). For each object $M^o$ with vertices $\{\mathbf{v}_{o,1}, \ldots, \mathbf{v}_{o,N_o}\}$, a vertical translation $\tau_o$ along the gravity-opposing axis $\mathbf{e}_z$ is introduced:

$\min_{\{\tau_o\}} \sum_o \frac{1}{N_o} \sum_{i=1}^{N_o} \left[ \max\left(0, -\phi_{\text{bg}}(\mathbf{v}_{o,i} + \tau_o \mathbf{e}_z)\right) \right]^2$

where $\tau_o$ is the vertical translation (scalar) for object $o$, $\{ \mathbf{v}_{o,i} \}_{i=1}^{N_o}$ are the vertices of object $o$'s mesh, $\phi_{\text{bg}}(\cdot)$ is the SDF of the background, and $\mathbf{e}_z$ is the unit vector along the gravity-opposing (upward) axis.

What it computes: the total penetration penalty across all objects. For each vertex shifted by $\tau_o$, the SDF is evaluated. If the vertex is outside the background ($\phi_{\text{bg}} > 0$), $-\phi_{\text{bg}}$ is negative and the $\max(0, \cdot)$ operator clamps the contribution to zero — no penalty. If the vertex is inside the background (penetrating, $\phi_{\text{bg}} < 0$), $-\phi_{\text{bg}}$ is positive and the squared value is added to the objective. The optimization adjusts each $\tau_o$ (lifting objects upward) to minimize this total squared penetration. Optimization uses Adam with gradient clipping and early stopping.

Why this form: restricting translations to the vertical axis $\mathbf{e}_z$ (lifting objects up) is a strong but effective prior — objects typically settle onto supporting surfaces under gravity, so initial penetrations are most naturally resolved by raising them slightly. The squared penalty produces smooth gradients that guide optimization, unlike a linear or absolute-value penalty that can produce discontinuous gradients at the SDF zero-crossing. The alternative — allowing full 6-DOF adjustment per object — would be a much harder optimization problem with many local minima and could produce physically impossible configurations (objects floating in mid-air or rotated sideways).

The result of these three steps — property assignment, gravity alignment, collision optimization — is a physically interactable digital twin: a simulation scene where objects have plausible physical properties, gravity points in the right direction, and objects start in a collision-free configuration. This digital twin is the environment in which the robotic policy will be trained.

---

Learning Targets: Object Pose Trajectory Extraction

With the physical world model established, the next step is to define what the policy should learn to do. The generated video shows how objects should move, but in pixel space. The physical world model provides 3D structure. The bridge between them is object pose trajectories — the 3D positions and orientations of each object at each timestep, extracted from the 4D reconstruction.

Why object poses rather than alternative motion representations. The paper evaluates three possible motion representations from prior work: (1) optical flow [6] — dense per-pixel motion vectors; (2) sparse point tracks [7] — trajectories of specific surface points; (3) object poses [8] — 6-DOF rigid-body transformations. Object poses are chosen because "object pose estimation is generally more robust than other motion representations" under the visual artifacts common in generated videos: motion blur makes optical flow unreliable (flow vectors become noisy or undefined), and occlusion makes point tracks drift or lose identity. Object poses, by contrast, leverage the full appearance of the object in each frame for estimation and can be recovered even when parts of the object are occluded (the visible portion still constrains the pose).

FoundationPose for per-frame estimation. For each frame $t$ in the generated video, FoundationPose [69] takes the RGB image $I_t$, depth map $D_t$, point cloud $P_t$, and the object mesh $M^o$ (from the mesh generation stage) as input, and outputs the object's 6-DOF pose:

$x_t^o = [\mathbf{p}_t^o, \mathbf{q}_t^o]$

where $\mathbf{p}_t^o \in \mathbb{R}^3$ is the object's 3D position (its centroid or reference point in world coordinates) and $\mathbf{q}_t^o \in \mathbb{R}^4$ is its orientation as a unit quaternion. FoundationPose works by rendering the object mesh from the estimated pose and comparing the rendered appearance and depth with the observed frame, optimizing for alignment. Because it has the mesh (from image-to-3D generation), the depth map (from calibrated MegaSaM), and the RGB frame, it can exploit both geometric and photometric cues for robust tracking.

The result is a trajectory $\{x_t^o\}_{t=0}^T$ — a sequence of object poses through time — that serves as the supervision signal for policy learning. The policy's objective is to make the real robot move the real object along this same trajectory.

Robustness requirement. The paper reports usable-video ratios (Table I) based on whether FoundationPose can successfully track the object. Videos where tracking fails (due to extreme motion blur, object disappearance, or severe hallucination) are discarded. This filtering is essential: if the pose trajectory is unreliable, the policy will learn to track an incorrect target and will fail at deployment.

---

Residual Reinforcement Learning

With a physical world model and object pose targets, the system now trains a policy to control the robot. The core design challenge is balancing sample efficiency (training in simulation should be fast) with robustness (the policy must handle the imperfections of real-world execution). The paper's solution is residual reinforcement learning: a classical planner provides nominal baseline actions, and an RL policy learns additive corrections.

Why not pure classical planning. The baseline approach [8] combines a grasping model (AnyGrasp [70]) with a motion planner (CuRobo [71]) to pick and place objects. Given a target object pose trajectory, AnyGrasp proposes grasp poses, and CuRobo generates collision-free arm motions to reach those grasps and move the object. This works in many cases (67% success rate in Figure 4) but fails when: (a) the grasp proposal is slightly misaligned with the actual object position, causing a failed grasp; (b) the planned trajectory doesn't account for the dynamics of the specific object (e.g., a tomato requires different handling than a rigid block); (c) small errors in object pose estimation compound into planning failures. The planner has no mechanism to detect or recover from these errors — it's an open-loop pipeline.

Why not pure RL from scratch. Training an RL policy from scratch in the physical world model would, in principle, learn to handle all these cases. However, RL from scratch requires extensive exploration to discover successful manipulation strategies — potentially millions of environment steps. The search space is enormous: the policy must learn where to move, how to grasp, how much force to apply, and how to recover from mistakes. Figure 7 shows that RL from scratch produces lower object tracking rewards and requires substantially more training steps. Reward design is also delicate: a sparse reward (success = 1, failure = 0) provides no learning signal until the task is completed; dense rewards require careful hand-engineering per task.

The residual formulation. The key insight of residual RL is that the baseline planner provides a reasonable but imperfect action. Rather than learning from scratch, the policy only needs to learn corrections:

$a_t = a_t^{\text{base}} + \pi_\theta(o_t)$

where $a_t$ is the executed action at time $t$, $a_t^{\text{base}}$ is the baseline action from the grasp-and-plan pipeline (the same pipeline used in [8]), and $\pi_\theta(o_t)$ is the learned residual — a small additive correction predicted by a neural network with parameters $\theta$, conditioned on the current observation $o_t$. The policy outputs $[\Delta \mathbf{p}_t, \boldsymbol{\omega}_t]$ — a translational correction $\Delta \mathbf{p}_t \in \mathbb{R}^3$ and a rotational correction $\boldsymbol{\omega}_t \in \mathbb{R}^3$ in axis-angle form. The executed end-effector command is:

$\mathbf{p}_t^{\text{cmd}} = \mathbf{p}_t^{\text{base}} + \Delta \mathbf{p}_t, \quad \mathbf{q}_t^{\text{cmd}} = \exp([\boldsymbol{\omega}_t]_\times) \, \mathbf{q}_t^{\text{base}}$

where $\mathbf{p}_t^{\text{base}}$ and $\mathbf{q}_t^{\text{base}}$ are the position and orientation from the baseline planner, $\Delta \mathbf{p}_t$ is the learned translational offset, $\boldsymbol{\omega}_t$ is the learned rotational correction (an axis-angle vector where the direction is the rotation axis and the magnitude is the angle), $[\boldsymbol{\omega}_t]_\times$ is the skew-symmetric matrix, and $\exp([\boldsymbol{\omega}_t]_\times)$ converts the axis-angle representation to a rotation matrix via the exponential map. The corrected quaternion $\mathbf{q}_t^{\text{cmd}}$ is the composition of the baseline orientation and the residual rotation.

Why this form: restricting the residual to additive corrections on position and multiplicative (exponential map) corrections on orientation keeps the policy's output space small and bounded. The policy doesn't need to learn to generate entirely new trajectories — it only needs to nudge the baseline trajectory toward feasibility. This makes the learning problem dramatically easier: the exploration space is a small neighborhood around the baseline actions rather than the full action space. Figure 7 confirms this by showing residual RL converging within "a few hundred iterations" while RL from scratch struggles to match the same reward level with significantly more training.

Observation space. The policy receives a state vector $o_t$ (not pixels — this is state-based RL, not visual RL, since the simulation provides ground-truth state):

$o_t = \left[ \mathbf{x}_t^{\text{ee}}, \mathbf{x}_t^{\text{obj}}, \tau_t, \mathbf{x}_t^o, \mathbf{x}^{\text{grasp}}, d^{\text{pre}}, \mathbf{x}_t^{\text{base}} \right]$

where $\mathbf{x}_t^{\text{ee}}$ is the current end-effector pose, $\mathbf{x}_t^{\text{obj}}$ is the current object pose (from simulation), $\tau_t \in [0, 1]$ is a normalized time index indicating progress along the trajectory, $\mathbf{x}_t^o$ is the target object pose from the generated video at time $t$, $\mathbf{x}^{\text{grasp}}$ is the grasp proposal (a fixed 6-DOF pose for the grasp phase), $d^{\text{pre}}$ is a pre-grasp offset (a distance to approach before closing the gripper), and $\mathbf{x}_t^{\text{base}}$ is the baseline planned end-effector pose at time $t$.

The normalized time $\tau_t$ is important: it tells the policy where it is in the trajectory, enabling it to modulate its behavior differently during approach, grasp, transport, and release phases. The inclusion of both the current object pose $\mathbf{x}_t^{\text{obj}}$ and the target object pose $\mathbf{x}_t^o$ gives the policy the error signal it needs to compute corrections — it can see whether the object is ahead or behind the target trajectory.

Reward functions. The paper uses three simple reward components that generalize across tasks without per-task tuning:

1. Object tracking reward $r_t^{\text{trk}}$ — encourages the object to stay close to its target pose:

$r_t^{\text{trk}} = w_{\text{pos}} e^{-k_{\text{pos}} \|\mathbf{p}_t^{\text{obj}} - \mathbf{p}_t^o\|_2} + w_{\text{ori}} e^{-k_{\text{ori}} \|\mathbf{q}_t^{\text{obj}} - \mathbf{q}_t^o\|_2}$

where $\mathbf{p}_t^{\text{obj}}$ is the current object position (from simulation), $\mathbf{p}_t^o$ is the target object position (from the video), $\mathbf{q}_t^{\text{obj}}$ and $\mathbf{q}_t^o$ are the corresponding quaternions, $\|\cdot\|_2$ is the Euclidean distance for positions and the quaternion geodesic distance for orientations. The exponential form $e^{-k \cdot d}$ maps distances to $(0, 1]$: when the distance is zero, the reward is 1 (scaled by $w_{\text{pos}}$ or $w_{\text{ori}}$); when the distance is large, the reward decays to zero. The parameters $k_{\text{pos}}$ and $k_{\text{ori}}$ control the decay rate (how quickly the reward drops as error increases), and $w_{\text{pos}}$, $w_{\text{ori}}$ weight the position and orientation terms relative to each other.

2. Grasp reward $r_t^{\text{grasp}}$ — penalizes losing contact with the object during grasp and transport:

$r_t^{\text{grasp}} = -w_{\text{grasp}} \, \mathbb{1}\left[ \|\mathbf{p}_t^{\text{ee}} - \mathbf{p}_t^{\text{obj}}\|_2 > \tau \right]$

where $\mathbf{p}_t^{\text{ee}}$ is the end-effector position, $\tau$ is a distance threshold, and $\mathbb{1}[\cdot]$ is the indicator function (1 if the condition is true, 0 otherwise). When the distance between the end-effector and object exceeds $\tau$ (the robot has dropped or lost the object), the policy receives a penalty of $-w_{\text{grasp}}$. This term is active only during grasp and transport phases (when the robot should be holding the object).

3. Planning reward $r_t^{\text{plan}}$ — penalizes kinematically or dynamically infeasible actions. When the inverse kinematics solver fails to find a joint configuration for the commanded end-effector pose, or when the motion planner detects a collision, the policy receives a negative reward. This term implicitly teaches the policy to avoid commanding poses that the robot cannot physically reach or that would cause collisions.

The total reward at each timestep is the sum of these terms. The weights $\{w_{\text{pos}}, w_{\text{ori}}, w_{\text{grasp}}, k_{\text{pos}}, k_{\text{ori}}\}$ and the threshold $\tau$ are presumably tuned once and kept fixed across tasks (the paper emphasizes that the rewards are designed to "generalize to diverse tasks").

PPO training. The policy $\pi_\theta$ is trained using Proximal Policy Optimization (PPO) [72] within the physical world model. PPO is chosen because it is a stable, widely-used on-policy RL algorithm that works well with continuous action spaces and state-based observations. Key implementation details (from standard PPO practice, not explicitly stated in the paper): the policy network is likely a small MLP (multi-layer perceptron) with a few hundred parameters, since the observation and action spaces are low-dimensional; training alternates between collecting rollout data in the simulation and updating the policy via clipped surrogate objectives; the value function is trained concurrently to estimate expected returns.

Why PPO rather than off-policy algorithms (SAC, TD3). On-policy algorithms like PPO are often preferred for residual RL because the baseline actions provide a narrow, well-behaved distribution. Off-policy algorithms that learn from a replay buffer of past experiences might struggle because the residual correction distribution changes as the policy improves — old data becomes stale quickly. PPO's on-policy updates ensure the policy is always trained on data generated by its current parameters, which is important when the residual corrections are small and the policy is making rapid progress.

Deployment to real world. After training converges in simulation, the policy $\pi_\theta$ is deployed zero-shot on the real robot. At each timestep, the robot's sensors provide the observation $o_t$ (using real object pose estimation from the RGB-D camera in place of simulation ground-truth), the baseline planner computes $\mathbf{x}_t^{\text{base}}$, the policy predicts $[\Delta \mathbf{p}_t, \boldsymbol{\omega}_t]$, and the combined command $\mathbf{x}_t^{\text{cmd}}$ is sent to the robot's low-level controller. No fine-tuning or adaptation on real hardware is performed — the entire learning happens in simulation, built from the single generated video and the first real RGB-D frame.

---

Summary of Design Choices and Their Justifications

• Veo3 as primary video generator: achieves 70% usable-video ratio (Table I), substantially higher than alternatives. Higher-quality generation cascades into more reliable pose estimation, better mesh generation, and ultimately higher policy success rates. The choice is empirical rather than principled — any sufficiently high-quality image-to-video model would work.

• MegaSaM + affine calibration for depth: leverages temporal consistency from MegaSaM's video-based depth estimation while correcting its scale ambiguity using the one frame where real depth is available. More robust than per-frame depth estimation with scale calibration per frame (which would introduce temporal jitter).

• Image-to-3D for object meshes rather than multi-view reconstruction: generated videos are typically monocular; multi-view reconstruction requires viewpoint variation that generated videos don't provide. The generative prior fills in unseen geometry at the cost of potential reconstruction error (7% in Figure 6).

• Object-background decomposition with inpainting: enables independent handling of dynamic objects (need complete geometry) and static backgrounds (need plausible occluded-region geometry). Monolithic reconstruction would fail on one or the other.

• VLM-based physical property estimation: provides zero-shot generalization to arbitrary objects without physics estimation from video or manual specification. Leverages the VLM's Internet-scale commonsense knowledge rather than requiring physical interaction data.

• Object-centric rather than embodiment-centric learning: generated videos often hallucinate hands or have inconsistent hand kinematics (embodiment-centric fails: 30%, 10% in Table II). Object motion is more stable, easier to estimate, and more robust.

• Residual RL rather than pure planning or pure RL: pure planning (67% in Figure 4) fails on grasping and tracking errors with no recovery mechanism. Pure RL from scratch (Figure 7) converges slowly and requires more careful reward design. Residual RL (82% in Figure 4) combines the efficiency of classical planning with the robustness of learned corrections, converging in a few hundred iterations.

• Simple, task-agnostic rewards (tracking + grasp + planning): enables the same reward structure to work across diverse tasks (pouring, placing, wiping, sweeping) without per-task tuning. The exponential tracking reward naturally bounds the reward signal and provides smooth gradients.

4. Key Insights and Innovations

Innovation 1: Physical World Modeling as the Missing Bridge Between Video Generation and Robot Action

The paper's most fundamental conceptual move is diagnosing exactly why prior zero-shot video-to-action approaches fail: not because video generation quality is insufficient, not because motion representations (flow, tracks, poses) are inadequate, but because there is no physical feedback loop during policy execution. Prior work treated the generated video as a script to be followed — extract motion cues, retarget them to the robot, execute open-loop. PhysWorld reframes the problem: the generated video is not a script but a source of approximate guidance that must be grounded through interaction with a physical model.

This is a diagnostic innovation, not just a method improvement. The paper doesn't simply add physics to an existing pipeline and get better numbers. It provides a causal explanation for the failure modes that prior work observed but couldn't articulate. When [8] gets 18% grasping failures (Figure 6), those failures aren't random — they happen because the grasping model, operating without physical feedback, has no way to compensate for small misalignments between the estimated object pose and reality. When [7] and [6] struggle with tracking, it's because optical flow and point tracks carry no information about whether a trajectory is physically feasible. PhysWorld's contribution is making this causality explicit: the absence of physics is not a minor inconvenience but the primary failure mechanism.

Prior real-to-sim-to-real work understood the importance of physics but required multi-view reconstruction (Polycam, BundleSDF [65]) that is incompatible with monocular generated videos. PhysWorld's innovation is recognizing that the world model doesn't need to be perfect to provide useful feedback. The paper quantifies this: 7% reconstruction errors (Figure 6) exist, but the net effect is a 15-percentage-point improvement over no world model at all. This is a practical insight with theoretical implications: it suggests that for manipulation policies, physical plausibility matters more than geometric fidelity — a slightly wrong mesh with correct dynamics beats a perfectly accurate mesh with no physics.

The evidence supporting this innovation is the side-by-side comparison in Figure 4 and the failure mode breakdown in Figure 6. PhysWorld's 82% vs. [8]'s 67% is not just "we made it better" — the specific reduction in grasping failures (18% → 3%) and elimination of tracking failures (5% → 0%) directly corroborates the claim that physical feedback addresses the failure modes that purely visual pipelines cannot.

---

Innovation 2: Generative Priors Enable Monocular Physical Scene Reconstruction Without Multi-View Capture

The second conceptual contribution is solving a chicken-and-egg problem that blocked prior real-to-sim methods from using generated videos. Real-to-sim-to-real pipelines need complete 3D geometry to build simulation environments, but they obtain it via multi-view reconstruction — which assumes the camera moves around the scene. Generated manipulation videos typically have a static camera looking at a tabletop. This means large portions of objects (backs, bottoms) and backgrounds (surfaces under objects) are never observed.

PhysWorld's solution — using image-to-3D generative models to hallucinate unseen geometry — is a fundamental reframing of what kind of 3D models are acceptable for simulation. The dominant assumption in real-to-sim work is that fidelity is paramount: the simulation should match reality as closely as possible, because any discrepancy creates a sim-to-real gap. PhysWorld argues the opposite: for the purpose of providing physical feedback during policy learning, completeness matters more than exactness. A hallucinated back-of-the-pan that has approximately correct mass distribution and collision geometry enables the policy to learn grasp strategies and motion corrections that transfer to reality. A perfectly accurate but incomplete mesh (with holes where the bottom should be) causes objects to fall through the table — providing feedback that is actively harmful.

This is an incremental but consequential shift in the real-to-sim paradigm. It doesn't replace multi-view reconstruction when multi-view data is available. Rather, it opens a new regime — monocular generated videos — that was previously considered incompatible with simulation-based policy learning. The specific technical realization (ObjectClear inpainting for background, TRELLIS-style image-to-3D for objects) is an engineering contribution, but the conceptual move is the recognition that generative priors are the right tool for this incompleteness problem.

Evidence: the paper provides this primarily through the qualitative results in Figure 3 and the system-level success rate in Figure 4. The 82% overall success rate with a pipeline that includes hallucinated geometry demonstrates transfer despite reconstruction error. The 7% reconstruction errors in Figure 6 are the cost of this approach, but the net gain over no reconstruction (67% → 82%) shows the tradeoff is favorable.

---

Innovation 3: Object-Centric Learning as a Robustness Strategy Against Video Generation Artifacts

The paper's third conceptual contribution is diagnosing that embodiment tracking is the wrong target for video-to-action transfer when videos are generated rather than recorded. Prior work on human-to-robot imitation focused on tracking human hand or body motions and mapping them to robot kinematics. This makes sense for real videos, where hand kinematics are physically consistent and can be reliably estimated. Generated videos, however, frequently hallucinate hands with impossible joint configurations, inconsistent finger counts, or morphing geometry between frames.

PhysWorld's response — track what the object does, not what the hand does — seems obvious in retrospect but represents a genuine shift in framing. The paper argues that object motion is the invariant signal across embodiments. A human hand, a robot gripper, and a generated hallucinated hand all move the object along roughly the same trajectory to complete the task. By learning to reproduce the object's trajectory rather than the embodiment's motion, the policy becomes robust to the specific failure modes of video generation.

This is more than a practical trick. It implies that for task specification, object motion is a more fundamental representation than embodiment motion. The task "pour the tomato from the pan onto the plate" is defined by where the tomato goes, not by the specific joint angles of the arm that transports it. Prior video-to-action methods implicitly treated embodiment motion as the task representation; PhysWorld explicitly separates task specification (object trajectory) from task execution (robot commands), with the physical world model providing the translation layer between them.

Evidence: Table II shows the dramatic gap — 90% vs. 30% and 80% vs. 10% for object-centric vs. embodiment-centric on two tasks. These are not small differences; they represent a qualitative regime change from "mostly fails" to "mostly succeeds." The paper attributes this to hallucinated hands and inconsistent kinematics in generated videos, but the deeper implication is that object-centric learning would likely be more robust even with perfect hand estimation, because object dynamics are simpler and more transferable across embodiments.

---

Innovation 4: Residual RL as an Efficient Coupling Mechanism Between Classical Planning and Learned Adaptation

The final conceptual innovation is the observation that residual RL is the natural formulation when a classical planner provides approximately correct actions and a physical simulator provides corrective feedback. This is not a new idea in isolation — residual RL has been used before in other contexts — but its application as the coupling mechanism between video-derived target trajectories and physics-based policy learning is distinctive.

The insight is about how to structure the learning problem to match the structure of the available information. PhysWorld has access to three things: (1) a baseline planner that is broadly competent but brittle (it can generate reasonable trajectories but fails on small errors with no recovery); (2) a physical simulator that provides accurate dynamics feedback; (3) target object trajectories from the generated video that are approximately correct but not physically guaranteed. Residual RL is the formulation that lets each piece do what it's good at. The planner constrains exploration to a small neighborhood of feasible actions, dramatically reducing the search space. The simulator provides dense reward signals (tracking, grasping, planning penalties) that guide the residual corrections. The target trajectories provide the objective but the policy isn't forced to follow them slavishly — it learns when to deviate to maintain physical feasibility.

The paper contrasts this with two alternatives, each of which uses only part of the available information: pure planning uses the trajectory but ignores physics (67%); pure RL from scratch uses physics but requires learning from scratch instead of leveraging the baseline (slower convergence, lower reward in Figure 7). Residual RL sits at the Pareto-optimal point: it gets the sample efficiency of planning and the robustness of RL by exploiting the structure of the problem.

Evidence: Figure 7 shows residual RL converging in a few hundred iterations while RL from scratch lags behind. Figure 4 shows the end result (82% success) substantially exceeding pure planning (67%). The convergence speed is the key practical contribution — it makes the approach feasible for real deployment, where training must happen quickly for each new task.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on 10 diverse real-world manipulation tasks, each with 10 rollouts (100 total evaluations per method). The tasks are: (1) Wipe the whiteboard; (2) Water the flowers; (3) Put the book in the bookshelf; (4) Pour the fish from the pan onto the plate; (5) Put the lid on the pot; (6) Put the spoon in the pan; (7) Put the shoe in the shoebox; (8) Pour the candies from the spoon onto the plate; (9) Sweep the paper scraps into the dustpan; (10) Pour the tomato from the pan onto the plate. These tasks cover a range of manipulation primitives (pouring, placing, wiping, sweeping, insertion) with varied objects and geometries. The paper does not use a pre-existing benchmark dataset — all tasks are designed and executed by the authors in a real-world setting with physical robots.

• Base model(s). PhysWorld is not a model in the conventional sense but a pipeline of components: Veo3 [63] for video generation, MegaSaM [64] for depth estimation, ObjectClear [66] for inpainting, a TRELLIS-style image-to-3D generator [67] for object mesh generation, FoundationPose [69] for object pose estimation, AnyGrasp [70] for grasp proposals, and CuRobo [71] for motion planning. The RL policy $\pi_\theta$ is a learned neural network (likely a small MLP, though the paper does not specify the architecture or parameter count) trained with PPO [72]. The physical property estimation uses an unspecified vision-language model (VLM). The choice of these components reflects a deliberate strategy of assembling state-of-the-art off-the-shelf models rather than training custom models — the paper's contribution is the integration architecture and the physical world modeling bridge, not new model architectures.

• Metrics. The primary metric is task success rate (%), defined as the fraction of 10 rollouts per task where the robot completes the specified task. Success is a binary per-trial judgment (task completed or not), and the paper reports per-task success rates and an average across all 10 tasks (Figure 4). For the video generation ablation (Table I), the metric is usable video ratio (%) — the fraction of 10 generated videos per model-task pair from which object poses can be robustly recovered using FoundationPose. For the RL convergence comparison (Figure 7), the metric is object tracking reward, the accumulated reward signal defined in Equation 8, plotted over training steps. For the failure mode analysis (Figure 6), failures are categorized into four types (grasping, tracking, dynamics, reconstruction) and reported as percentages of total rollouts.

• Baselines. The paper compares against three zero-shot methods that do not use physical world modeling: (i) RIGVid [8] — directly tracks object poses from generated videos and uses AnyGrasp [70] for grasping and CuRobo [71] for motion planning, representing the state-of-the-art in direct visual cue retargeting; (ii) Gen2Act [7] — extracts sparse point tracks from generated videos as tracking objectives (the paper uses a modified version from [8]); (iii) AVDC [6] — leverages depth and optical flow estimation to represent object and embodiment motions. All three are prior published methods that, like PhysWorld, operate without real-world robot data collection. The paper also includes an internal comparison against embodiment-centric learning (Table II), where a human hand mesh is reconstructed and finger keypoints are mapped to the robot end-effector, and against RL from scratch (Figure 7), where a policy is trained in the same physical world model but without the residual formulation (no baseline grasp-and-plan actions).

• Generation budget / compute accounting. The paper does not report FLOPs or wall-clock training time. The only generation budget discussed is the number of videos generated per model-task pair for the video quality ablation: 10 videos per combination (Table I), with the usable ratio computed from these. For policy training, the compute budget is measured in PPO training steps (Figure 7, x-axis labeled "Step"), with residual RL described as converging "within a few hundred iterations." The paper does not specify the number of environment steps per PPO iteration, the total number of rollouts collected during training, or the hardware used for simulation or training. For real-world evaluation, each method receives 10 rollouts per task, and the robot executes the full policy for each rollout with no intermediate resets unless the task is completed or an unrecoverable failure occurs.

• Cross-validation / statistical protocol. The paper does not report any cross-validation, statistical significance testing, or confidence intervals. Results are reported as point estimates (success rate percentages from 10 rollouts per task). With only 10 rollouts per task, the standard error on a per-task success rate is approximately ±15 percentage points (assuming binomial distribution at 50% success rate), meaning differences between methods on individual tasks should be interpreted cautiously. The aggregated average across 10 tasks (100 total rollouts) provides a more reliable comparison, with a standard error of approximately ±5 percentage points at 82% success rate, but this is not quantified in the paper. The ablation comparing video generation models (Table I) uses 10 videos per combination, and the usable ratio is a simple fraction — no error bars or significance tests are reported.

Main Quantitative Results

Overall System Performance: PhysWorld vs. Zero-Shot Baselines (Figure 4)

The headline result is that PhysWorld achieves an average success rate of 82% across 10 real-world manipulation tasks, substantially outperforming the strongest zero-shot baseline, RIGVid [8], at 67% — a 15-percentage-point improvement. This is the central empirical claim of the paper: adding physical world modeling to the video-to-action pipeline yields a large and consistent improvement in manipulation reliability.

Breaking down per-task results from Figure 4:

(Exact per-task numbers are estimated from the bar chart in Figure 4; the paper provides exact values only for the average and for "Put the book in the bookshelf" and "Put the shoe in the shoebox" in Table II.)

Several patterns emerge from these per-task results:

• PhysWorld improves over RIGVid on every task, with gains ranging from roughly 10% (Pour the tomato, Pour the fish) to approximately 30% (Put the book, Put the shoe). There are no tasks where the baseline outperforms PhysWorld — the improvement is consistent across all manipulation categories.

• The largest gains appear on tasks requiring precise insertion or alignment: "Put the book in the bookshelf" (90% vs. 60%) and "Put the shoe in the shoebox" (80% vs. 60%). These are tasks where small grasping or positioning errors are catastrophic — if the book is slightly misaligned, it won't slide into the shelf. The physical world model enables the residual policy to learn corrective adjustments that compensate for these small errors.

• Tasks involving pouring (Pour the fish, Pour the tomato, Pour the candies) show smaller but still meaningful improvements (roughly 10–20%). Pouring tasks are more challenging for the entire pipeline because they involve dynamic object motion (liquid-like or granular behavior of multiple objects) that the rigid-body physics simulator may not accurately capture.

• The baseline methods Gen2Act [7] (sparse point tracks) and AVDC [6] (optical flow) perform substantially worse than RIGVid [8] (object poses), averaging approximately 46% and 44% respectively. This confirms the paper's design choice that object pose tracking is more robust than alternative motion representations for generated videos, consistent with the motivation discussed in Section III-B.

• AVDC and Gen2Act show particularly poor performance on tasks like "Put the book in the bookshelf" (~30% each), likely because point tracks and optical flow suffer from tracking drift during the fine manipulation required for insertion — the precise alignment needed is not captured well by sparse or dense 2D motion signals.

The improvement from PhysWorld over RIGVid is attributed to the physical world model providing corrective feedback that the purely visual pipeline lacks. Figure 6 provides the mechanistic breakdown: the physical world model reduces grasping failures from 18% to 3% and eliminates tracking failures from 5% to 0%, while introducing a new failure category — reconstruction errors at 7%. The net effect is a substantial reduction in total failures (28% → 18%), consistent with the 15-percentage-point improvement in average success rate.

Failure Mode Analysis: Where the Gains Come From (Figure 6)

Comparing PhysWorld against RIGVid [8], the failure mode breakdown in Figure 6 reveals:

• Grasping failures: 18% (RIGVid) → 3% (PhysWorld). RIGVid's grasping failures occur because AnyGrasp [70] proposes grasp poses based solely on the estimated object pose from the video — if the pose estimate is slightly off (due to video artifacts, depth errors, or calibration drift), the grasp fails in open-loop execution. PhysWorld's residual policy, trained with physical feedback, learns to adjust the end-effector position to compensate for these misalignments. The policy can effectively "feel" the object in simulation and correct its approach.

• Tracking failures: 5% (RIGVid) → 0% (PhysWorld). Tracking failures occur when the object pose trajectory extracted from the video contains errors (e.g., jitter, drift during occlusion) that cause the motion planner to generate infeasible or unsuccessful trajectories. PhysWorld eliminates these entirely because the policy is not forced to follow the trajectory exactly — it receives the target object pose as part of its observation (Equation 6) but learns to track it while maintaining physical feasibility, effectively smoothing or correcting erroneous trajectory segments.

• Dynamics failures: 3% (RIGVid) → 2% (PhysWorld). This category shows a small, possibly non-significant difference given the limited number of rollouts. Dynamics failures are those where the planned motion is kinematically or dynamically infeasible (e.g., requiring velocities or forces the robot cannot produce). The residual policy provides only a small improvement here, suggesting that the baseline planner (CuRobo [71]) is already relatively good at generating feasible motions, and the primary failures are upstream in grasping and tracking.

• Reconstruction errors: 0% (RIGVid) → 7% (PhysWorld). This is a failure mode introduced by PhysWorld that does not exist in RIGVid (which has no reconstruction). These errors occur when the hallucinated geometry from the generative mesh pipeline does not match the real scene — for example, if the image-to-3D generator produces an object mesh with a different size, shape, or mass distribution than the real object, the policy learned in simulation may transfer poorly. However, the 7% reconstruction error rate is substantially lower than the 23% combined grasping and tracking error rate eliminated by the physical world model, so the net effect is strongly positive.

The failure mode analysis provides direct causal evidence for the paper's central claim: physical feedback during learning addresses the specific failure modes (grasping, tracking) that limit purely visual pipelines, and the cost (reconstruction error) is manageable.

Object-Centric vs. Embodiment-Centric Learning (Table II)

Table II reports a head-to-head comparison on two tasks, comparing policies trained to track object motion (object-centric, the PhysWorld default) against policies trained to track human hand motion (embodiment-centric, following the approach of prior human-to-robot imitation work):

The gap is dramatic — object-centric learning outperforms embodiment-centric by 60 and 70 percentage points respectively. The paper attributes this to two factors: (1) generated videos frequently hallucinate hands with inconsistent kinematics, making hand pose estimation unreliable; (2) even when hand pose is estimated correctly, mapping human hand kinematics to a robot gripper introduces additional errors because the kinematic structures differ (human fingers vs. parallel-jaw gripper). Object motion, by contrast, is both easier to estimate (FoundationPose [69] with the generated mesh) and more directly transferable (the robot needs to move the object, not mimic the hand).

This result validates a key design choice and provides evidence for the broader claim that object motion is a more robust cross-embodiment representation than embodiment motion when the source of demonstrations is video generation rather than real human video. The finding also helps explain why prior human-to-robot imitation methods that work well with real videos may fail when applied to generated videos — the hand hallucination problem is specific to generation and has not been a concern in the real-video imitation literature.

Residual RL vs. RL from Scratch (Figure 7)

Figure 7 compares the convergence of residual RL against training a policy from scratch on the "Pour the tomato from the pan onto the plate" task, measuring object tracking reward (Equation 8) over PPO training steps.

• Residual RL converges within a few hundred iterations (the x-axis shows approximately 0–500 steps) and reaches a high, stable reward level (the paper does not provide exact reward values, but the curve is consistently above the RL-from-scratch curve).

• RL from scratch learns more slowly and achieves lower asymptotic reward. At step 500, RL from scratch has not caught up to residual RL's performance at step 100.

The paper attributes residual RL's faster convergence to the baseline actions constraining exploration: the policy only needs to learn small corrections in a neighborhood around approximately correct actions, rather than exploring the full action space to discover manipulation strategies from scratch. The baseline effectively provides a strong behavioral prior that dramatically reduces the sample complexity of RL.

This result is important for the practical feasibility of the approach: if training required millions of environment steps per task, the framework would be computationally expensive to deploy for new tasks. Convergence in a few hundred iterations means training is fast enough to be practical — the paper does not provide wall-clock times, but standard PPO with state-based observations in MuJoCo-style simulators typically completes a few hundred iterations in minutes on a single GPU.

Video Generation Quality Analysis (Table I)

Table I evaluates four image-to-video models on their usable-video ratio across the task set:

The usable ratio measures the fraction of 10 generated videos per task-model pair from which FoundationPose can robustly recover object poses. This is a downstream metric, not a generic video quality metric — it measures fitness for the PhysWorld pipeline specifically.

Two patterns are notable:

• Proprietary models substantially outperform open-source models. Veo3's 70% is 1.9× higher than Tesseract (the next best, at 36%) and 17.5× higher than CogVideoX1.5-5B (4%). This gap is large enough that the PhysWorld pipeline is effectively unusable with current open-source video generators — at 2–4% usable ratio, generating 25–50 videos to find one usable one would be impractical for deployment.

• Robotic data fine-tuning helps but is not sufficient. Tesseract [11] is fine-tuned on robotic data and achieves 36% — better than generic open-source models (4%, 2%) but still far below Veo3 (70%). This suggests that generic video generation capability (Veo3's large-scale pretraining) currently matters more for the PhysWorld pipeline than domain-specific fine-tuning, though the paper notes that Tesseract "tends to outperform generic generators," indicating the direction is promising.

This analysis reveals a brittleness in the PhysWorld pipeline: its feasibility is currently tied to the availability of high-quality proprietary video generation. The paper does not investigate what specific video qualities (temporal consistency, object permanence, minimal hallucination, camera stability) drive the usable-video ratio, which would be valuable for guiding future improvements in video generation for robotics.

Ablation Studies and Robustness Checks

Physical world modeling vs. no physics (Figure 4): The primary ablation is the comparison of PhysWorld against RIGVid [8], which uses the identical object-pose-tracking pipeline but without physical world modeling (directly retargeting poses with grasping and planning, no simulation-based RL). The 82% vs. 67% average success rate, with consistent improvement across all 10 tasks, provides the core evidence that physical feedback is necessary. The failure mode analysis (Figure 6) further decomposes this into specific mechanisms (grasping improvement, tracking elimination). This is not a controlled component ablation (the systems differ in multiple ways — physics, RL training, residual formulation) but rather a system-level comparison that establishes the overall benefit of the physical world modeling approach.

Object-centric vs. embodiment-centric tracking (Table II): This ablates the choice of tracking target, keeping the rest of the PhysWorld pipeline fixed. The large gap (90% vs. 30% and 80% vs. 10%) validates that object motion is the correct learning target when demonstrations come from generated videos. The paper does not report an object-centric vs. embodiment-centric ablation with RIGVid [8] or other baselines, so we cannot determine whether the benefit of object-centric learning is specific to PhysWorld (where the physical world model may better support object tracking) or general to all video-to-action methods.

Residual RL vs. RL from scratch (Figure 7): This isolates the contribution of the residual formulation within the physical world model, keeping the simulator, rewards, and observation space identical. Residual RL converges faster and achieves higher asymptotic reward, demonstrating that the baseline actions provide a useful behavioral prior. The paper does not compare residual RL against pure grasp-and-plan without any learned correction (i.e., the RIGVid approach but inside the physical world model rather than deployed directly), which would clarify whether the RL component is adding value beyond just the physics feedback during planning.

Video generation model comparison (Table I): This ablates the choice of video generator, measuring how generation quality cascades into downstream pipeline performance. The 70% → 2% range shows that video generation is a critical bottleneck — the rest of the pipeline cannot compensate for poor generation. However, the paper does not report success rates for PhysWorld when using different video generators (only the intermediate usable-video metric), so we cannot quantify how the usable-video ratio translates into final task success. A video that is marginally "usable" (poses recovered but with noise) might still lead to lower policy success than a video from a higher-quality generator, even if both pass the usability threshold.

Notable absences in the ablation suite. Several ablations that would strengthen the paper are not reported: (i) No ablation of the physical property estimation — what happens if default (e.g., uniform mass and friction) properties are used instead of VLM-estimated properties? The VLM query is an interesting component, but its contribution to overall performance is unmeasured. (ii) No ablation of the collision optimization step — how often do initial penetrations cause simulation failures without optimization, and does the residual policy learn to handle them anyway? (iii) No ablation of the metric depth calibration (MegaSaM → affine alignment) — what if uncalibrated MegaSaM depths are used directly? (iv) No ablation of the image-to-3D generator quality — would a simpler shape-completion method (e.g., extruding the visible geometry, fitting primitive shapes) work nearly as well as a learned generative model? (v) No sensitivity analysis on the number of environment steps, PPO hyperparameters, or reward weights — the paper reports one training run per task with fixed settings.

Critical Assessment

Claim 1: PhysWorld substantially improves manipulation accuracy compared to previous approaches (82% vs. 67% average). This claim is supported by the data in Figure 4, with consistent improvement across all 10 tasks. However, the "substantial" characterization requires careful interpretation. With only 10 rollouts per task, the per-task differences are noisy — a 10-percentage-point difference on a single task (e.g., 70% vs. 60%) could arise from one additional successful rollout and is not statistically distinguishable from zero improvement at conventional significance levels with this sample size. The aggregate result (82% vs. 67% over 100 rollouts) is more reliable, with approximately ±5 percentage point standard error. The improvement is real and practically meaningful, but the paper overstates per-task precision by reporting exact percentages without uncertainty quantification.

Additionally, the comparison against baselines is not fully controlled. RIGVid [8] uses AnyGrasp + CuRobo directly on the real robot; PhysWorld uses the same components inside simulation with residual RL on top. The improvement could be attributed to (a) the physical feedback during learning, (b) the residual RL formulation itself (learning corrections from simulation interaction, even with imperfect physics), or (c) simply having a learned policy rather than an open-loop pipeline. The paper does not disentangle these. The comparison against Gen2Act [7] and AVDC [6] is weaker than it appears because these methods use fundamentally different motion representations (point tracks, optical flow) that the paper already argues are inferior to object poses — their lower success rates may reflect representation choice rather than the absence of physical world modeling. A fairer baseline would be RIGVid augmented with a non-physics simulator (e.g., kinematic replay of the trajectory in a collision-free environment) to isolate the contribution of physical dynamics specifically.

Claim 2: Physical world modeling eliminates the need for real-world data collection and enables zero-shot manipulation. This claim is supported in the specific sense that PhysWorld uses no real robot demonstrations for training. However, the claim requires careful boundary-drawing. PhysWorld uses a real RGB-D image as input — this is sensor data, not demonstration data, but it is real-world data. The physical property estimation queries a VLM trained on Internet data that includes descriptions of physical object properties — this is prior knowledge derived from human experience with the physical world. The video generator (Veo3) is trained on Internet videos that include real-world physics — the generated videos inherit implicit physical knowledge from their training data. PhysWorld does not use task-specific robot demonstrations, but it extensively leverages real-world data in its component models. The "zero-shot" characterization is accurate with respect to robot data collection for the specific tasks evaluated, but the strong claim that the framework "eliminates the need for real-world data collection" is misleading without the qualifier that it substitutes robot data collection with Internet-scale data collection embedded in pretrained models.

Claim 3: Object-centric learning is more robust than embodiment-centric learning for generated videos. Strongly supported by Table II, but limited to two tasks. The paper does not explain why these two tasks were chosen or whether the same pattern holds across the other eight tasks. The attribution to hand hallucination and kinematic inconsistency is plausible but not directly evidenced — the paper does not show examples of hallucinated hands or quantify how often hand pose estimation fails in the generated videos. A more complete evaluation would report hand-pose estimation success rates (analogous to the object-pose usable-video ratio) to confirm the mechanism.

Claim 4: Residual RL accelerates convergence compared to RL from scratch. Supported by Figure 7, but limited to a single task ("Pour the tomato from the pan onto the plate"). The convergence advantage may vary across tasks — tasks where the baseline planner is already near-optimal would show a large residual RL advantage (small corrections needed), while tasks where the planner is very poor might show a smaller or no advantage. The paper does not report this ablation on other tasks. Additionally, the Figure 7 comparison does not control for total environment interactions: the residual RL policy may converge faster in terms of PPO steps, but if the baseline action computation adds significant overhead, the wall-clock time advantage might be smaller.

Missing experiments that would strengthen the paper:

• Multi-task training and generalization. The paper evaluates 10 tasks independently, training a separate policy for each. Does the physical world model enable generalization across tasks, or does each task require a separate reconstruction and training process? An experiment training one policy on multiple tasks (or few-shot adaptation to new tasks) would test whether the physical world model captures transferable manipulation skills or is merely a per-task simulation environment.

• Sensitivity to reconstruction quality. The paper reports 7% reconstruction errors but does not systematically vary reconstruction fidelity to measure how it affects policy success. Does success degrade gracefully as reconstruction error increases, or is there a sharp threshold below which policies fail to transfer? This is critical for understanding whether PhysWorld's approach generalizes to scenes where image-to-3D generation is less reliable (unusual objects, cluttered scenes, transparent or reflective objects).

• Physics fidelity ablation. The paper uses a rigid-body simulator with VLM-estimated physical properties. How important are the specific property values? If all objects are assigned a default mass and friction, does performance degrade significantly? This would test whether the benefit of the physical world model comes from having any physics (forces, contacts, collisions) or from having accurate physics (correct masses, correct friction).

• Comparison against a stronger RIGVid baseline. RIGVid [8] uses AnyGrasp + CuRobo open-loop. Would RIGVid augmented with simple recovery behaviors (re-grasp on failure, re-plan on collision) close the gap with PhysWorld? The paper's framing implies that physics is necessary, but an alternative hypothesis is that any feedback mechanism — even non-physical, rule-based recovery — would provide similar benefits. Testing this would strengthen the claim that physical feedback specifically (rather than feedback in general) is the critical ingredient.

• Statistical rigor. With 10 rollouts per task, reporting confidence intervals or conducting hypothesis tests would substantially strengthen the quantitative claims. The paper's point estimates (e.g., 82% vs. 67%) are suggestive but not statistically conclusive on their own.

Where the claims hold and where they are conditional:

The central claim — that coupling video generation with physical world modeling improves manipulation accuracy over direct visual cue retargeting — holds across the 10 evaluated tasks (Figure 4). However, it is conditional on:

• Video generation quality: the framework requires a sufficiently high-quality video generator (Veo3 at 70% usable ratio); with open-source generators at 2–4% usable ratio, the pipeline is currently infeasible. The claim does not hold for arbitrary video generation models — it assumes access to a state-of-the-art proprietary generator.
• Object pose estimability: the tasks involve rigid or semi-rigid objects for which FoundationPose [69] can recover poses. Deformable objects (cloth, rope, dough) or highly specular/transparent objects (glass, metal with reflections) would likely break the pose estimation step and thus the entire pipeline. The paper does not evaluate such objects, and the claim should not be extended to them without evidence.
• Single-object manipulation: the tasks involve moving one primary object at a time. Multi-object manipulation (e.g., setting a table with multiple items, assembling multiple parts) is not tested, and the object-centric tracking formulation (which focuses on one object trajectory) may not extend naturally.
• Tabletop manipulation in structured scenes: all tasks are tabletop manipulation with a clear supporting surface and visible objects. Cluttered scenes, occluded objects, or non-tabletop settings (fridge, drawer, cabinet) are not evaluated. The gravity alignment and collision optimization steps assume a dominant ground plane, which may not exist in more complex environments.

6. Limitations and Trade-offs

Severe Dependence on Proprietary Video Generation Quality

The assumption or constraint. PhysWorld's entire pipeline is gated by the quality of the first step: video generation. The system requires generated videos from which object poses can be robustly recovered, and the paper's own analysis (Table I) shows that only Veo3, a proprietary model, achieves a usable-video ratio (70%) that makes the pipeline practically deployable. Open-source models fall dramatically short: Tesseract achieves 36%, CogVideoX1.5-5B only 4%, and Cosmos-2B a mere 2%. The paper is transparent about this dependence, stating that "higher-quality, task-consistent video generation is necessary for reliable manipulation" (Section IV-A). However, it does not analyze what specific video properties drive usability — temporal consistency, object permanence, minimal hallucination, camera stability — making it unclear what capabilities future open-source models would need to develop to close this gap.

The consequence. At 2–4% usable-video ratios, a practitioner would need to generate 25–50 videos per task just to find one usable instance, making the pipeline economically and computationally infeasible with open-source tools. This creates a hard dependency on continued access to proprietary models (Veo3 or equivalent), which may change in capability, pricing, or availability. Furthermore, even with Veo3's 70% ratio, 30% of generated videos are still unusable — meaning the system must have a mechanism to detect and discard these (the paper implies FoundationPose tracking failure serves this role) and regenerate, adding cost and latency that is not accounted for in the reported success rates. The paper does not measure how this regeneration overhead affects end-to-end deployment cost or wall-clock time.

What evidence exists in the paper. Table I provides the usable-video ratios across four models, and the qualitative evaluation (Figure 5) shows selected successful Veo3 generations. However, the paper does NOT report (a) the end-to-end success rate of PhysWorld when using non-Veo3 generators, (b) how many regeneration attempts are typically needed per task even with Veo3, (c) what characterizes unusable videos versus usable ones (beyond "object poses can be recovered robustly"), or (d) how the pipeline's performance degrades as a function of video quality — whether there is a sharp threshold or a smooth degradation.

Mitigation status. The paper acknowledges this limitation implicitly by evaluating multiple video generators (Table I) and noting that "robotic data fine-tuning (e.g., Tesseract) tends to outperform generic generators." It does not propose any mechanism to improve the usable-video ratio or to make the pipeline more robust to lower-quality video generation. The dependence on proprietary models is not discussed as a limitation or flagged for future work — it is presented as an empirical finding rather than as a deployment constraint. A natural mitigation (training an open-source video generator specifically for PhysWorld-compatible outputs, perhaps using reinforcement learning with a pose-estimation reward) is not explored.

---

Difficulty Estimation and Filtering Cost Is Unaccounted For

The assumption or constraint. The PhysWorld pipeline generates one usable video per task execution, but "usability" (defined as robust object pose recovery by FoundationPose) can only be determined after running pose estimation on the generated video — a process that involves MegaSaM depth estimation, metric calibration, and FoundationPose tracking across all T frames. This filtering step is computationally non-trivial and is performed before the main pipeline (mesh generation, physical scene construction, RL training) can proceed. The paper does not account for the cost of generating and filtering multiple videos to obtain a single usable one, stating only the usable-video ratios without quantifying the computational overhead of the filtering process itself or the regeneration cost when videos are rejected.

The consequence. The headline result — "PhysWorld eliminates the need for real-world data collection" — is true only after a usable video has been obtained. In practice, a deployment would need to (a) generate N videos (where N depends on the video model's usable ratio), (b) run the full 4D reconstruction and pose estimation pipeline on each, and (c) select the best one. For Veo3 at 70% usable ratio, the expected number of generations per task is approximately 1.4 — a modest overhead. But for Tesseract at 36%, this rises to approximately 2.8 generations; for open-source models, it becomes 25–50 generations. The computational cost of filtering — running MegaSaM on each video, calibrating depths, running FoundationPose — scales linearly with the number of generated videos and may dominate the cost of the downstream pipeline for lower-quality generators. This overhead is not reflected in any reported efficiency metric.

What evidence exists in the paper. Table I reports usable-video ratios but does NOT report the computational cost of filtering (FLOPs, GPU-hours, or wall-clock time), the rejection criteria beyond "poses can be recovered robustly," or the regeneration strategy (does the system simply generate until a usable video appears? Is there a timeout? What happens if no usable video is found after K attempts?). The paper also does not report whether pose estimation quality (even for "usable" videos) correlates with downstream policy success — a video that barely passes the usability threshold might still produce lower-quality pose trajectories that degrade policy performance.

Mitigation status. The paper does not address this limitation. There is no discussion of filtering cost, no proposal for cheaper usability prediction (e.g., a lightweight classifier that predicts whether FoundationPose will succeed without running it), and no analysis of whether the filtering stage could be integrated with the downstream pipeline (e.g., training the policy to be robust to noisy poses from marginally-usable videos rather than discarding them). The paper treats "usable video" as a binary property and does not explore the continuum of video quality between usable and unusable.

---

Single Benchmark Domain with Limited Task and Object Diversity

The assumption or constraint. All experiments are conducted on 10 real-world tabletop manipulation tasks designed by the authors, evaluated with 10 rollouts per task (100 total evaluations per method). The tasks, while diverse in manipulation primitives (pouring, placing, wiping, sweeping, insertion), are all tabletop scenarios with rigid or semi-rigid household objects in structured settings with a clear supporting surface. The paper does not evaluate (a) deformable objects (cloth, rope, dough) that would break the rigid-body object-centric tracking assumption, (b) transparent or highly specular objects that would challenge depth estimation and pose tracking, (c) multi-object manipulation requiring coordination of multiple items simultaneously, (d) non-tabletop settings (cabinets, drawers, shelves at varying heights), or (e) cluttered scenes with heavy occlusions. The paper states the tasks cover "diverse real-world manipulation tasks" (Section IV) but does not claim broader generalization, nor does it discuss what task properties are necessary for PhysWorld to succeed.

The consequence. A practitioner cannot determine from this paper whether PhysWorld would work for their specific manipulation task if it falls outside the evaluated distribution. Key unknowns include: (a) whether the image-to-3D generator [67] produces usable meshes for deformable or unusual-shaped objects (a crumpled napkin, a coiled cable, a transparent bottle); (b) whether FoundationPose [69] can track poses for objects with few visual features or with specular highlights; (c) whether the physical property estimation via VLM works for objects with non-obvious physical behavior (a spring-loaded clip, a hinged lid, a viscous liquid); (d) whether the gravity alignment and collision optimization steps generalize to scenes without a dominant planar support surface (e.g., objects on shelves, in bins, or hanging). The paper's strong results (82% average success) establish proof-of-concept on a curated task set but provide no evidence about how performance would degrade on tasks with different physical or perceptual characteristics.

What evidence exists in the paper. The paper evaluates 10 tasks, each with 10 rollouts (Figure 4). Per-task success rates range from approximately 60% to 90%, showing some task-dependent variation. However, the paper does NOT report (a) what object, scene, or task properties correlate with success or failure within PhysWorld (beyond the aggregate failure mode analysis in Figure 6), (b) any evaluation on tasks where PhysWorld was expected to fail and did (negative examples that would characterize the failure boundary), or (c) any evaluation on standard robotics benchmarks (e.g., RLBench, MetaWorld, CALVIN) that would enable comparison with a broader range of prior methods beyond the three zero-shot video-based baselines. The test set of 10 tasks with 10 rollouts each also means the statistical reliability of per-task comparisons is limited — a standard error of approximately ±15 percentage points for a single task at 50% success rate.

Mitigation status. The paper does not discuss limitations regarding task or object diversity, does not characterize the expected operating envelope of the method, and does not provide failure case analysis on out-of-distribution tasks. The only acknowledgment of task-specific difficulty is the variation in per-task success rates (visible in Figure 4), which is not analyzed. The lack of standard benchmark evaluation means the method cannot be positioned relative to the broader robot learning literature beyond the specific zero-shot video-based setting.

---

Physical World Model Fidelity Is Unquantified and Likely Task-Limiting

The assumption or constraint. PhysWorld reconstructs a physically interactable scene from a single generated video, but the reconstruction involves multiple sources of approximation that are never individually evaluated. The key approximations include: (a) generative image-to-3D meshes that hallucinate unseen object geometry — the paper acknowledges 7% reconstruction errors (Figure 6) but does not characterize what types of errors occur (global scale mismatches, local shape distortions, missing fine details) or how they affect specific manipulation phases; (b) VLM-estimated physical properties (mass, friction) that are "typical" values for the object category, not measured from the actual object — a ceramic plate and a plastic plate of the same shape would be assigned different properties, but a heavy ceramic plate vs. a lightweight ceramic plate would not; (c) rigid-body simulation that cannot model deformable, granular, or liquid behavior — tasks involving pouring of multiple small items (candies, tomatoes) inherently violate the rigid-body assumption; (d) a fixed gravity alignment and collision optimization that assumes a dominant planar supporting surface.

The consequence. The 7% reconstruction error rate reported in Figure 6 is a lower bound on the fidelity problem — it only counts cases where reconstruction errors were the identifiable cause of task failure, not cases where reconstruction inaccuracies degraded policy quality but the task still succeeded, or cases where reconstruction errors interacted with other failure modes. More critically, the paper provides no sensitivity analysis: would performance degrade gracefully if reconstruction quality were worse, or is there a sharp threshold? For tasks involving non-rigid behavior (pouring candies, pouring tomatoes), the physics simulator fundamentally cannot capture the true dynamics — the policy must learn to compensate for this mismatch, but the paper does not measure how much the rigid-body assumption reduces success on pouring tasks compared to what would be achievable with more accurate physics (e.g., a granular material simulator).

The VLM-based physical property estimation is particularly concerning because it introduces a source of error that the paper never measures. If the VLM assigns incorrect mass or friction to an object, the RL policy learns in a simulation with wrong dynamics. The paper's implicit claim is that the residual policy can compensate — but this compensation has limits, and the paper does not test them. A heavy object assigned a light mass will behave unrealistically in simulation (moving too easily, not settling properly under gravity); the policy may learn to apply insufficient force, causing real-world failure.

What evidence exists in the paper. The failure mode analysis (Figure 6) attributes 7% of failures to reconstruction errors. Pouring tasks (Pour the tomato, Pour the fish, Pour the candies) have lower success rates (60–80%) than insertion tasks (80–90%) in Figure 4, which is consistent with the hypothesis that rigid-body simulation limits performance on dynamic, multi-object tasks. However, the paper does NOT report (a) an ablation comparing VLM-estimated properties against default uniform properties, (b) an evaluation of mesh reconstruction accuracy (by comparing generated meshes against scanned ground-truth meshes of the real objects), (c) a sensitivity analysis showing how policy success varies with increasing reconstruction error, or (d) a comparison against a baseline using a more sophisticated physics simulator for granular/fluid tasks.

Mitigation status. The paper acknowledges this limitation only through the failure mode analysis. The discussion in Section V states: "Physical world modeling is bounded by the fidelity of physical simulators and may introduce additional sim-to-real gaps. However, from the evidence in Figure 4, we still believe in the necessity of introducing a world model to provide reliable physical feedback for more robust learning." This is a reasonable position — even imperfect physics is better than no physics, as the numbers show — but it does not address the practical question of whether the current fidelity level is sufficient for deployment, or how much improvement could be gained from better physics, better property estimation, or better mesh reconstruction. The paper does not propose future work on improving physical fidelity or on characterizing the relationship between fidelity and downstream success.

---

No Dynamic or Adaptive Difficulty Response — Static Pipeline with Fixed Strategy

The assumption or constraint. PhysWorld uses a fixed, sequential pipeline (video generation → reconstruction → RL training → deployment) that applies the same strategy regardless of task characteristics. All 10 tasks receive the same pipeline with the same components (Veo3, MegaSaM, TRELLIS-style 3D generation, residual PPO with fixed reward weights). There is no mechanism for adapting the strategy based on task difficulty (e.g., easy tasks might need simpler meshes or shorter RL training; hard tasks might need multiple video generations or more RL iterations). The paper does not explore whether the computational cost could be reduced on easy tasks or whether difficult tasks would benefit from more compute at specific stages. The difficulty of a task (as reflected in its per-task success rate, ranging from ~60% to ~90% in Figure 4) does not inform resource allocation.

The consequence. The current pipeline treats all tasks as equally demanding, spending approximately the same compute budget (one video generation, one reconstruction, one RL training run) regardless of whether the task is easy or hard. This is likely suboptimal. For example, "Pour the tomato from the pan onto the plate" achieves only ~60% success with PhysWorld, while "Wipe the whiteboard" achieves ~90%. The pouring task might benefit from generating multiple videos and selecting the one with the most physically plausible trajectory, or from longer RL training to better compensate for the rigid-body simulation mismatch with granular behavior. Conversely, the wiping task might achieve 90% with a fraction of the RL training steps, saving compute. Without difficulty-adaptive allocation, the pipeline is compute-inefficient: it under-invests in hard tasks (leaving performance on the table) and over-invests in easy tasks (wasting compute).

More fundamentally, the static pipeline has no mechanism for detecting that a task is likely to fail and escalating — for instance, if the video generation produces a marginally usable video with noisy poses, the policy trained on those poses will likely achieve low success, but the pipeline executes it anyway. A smarter system could detect this upstream (after pose estimation shows high uncertainty) and regenerate the video or request a different prompt formulation before committing to the expensive RL training step.

What evidence exists in the paper. Figure 4 shows task-level variation in success rates (60–90%), demonstrating that the same pipeline produces different-quality results across tasks. Figure 7 shows that residual RL converges in a few hundred iterations for one task, but doesn't show whether convergence speed or asymptotic performance varies across tasks. Table I shows video quality varies by generator model, but doesn't show task-level variation in usable-video ratios within Veo3 — some tasks might consistently produce lower-quality videos than others. The paper does NOT report (a) compute-to-performance curves showing how success rate scales with RL training steps per task, (b) an analysis of whether recurrent failure modes on hard tasks could be addressed with more compute, or (c) any resource allocation strategy beyond the fixed pipeline.

Mitigation status. The paper does not address this limitation at all. There is no discussion of compute efficiency, no analysis of how training time varies across tasks, and no proposal for adaptive or difficulty-aware resource allocation. This is a missed opportunity because the framework naturally supports adaptation: video generation quality can be assessed upstream (FoundationPose tracking confidence), mesh reconstruction fidelity can be evaluated before training (by comparing rendered meshes with the observed point cloud), and RL training can be monitored for convergence (tracking reward plateau). A system that used these signals to dynamically allocate compute could substantially improve efficiency without modifying the core method.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new fundamental algorithm — the individual components (Veo3, MegaSaM, image-to-3D generation, FoundationPose, PPO) are all prior work. Its contribution is architectural: it demonstrates that coupling video generation with explicit physical world reconstruction, and training residual policies within that reconstructed world, closes a gap that prior zero-shot video-to-action methods could not bridge. The shift is from treating generated videos as scripts to be retargeted to treating them as approximate guidance to be grounded through physical interaction. This is a reframing rather than a paradigm shift — it changes how we think about the problem rather than what problems we can solve — but it is a practically consequential reframing because it converts the field's existing components from a pipeline that fails ungracefully (RIGVid's 67% average, with 23% combined grasping and tracking failures) into one that succeeds more reliably (82% average, with those failure categories largely eliminated).

The reconciliation of prior contradictions is one of the paper's more valuable contributions, even if largely implicit. Prior work on video-to-action transfer appeared to reach contradictory conclusions: methods that learned action models from generated videos using real robot data ([1]–[5]) claimed video generation was useful for policy learning, while methods that extracted actions purely from visual cues ([6]–[8]) showed promise but with clear reliability limitations (67% in [8] leaves substantial room for improvement). PhysWorld's diagnostic is that the reliability gap is not about video quality or motion representation choice — it's about the absence of physical feedback. This explains why methods that use real robot data for alignment partially succeed (the real data provides implicit physical grounding through the action labels) while zero-shot visual methods hit a ceiling (no physical grounding mechanism exists). The paper resolves this apparent contradiction by identifying physics as the missing term, and the numbers in Figure 4 and Figure 6 make this causal narrative quantitative rather than speculative.

The work also makes a specific methodological contribution to the real-to-sim-to-real literature by showing that generative priors can substitute for multi-view capture when building simulation environments for policy learning. Prior real-to-sim pipelines required dedicated multi-view scans of each scene ([50]–[62]), tying them to specific physical locations and making them incompatible with generated videos (which are monocular by nature). PhysWorld demonstrates that image-to-3D generators and inpainted background reconstruction can produce simulation environments from a single view, trading geometric fidelity for completeness. The 7% reconstruction error rate (Figure 6) versus the 23% reduction in other failure categories quantifies this tradeoff: the approach works because grasping and tracking errors are more lethal than geometric mismatch for the tasks evaluated. This insight — that for manipulation policy learning, physical completeness may matter more than geometric fidelity — could reshape how the real-to-sim community thinks about reconstruction requirements.

The research directions that become more attractive after this work include: (a) improving the quality and controllability of video generation for robotics, since Table I shows generation quality is the bottleneck; (b) developing lightweight, adaptive physical world models that can be built quickly from monocular input; (c) designing reward functions and policy architectures that are robust to the specific kinds of simulation error (mesh inaccuracy, simplified physics) that real-to-sim pipelines introduce. Directions that become relatively less attractive — or at least need re-evaluation — include pure visual retargeting without physical feedback (the 67% ceiling in [8] suggests diminishing returns from better tracking or flow estimation alone) and embodiment-centric learning from generated videos (Table II's 10–30% success rates suggest this is a dead end without solving the hand hallucination problem).

Follow-Up Research This Work Enables

Characterizing and closing the video generation bottleneck with a robotics-specific evaluation suite. Table I reveals a 35× gap in usable-video ratio between Veo3 (70%) and the best open-source model (CogVideoX1.5-5B at 4% before Tesseract's 36%). The paper does not decompose "usability" into specific failure modes. A strong follow-up would build a diagnostic benchmark: for each generated video, independently measure temporal consistency (optical flow smoothness), object permanence (does the object disappear or morph?), camera stability (unwanted viewpoint changes), and pose estimability (FoundationPose tracking success per frame). This would identify which generation failures dominate — is it primarily object hallucination (reducing object permanence), motion blur (reducing pose estimability), or something else? The benchmark could then guide targeted improvements: if object permanence is the bottleneck, training video generators with object-level consistency losses; if motion blur is the problem, enforcing temporal smoothness priors. The actionable output would be an open-source model matching Veo3's 70% on this specific robotics-usability metric, which would make PhysWorld-style pipelines deployable without proprietary API dependence.

Training policies that are robust to reconstruction error rather than filtering for perfect reconstructions. The paper treats mesh reconstruction as a binary gate: if poses can be recovered, the video is usable; if not, it's discarded (7% of failures attributed to reconstruction errors, Figure 6). This discards information — a mesh that is imperfect but not catastrophically wrong might still support effective policy learning if the policy is trained to handle geometric uncertainty. A concrete experiment: introduce controlled perturbations to the generated meshes during RL training (random scaling, vertex noise, mass randomization) and measure whether policies trained with domain randomization transfer more robustly than policies trained on a single deterministic mesh. The hypothesis is that domain-randomized policies would reduce the 7% reconstruction error failure rate by learning to handle the kinds of mismatch that occur in practice (slightly wrong object scale, slightly off mass distribution). If this works, it would also reduce the usable-video threshold — currently "unusable" videos whose only flaw is moderate mesh inaccuracy might become usable with robustness training. The experiment requires: (a) a mesh perturbation model that captures realistic reconstruction errors (based on comparing generated meshes against ground-truth scans of the same objects), (b) a comparison of success rates for policies trained with and without domain randomization, evaluated on both high-quality and deliberately-degraded meshes.

Scaling the residual RL formulation across tasks to test whether the physical world model enables transfer learning. PhysWorld trains a separate policy for each of 10 tasks, each in its own task-specific reconstruction (one generated video per task). This treats the physical world model as a per-task simulation environment. But the world model encodes general physical knowledge — gravity, contact dynamics, collision geometry — that should transfer across tasks. A natural follow-up: reconstruct the same scene once, generate videos for multiple tasks in that scene, and train a multi-task policy (or a meta-policy that adapts to new tasks with minimal fine-tuning). The key measurement is: does a policy pretrained in the physical world model on Task A learn Task B faster than a policy trained from scratch on Task B? If yes, it demonstrates that the physical world model provides a reusable training substrate — the RL agent learns general manipulation skills (how hard to grip, how to approach objects, how to maintain contact) that transfer because the physics is shared. This would position PhysWorld not just as a way to solve specific tasks zero-shot, but as a way to build general-purpose manipulation priors in simulation without any real robot data. The experiment is technically feasible with PhysWorld's current pipeline: build one reconstruction per scene, generate multiple task videos, and modify the RL training loop to support multi-task or continual learning.

Stress-testing the pipeline on tasks that deliberately violate its assumptions to map the failure boundary. The paper evaluates 10 tabletop tasks with rigid household objects and a clear supporting surface. We don't know where PhysWorld breaks. A systematic stress test would push each assumption: (a) Deformable objects — a cloth folding task, a rope manipulation task. Does the object-centric tracking formulation work when the object's shape (not just pose) changes? If FoundationPose cannot track a deforming object, does substituting a keypoint-based tracker and training the policy on keypoint trajectories work? (b) Transparent or specular objects — a task involving a glass cup or a shiny metal tool. MegaSaM's depth estimation and FoundationPose's pose estimation both rely on visual features that may fail on specular surfaces. What is the combined failure rate, and can active depth sensing (the real D in RGB-D) partially compensate? (c) Cluttered scenes — a task where the target object is partially occluded by distractors. Does the object-background decomposition (ObjectClear inpainting) still work? Does the policy learn to move distractors out of the way, or does it get stuck? (d) Non-tabletop settings — a task inside a cabinet or drawer where the camera viewpoint is constrained. Does gravity alignment still work when the supporting surface is not a simple plane? The output would be a failure-mode map: a table showing success rates as a function of object properties (rigid/deformable, opaque/specular, isolated/cluttered) and scene properties (tabletop/constrained). This map would tell practitioners exactly which tasks PhysWorld can be trusted on and which require fundamentally different approaches.

Quantifying the contribution of each world model fidelity component through systematic ablation. The paper's central claim is that physical world modeling improves manipulation accuracy, but the world model is a bundle of components (mesh generation, physical property estimation, gravity alignment, collision optimization, rigid-body simulation) whose individual contributions are unknown. A systematic ablation would train policies in progressively degraded versions of the world model: (a) no world model (equivalent to RIGVid [8]), (b) world model with default uniform physical properties instead of VLM-estimated properties, (c) world model with no collision optimization (objects start with penetrations), (d) world model with simple primitive geometry (bounding boxes or convex hulls instead of image-to-3D meshes), (e) world model with no gravity alignment (simulation in camera frame). Each ablation measures the drop in policy success rate. This would identify which components are load-bearing — if uniform properties perform nearly as well as VLM-estimated ones, the VLM query can be dropped to reduce complexity; if primitive geometry causes a large drop, image-to-3D generation is essential. It would also distinguish between physics itself matters (any reasonable physics is better than no physics) versus accurate physics matters (the specific properties and geometry are important). This distinction matters for practitioners: the former means PhysWorld's approach is robust and can be simplified; the latter means careful tuning of the world model is critical and the 7% reconstruction error rate might be a real performance ceiling.

Closing the sim-to-real loop: using PhysWorld's successes and failures to improve the video generator. The paper uses video generation as a static input and builds the rest of the system around it. But the physical world model produces a signal — task success or failure, and the specific failure mode — that could be used to improve the video generator itself. Concretely: for tasks where PhysWorld fails, identify whether the failure traces back to a physically implausible video (e.g., an object trajectory that violates rigid-body dynamics, causing the RL policy to learn an infeasible target). Collect a dataset of (video, success/failure, failure reason) pairs. Fine-tune the video generator to produce videos that are more likely to lead to PhysWorld success — using reinforcement learning where the reward is downstream task success rate, or using preference optimization where "good" videos are those whose object trajectories are physically realizable (as measured by whether the residual policy can track them with high reward in the simulator). This closes the loop: video generation initializes the system, physical world modeling provides a quality signal, and that signal improves future video generations. The experiment requires (a) differentiable or at least queryable access to the video generator (Veo3 is proprietary, but this could be done with Tesseract [11] at 36% usable ratio to start), (b) a metric connecting video properties to PhysWorld success that can serve as a training signal, and (c) careful experimental design to avoid reward hacking (the generator might learn to produce videos that are easy for PhysWorld but don't actually depict the task correctly). This direction, if successful, would transform PhysWorld from a consumer of video generation into a self-improving system where the physical world model provides the feedback for better visual guidance.

Practical Applications and Downstream Use Cases

Rapid deployment of robot manipulation skills in warehouses and logistics centers where task variety is high but per-task volume is moderate. In e-commerce fulfillment, robots handle thousands of different product types with constantly changing task requirements (new packaging, new item categories, new picking-and-placing workflows). Traditional robot programming requires engineers to hand-code each task, and imitation learning requires collecting demonstrations for each new product. PhysWorld's pipeline — one image of the workspace, one language command, no real robot data — could reduce the per-task deployment cost from hours/days (engineering or data collection) to minutes (video generation + reconstruction + RL training). The 82% average success rate suggests the system would need human oversight for the remaining 18% of attempts, but even an 82% autonomous success rate with quick human intervention on failures is economically viable for many logistics operations. The primary practical challenge for this use case is the proprietary video generation dependency (Veo3) and the need for a depth camera on each robot workstation — both solvable with current technology, though the API cost of Veo3 per task would need to be amortized over the number of items processed.

Home robotics for assistive manipulation, where the environment is unique to each user and task specification must be done by non-experts through natural language. A home robot deployed in an elderly person's apartment needs to handle tasks like "put the medicine bottle on the nightstand" or "pour the soup from the pot into the bowl." The apartment's layout, objects, and tasks are specific to that user and cannot be pre-programmed. PhysWorld's ability to generate task-specific policies from a single image of the current scene and a natural language command makes it uniquely suited to this setting — the user describes what they want in plain language, the robot takes a picture, and a policy is synthesized without any pre-existing map of the home or database of object models. The 82% average success rate is not yet high enough for unsupervised home deployment (a 18% failure rate means a spill or a dropped object roughly one in five attempts, which is unacceptable for a vulnerable user), but it is high enough for a supervised or semi-autonomous system where a remote human operator can intervene on failures. The most critical practical barrier is that home environments contain many objects that violate PhysWorld's assumptions — deformable items (clothing, towels), transparent containers (glass cups), cluttered countertops — and the paper provides no evidence about success rates on such objects. A deployment would need the stress-testing experiments described above to characterize the expected failure rate on home-typical object distributions.

Self-improving data generation for training generalist robot policies. The dominant paradigm for training generalist robot policies (e.g., RT-2, Octo, π0) is to collect massive datasets of real robot demonstrations and train large transformer models on them. This approach is bottlenecked by data collection cost and diversity. PhysWorld could serve as a data engine: for each new scene and task encountered, generate a video, reconstruct the physical world, train a PhysWorld policy, execute it on the real robot, and record the successful episodes as new training data for the generalist policy. Critically, the successful PhysWorld executions are real robot data — the policy was trained in simulation but executed in reality, so the recorded trajectories are real demonstrations with real physics. This creates a virtuous cycle: PhysWorld enables zero-shot execution, successful executions become training data, the generalist policy improves, making future zero-shot execution easier or enabling the generalist policy to replace PhysWorld entirely for common task types. The 82% average success rate means roughly 4 in 5 attempts produce usable demonstration data — vastly more efficient than teleoperation-based collection, which is 100% successful but requires constant human attention, or fully autonomous exploration, which typically has much lower success rates for complex manipulation. The practical requirement is that the robot setup includes the necessary sensors (RGB-D camera) and compute (GPU for video generation, simulation, and RL training), which is feasible for a dedicated data-collection workstation.

When to Prefer This Method

The paper compares PhysWorld against zero-shot video-to-action methods ([6]–[8]) that operate without physical world modeling, and against embodiment-centric learning (Table II). The tradeoffs are implicit in the results rather than explicitly articulated, but the data supports the following decision criteria:

• Prefer PhysWorld over RIGVid [8] when: the task involves precise manipulation where small grasping or positioning errors cause failure (insertion, placement). Figure 6 shows PhysWorld eliminates tracking failures (5% → 0%) and reduces grasping failures (18% → 3%), with the largest gains on "Put the book in the bookshelf" (90% vs. 60%) and "Put the shoe in the shoebox" (80% vs. 60%). If the cost of generating the physical world model (video → reconstruction → RL training) is acceptable, the reliability improvement justifies it. If the task is simple pick-and-place with large tolerances, RIGVid's 67% average may be sufficient and the additional complexity of PhysWorld may not be warranted — but the paper does not identify which specific tasks fall into this category.

• Prefer object-centric over embodiment-centric learning when: the source of demonstrations is generated video rather than real human video. Table II shows 60–70 percentage point gaps on two tasks, attributed to hand hallucination in generated videos. If the demonstrations come from real human videos (where hand kinematics are consistent and estimable), embodiment-centric learning might perform better than the 10–30% reported here — but the paper provides no data on this regime. The decision rule is therefore specific to the video generation context.

• Prefer residual RL over RL from scratch when: training time or sample efficiency is a constraint. Figure 7 shows residual RL converging within a few hundred iterations versus RL from scratch lagging at the same budget, on the "Pour the tomato" task. If the baseline grasp-and-plan actions are available (requiring AnyGrasp [70] and CuRobo [71] or equivalents), residual RL is the more practical choice. If baseline actions are not available (unusual robot kinematics, tasks where grasping and planning models fail entirely), RL from scratch becomes the fallback, but the paper provides no data on how it performs across the full task set — only the single-task comparison in Figure 7.