VideoWorld 2: Learning Transferable Knowledge from Real-world Videos

ArXiv: 2602.10102

🎯 Pitch

VideoWorld 2 shows that AI can learn transferable task knowledge—like paper folding—directly from unlabeled real-world videos by decoupling the rules of action from visual appearance. It completes minute-long 7-step crafts with 68.8% success where prior video generation models fail completely after step 4, and this learned knowledge even transfers across robots and domains.

---

1. Executive Summary

This paper introduces VideoWorld 2, which extends the prior VideoWorld framework to investigate—for the first time—learning transferable knowledge for complex, long-horizon tasks directly from raw real-world videos. The core innovation is a dynamics-enhanced Latent Dynamics Model (dLDM) that decouples action dynamics from visual appearance by offloading appearance modeling to a pretrained video diffusion model, enabling the learned latent codes to focus exclusively on compact, task-relevant dynamics rather than superficial visual details. Evaluated on the newly introduced Video-CraftBench (minute-long handicraft tasks like paper folding and block building) and on robotic manipulation via the Open-X → CALVIN transfer setup, VideoWorld 2 achieves up to 70% improvement in task success rate over competing methods—completing full 7-step paper folding sequences with 68.8% success where all baselines fail entirely after step 4—while also producing coherent long-horizon execution videos in unseen environments. The work establishes that appearance-dynamics disentanglement is critical for real-world knowledge acquisition, with the learned representations proving transferable across agents and domains only when a pretrained video diffusion model absorbs the visual appearance burden.

2. Context and Motivation

The Core Problem: Learning Transferable Knowledge from Real-World Videos Without Labels

The fundamental question this paper tackles is: Can AI models learn generalizable, transferable knowledge for complex, long-horizon tasks directly from unlabeled real-world videos? This is not merely about video understanding or generation—it is about extracting the underlying rules, dynamics, and planning capabilities that constitute task-solving knowledge from raw visual observation alone, without any language supervision, action labels, or explicit reward signals.

This matters because the real world is fundamentally a visual domain where vast amounts of implicit knowledge are encoded in how things move, interact, and transform. A child watching someone fold a paper airplane learns not just the specific video they saw, but an abstract understanding of the folding procedure that can be applied to different paper materials, in different environments, under different lighting conditions. Current AI systems cannot do this. They can generate visually stunning videos, but they do not extract the procedural knowledge embedded in those videos—the sequence of actions, the physical transformations, the causal relationships that constitute "knowing how to fold a paper airplane."

The paper frames this gap with theoretical and practical significance:

• Theoretical significance: Learning from unlabeled video represents a path toward more human-like knowledge acquisition. Unlike text, video contains rich information about world dynamics, spatial relationships, and underlying physical laws that cannot be fully captured through language alone. If AI systems could learn from the enormous corpus of video content available on the internet—tutorials, demonstrations, recordings of real-world interactions—they could acquire a far more grounded understanding of how the world works.

• Practical significance: The ability to learn transferable task knowledge from video would enable applications where language instruction is insufficient or unavailable. Fine-grained manipulation skills (handicrafts, surgery, repair procedures) are notoriously difficult to describe precisely with words—the subtle hand movements, material deformations, and spatial reasoning involved resist verbal articulation. Video provides a natural medium for capturing and transferring such knowledge.

Conflicting Evidence: Why Existing Approaches Fail

The paper is motivated by a clear failure pattern observed across multiple lines of prior work. There is no contradiction to resolve—rather, there is a consistent inability that manifests differently depending on the approach taken.

VideoWorld's promise and limitations. The original VideoWorld (Ren et al., 2025) was among the first works to demonstrate that models can learn rules, reasoning, and planning capabilities solely from visual signals using an autoregressive video generation paradigm. It succeeded on synthetic domains—Go game records and simulated robotics environments—where visual complexity is controlled and limited. However, the paper states clearly that "extending this paradigm beyond synthetic domains remains an open challenge":

"Real-world videos exhibit substantial visual diversity, complex action dynamics, and often involve long-horizon, multi-step interactions. These characteristics prevent the training approach and model design of VideoWorld from being directly applied to realistic settings."

When presented with minute-long, multi-step real-world task videos, VideoWorld fails. It cannot extract core task-solving knowledge or generalize it to novel scenarios through observation alone—"even for tasks such as paper folding that are easily mastered by children." The failure mode is specific: VideoWorld's latent codes capture irrelevant visual details such as background motion, lighting changes, textures, and camera displacement, making the model sensitive to environmental changes. As shown in qualitative results (Figure 8), predictions contain severe errors including distorted hand poses, incorrect object shapes, and inconsistent environmental appearances. The model overfits to appearance details of the training environment and fails to generalize.

State-of-the-art video generation models produce visually appealing but task-useless outputs. The paper tests four leading video generation models—NVIDIA Cosmos AR 4B, Cosmos DiT 2B, Wan2.2 14B, and HunyuanVideo 13B—fine-tuned on Video-CraftBench with step-by-step textual descriptions provided. These models achieve high success rates (>68%) on the first step of paper folding and up to 38% on block stacking. However, their performance deteriorates rapidly: by step 4 of the 7-step paper folding task, success rates drop to ≤10.6%, with total failure to generate subsequent steps. The paper states:

"We conjecture that these models are unable to disentangle task-core actions embedded in visual changes and motion dynamics, and instead overfit to irrelevant visual details. This overfitting leads to degraded task performance and unstable long-horizon generation."

Crucially, these models fail despite receiving detailed textual descriptions of each step—they have language guidance that a child does not need—yet they still cannot reproduce the task in novel environments.

Latent action models (LAMs) address short-horizon tasks but collapse on long sequences. The paper evaluates five concurrent approaches that compress inter-frame visual changes into latent codes for manipulation tasks: Moto, AdaWorld, LAPA, iVideoGPT, and CoLA. While these methods can generate coherent videos, they fail to generalize to novel environments in the long-horizon setting. Specifically:

• None successfully completed the full 7-step paper folding sequence in new environments with different desktops, paper textures, or backgrounds.
• Their predictions exhibit "substantial action errors and scene distortions, indicating that their latent codes overfit to irrelevant appearance information."
• LAPA's decoding into long-horizon sequences "causes severe degradation" due to structural constraints.
• CoLA, the closest concurrent work that also uses a VDM, is "limited to short 2-frame transitions and ignores the structured temporal cues from coarse VAE outputs."

The paper explicitly notes that "existing latent action models typically focus on short-horizon or pairwise transitions" involving tasks like grasping, toggling, or 2D games—settings far less challenging than the minute-long handicraft tasks with complex visual dynamics, multiple stages, and substantial appearance variation targeted here.

"Disentanglement" in prior video generation work addresses a different problem. The paper draws a clear distinction between its objectives and prior work on disentangling motion from appearance. In prior approaches, disentanglement typically refers to separating motion from appearance for applications like style transfer or visual editing—isolating camera movement or object-specific changes. These methods often rely on explicit geometric supervision, capture only coarse global motion semantics, or depend on handcrafted residual encoding. VideoWorld 2 targets a fundamentally different objective: "reducing task-irrelevant information to learn transferable visual dynamics for complex long-horizon tasks," which is "beyond the capability of prior approaches."

The Entanglement Problem: Why Appearance and Dynamics Must Be Separated

The paper's central diagnostic insight is that all prior approaches fail because of insufficient disentanglement of action dynamics and visual appearance. This is not merely an observation—it is a conjectured mechanism with specific implications.

In the basic video generation framework, models encode videos into compressed representations that must capture the full spectrum of visual information: object textures, lighting, backgrounds, camera angles, and the subtle hand movements and object deformations that constitute the task-relevant actions. These representations require "thousands of discrete tokens or continuous embeddings," inevitably leading to "spatiotemporal redundancy and a sparse distribution of knowledge." The critical task-relevant information—the action dynamics—is diluted in a sea of appearance detail.

VideoWorld attempted to mitigate this through its Latent Dynamic Model (LDM), which compresses future visual changes into compact latent codes using learnable query embeddings with cross-attention. However, the paper finds that this compression alone is insufficient: the learned latent codes still encode irrelevant visual details that make the model sensitive to environmental changes. The reconstruction objective—the original LDM minimizes ℓ₂ distance between original and reconstructed frames—forces the latent codes to preserve appearance fidelity alongside dynamics, preventing true disentanglement.

The paper's key diagnostic statement:

"We conjecture that these limitations stem from the insufficient disentanglement of action dynamics and visual appearance. In the basic framework, the joint modeling of these components impedes the effective extraction of task-core action knowledge."

This leads to two distinct failure modes: (1) the model overfits to appearance details of the training environment and fails to generalize to novel visual settings, and (2) error accumulation in long-horizon generation—small appearance errors compound over time, eventually corrupting the action sequence itself.

How This Paper Positions Itself

VideoWorld 2 positions itself as addressing the specific gap between synthetic-domain success and real-world failure. The paper does not propose an entirely new paradigm; rather, it identifies the specific architectural limitation preventing VideoWorld from scaling to real-world settings and proposes a targeted solution: explicitly delegating appearance modeling to a pretrained video diffusion model (VDM) so that the latent dynamics codes are forced to focus exclusively on task-relevant visual changes.

The key distinction from prior work is methodological precision about what should be decoupled and how:

• From VideoWorld: VideoWorld 2 inherits the autoregressive latent dynamics modeling framework but replaces the standard decoder with a pretrained VDM. This is not merely swapping components—it fundamentally changes the training objective for the latent codes. In VideoWorld, the decoder must reconstruct both appearance and motion, forcing codes to encode both. In VideoWorld 2, the VDM handles appearance, so codes face pressure to only capture dynamics that the VDM cannot produce on its own—the task-relevant actions.

• From video generation models: VideoWorld 2 uses pretrained VDMs not as end-to-end generators (which the paper shows fails), but as appearance decoders that render high-fidelity frames conditioned on separately learned dynamics codes. This inverts the relationship: the VDM serves the dynamics model, rather than the dynamics being implicitly (and inadequately) represented within the VDM's generation process.

• From latent action models: VideoWorld 2's dLDM models multi-step temporal dynamics (processing 93-frame clips) rather than pairwise transitions, and reuses coarse VAE decoder outputs as structured temporal cues for the VDM—a mechanism "critical for long-horizon tasks" that prior work (including CoLA, which also uses a VDM) does not employ.

The paper also positions itself through the introduction of Video-CraftBench, a benchmark explicitly designed to test long-horizon real-world knowledge acquisition where prior benchmarks are insufficient. The handicraft tasks—paper airplane folding, paper boat folding, block building—require minute-long sequences of fine-grained manipulations under diverse visual conditions. This benchmark provides a foundation for future research that the authors argue is currently missing, since existing benchmarks focus on short-horizon robotic tasks or synthetic environments.

3. Technical Approach

3.1 Reader Orientation

What the system is: VideoWorld 2 is a generative model that learns to produce long-horizon task execution videos in unseen environments after training only on unlabeled demonstration videos from other environments. It operates by first compressing the visual changes in a video into a compact set of discrete latent codes that represent task-relevant actions, then modeling these codes autoregressively to plan action sequences, and finally decoding the planned codes back into high-fidelity video frames using a pretrained video diffusion model.

What problem it solves and the shape of the solution: The core problem is that prior approaches—whether end-to-end video generation models or latent dynamics models—entangle task-critical action information with irrelevant visual appearance details (backgrounds, lighting, textures, camera viewpoints), causing them to either fail on long sequences or fail to generalize to new environments. The solution is architectural: rather than asking a single model to handle both appearance and dynamics, VideoWorld 2 explicitly splits these responsibilities. A causal VQ-VAE encoder extracts compact latent dynamics codes from visual changes between frames; a pretrained video diffusion model (VDM) handles all appearance modeling during reconstruction; and the latent codes are thus freed from encoding visual detail and forced to focus exclusively on capturing transferable action dynamics. An autoregressive transformer then learns to predict these codes from an initial frame, enabling it to serve as a policy that plans action sequences in new environments.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, connected in a training pipeline that produces a model deployable for inference:

1. Dynamics-enhanced Latent Dynamics Model (dLDM) Autoencoder: The heart of the system. It takes a video clip as input and compresses the visual changes between the first frame and subsequent frames into a small set of discrete latent codes. During training, it reconstructs the future frames through two parallel paths: a lightweight decoder that produces coarse, motion-rich reconstruction (used as conditioning), and a pretrained VDM that produces the final high-fidelity output. The codes represent "what changed in a task-relevant way."

2. Pretrained Video Diffusion Model (VDM): A frozen or fine-tuned large video generation model (Cosmos DiT 2B) that serves as the appearance renderer. It takes the first frame, the coarse decoder output (as motion guidance), and the latent dynamics codes (via cross-attention) and generates high-quality future frames. It handles all visual detail so the latent codes do not have to.

3. Autoregressive (AR) Transformer: After the dLDM is trained, it processes every training video to extract a sequence of latent dynamics codes. The AR transformer (Cosmos AR 4B) is then trained on these code sequences to predict future codes given past codes and an initial observation, effectively learning a policy over the latent action space.

4. Inference Pipeline: Given a single frame from a new, unseen environment, the AR transformer autoregressively predicts a sequence of latent dynamics codes. These codes are fed to the dLDM decoder (specifically, to the VDM) which generates the corresponding long-horizon task execution video in the new environment.

Information flows as follows during training: a video clip enters → the dLDM encoder extracts latent dynamics codes → these codes are quantized → a lightweight decoder reconstructs coarse motion frames → the VDM generates high-fidelity frames conditioned on the codes, the coarse frames, and the first frame → the dLDM is trained with reconstruction and denoising losses. Separately, the trained dLDM processes full videos to produce code sequences → the AR transformer is trained to predict these codes autoregressively. During inference: an initial frame enters → the AR transformer predicts a code sequence → the VDM decodes the codes into frames in the target environment.

3.3 Roadmap for the Deep Dive

• First, the formal problem statement—how VideoWorld 2 defines generative knowledge learning from unlabeled videos, the tuple notation, and what exactly the model must learn to predict. This establishes the mathematical objective that everything else serves.
• Second, the basic generative framework and why it fails—how standard video generation models (VDMs with VQ-VAE encoders) represent videos, why thousands of tokens are needed, and how this leads to the entanglement problem. This provides the baseline that VideoWorld attempts to improve upon.
• Third, the original Latent Dynamic Model (LDM) from VideoWorld—how it compresses visual changes into compact latent codes, the query-based attention mechanism, the quantization step, and its reconstruction objective. Understanding the LDM is essential because the dLDM modifies it, not replaces it.
• Fourth, the dynamics-enhanced LDM (dLDM)—the core innovation—how the pretrained VDM replaces the original decoder, the three conditioning pathways into the VDM, the role of the coarse VAE decoder output as motion guidance, and why gradient-stopping is critical. This is where the disentanglement actually happens.
• Fifth, the autoregressive transformer training and inference—how the dLDM's latent codes become the prediction targets for a policy model, how the task instruction and initial frame condition the AR transformer, and how inference works in new environments.
• Finally, the training procedure and key hyperparameters—the warm-up strategy that teaches the latent codes to capture motion before the VDM is introduced, the loss functions, and all numerical configurations (optimizer, learning rates, batch sizes, codebook sizes, clip lengths).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that the failure of prior approaches on real-world video knowledge learning stems from entanglement of appearance and dynamics, and that explicitly delegating appearance modeling to a pretrained VDM—while forcing latent codes to capture only task-relevant visual changes—produces transferable, generalizable dynamics representations.

---

Generative Knowledge Learning from Unlabeled Videos

The paper formalizes the problem of learning from unlabeled videos as a generative modeling task. A video is treated as a demonstration trajectory that captures world state transitions and the underlying action policy. The objective is to train a model that can reproduce these trajectories in new contexts without access to action labels or language supervision.

The setting is defined as a tuple:

$\mathcal{G} = \langle \mathcal{X}, \mathcal{A}, \rho \rangle$

where $\mathcal{X}$ is the observation space (video frames), $\mathcal{A}$ is the action space (the latent actions that cause state transitions), and $\rho$ is a video generator.

What it computes: This tuple is a formal specification that the system must contain three things: a space of possible visual observations, a space of possible actions or interventions, and a generator that can produce future observations when conditioned appropriately. It frames the problem as one of learning $\rho$—the generator—from data.

Why this form: The generator $\rho$ is the unifying abstraction. It must serve two roles: during training, it models the conditional distribution of the next frame(s) given observation history, enabling knowledge acquisition through video generation without explicit labels. During deployment, it functions as a policy model $\pi(\cdot | x_{0:t}) : \mathcal{X} \to \mathcal{A}$ that maps visual state transitions into action space and predicts actions based on historical observations. This dual-use formulation—generator as both world model and policy—is what makes the approach "knowledge learning" rather than mere video synthesis.

The conditional distribution that $\rho$ must model is:

$p(x_{t+1} | x_{0:t})$

where $x_{0:t}$ is the sequence of video frames from time 0 to time $t$, and $x_{t+1}$ is the next frame to be predicted.

What it computes: Given all frames observed so far, predict the next frame. This is a standard autoregressive next-frame prediction objective, but with a crucial property: if the model can consistently predict correct future frames in novel environments, it must have learned the underlying dynamics and policy that generate those frames—the "knowledge" the paper aims to extract.

Why this form: Autoregressive next-frame prediction is the most general formulation because it makes no assumptions about what constitutes an "action" or what the observation space looks like. It reduces all knowledge acquisition to a single prediction problem. The challenge—and what this paper addresses—is that naively predicting pixels leads to models that memorize visual appearance rather than learning transferable dynamics.

---

The Basic Video Generation Framework and Why It Fails

The paper adopts mainstream video generation models (VQ-VAE + generative model) as the basic framework and then explains why they are inadequate for knowledge learning. This section establishes the baseline that the original VideoWorld attempted to improve.

Standard video generation models operate as follows:

1. A VQ-VAE encodes each video into a compressed latent representation. The encoder maps raw frames to a sequence of feature vectors, which are then quantized by assigning each to the nearest entry in a learned codebook. The decoder reconstructs frames from these quantized codes.

2. A generative model (typically a transformer or diffusion model) operates in this compressed latent space, learning to predict future latent codes from past ones.

3. At inference, the generative model predicts future latent codes, which the VQ-VAE decoder converts back to RGB frames.

The problem: these latent representations require "thousands of discrete tokens or continuous embeddings to capture the full spectrum of visual information." This leads to two specific pathologies:

• Spatiotemporal redundancy: The same background pixels, static objects, and slowly-changing visual features are encoded repeatedly across frames, consuming representation capacity that could be used for action-relevant information.
• Sparse distribution of knowledge: The critical information about task-relevant actions—hand movements, object deformations, spatial relationships—is distributed across many tokens, interleaved with tokens encoding static visual detail. The model has no structural incentive to isolate the action information from the appearance information.

As the paper states, "such representations result in an inefficient encoding of the visual changes and motion dynamics tied to critical decisions and actions, thereby hindering the framework from acquiring essential task knowledge from raw videos." The model learns to reproduce pixels but does not learn the underlying procedural knowledge because it is not forced to compress in a way that separates what changes (actions) from what stays the same (appearance).

---

The Original Latent Dynamic Model (LDM) from VideoWorld

VideoWorld (the prior work) introduced the Latent Dynamic Model (LDM) to address the inefficiency of standard VQ-VAE representations. The LDM's key insight is to compress only the changes between frames—not the full frame content—into a small set of latent codes. The architecture is a causal VQ-VAE with a specialized query-based encoder.

Encoder. The LDM uses a MAGVITv2-style causal codec. The encoder first maps an input clip $x$ of length $T$ to a feature sequence $f_{0:K}$, where:

$K = 1 + \left\lfloor \frac{T-1}{s} \right\rfloor$

and $s$ is the temporal downsampling stride. The index $k=0$ corresponds to the first frame, and $k=1$ to $K$ correspond to temporal positions in the feature sequence after downsampling.

What it computes: Given a video clip of $T$ frames, this produces $K+1$ feature vectors spaced by stride $s$ through time. For example, with $T=93$ frames and a stride of $s=4$, this yields $K = 1 + \lfloor 92/4 \rfloor = 24$ feature positions.

Why this form: The causal constraint means that features at position $k$ can only depend on frames up to time $k \cdot s$, preventing information leakage from future frames. This is essential because the model will eventually be used autoregressively—during inference, future frames are not available, so the encoder must not rely on them during training.

Query-based change extraction. The key innovation is a set of $N$ learnable query embeddings:

$q = \{q^n\}_{n=1}^N$

These queries use cross-attention to attend to the feature sequence $\{f_{0:k}\}_{k=1}^K$—that is, all features from time 0 to time $k$—and extract a continuous representation:

$z = \{z_k^n\}_{k=1,n=1}^{K,N}$

What it computes: For each temporal position $k$ and each query $n$, the cross-attention mechanism computes a weighted combination of the feature vectors $f_{0:k}$, producing a single vector $z_k^n$. Intuitively, each query learns to "look for" a specific type of visual change—one query might attend to hand position, another to object orientation, another to relative spatial configuration. The set of $N$ queries together captures the full visual change from time 0 to time $k$.

Why this form: The cross-attention mechanism allows the queries to actively select which spatiotemporal features to incorporate, rather than passively compressing everything. The first-frame feature $f_0$ is included in the attention context, which grounds the representation—the queries encode changes relative to the initial state. This is fundamentally different from a standard VQ-VAE that encodes absolute frame content.

Quantization. The continuous representation $z$ is quantized to prevent the LDM from learning shortcuts, "e.g., trivially copying $f_k$ to $z_k$." The paper uses Finite Scalar Quantization (FSQ) with a default vocabulary size of 1000 (FSQ levels [8, 5, 5, 5]). Quantization forces the latent space to be discrete and bounded, preventing the encoder from encoding low-level pixel information that the decoder could trivially copy.

Decoder. The decoder takes two inputs: the first-frame feature $f_0$ and the quantized latent codes $z$. It reconstructs the subsequent frames in a causal manner—the reconstructed frame at time $t$ only depends on $f_0$ and codes up to time $t$.

Training objective. The LDM is trained to minimize the $\ell_2$ distance between original and reconstructed frames:

$\mathcal{L}_{\text{LDM}} = \|x_{\text{reconstructed}} - x_{\text{original}}\|_2^2$

What it computes: The mean squared error between every pixel in every reconstructed frame and its corresponding pixel in the original video. This is a per-pixel reconstruction loss that penalizes any deviation from the ground truth.

Why this form: $\ell_2$ is standard for image reconstruction because it penalizes large errors more than small ones, which tends to produce less blurry reconstructions than $\ell_1$. However, this is also the LDM's fundamental limitation: $\ell_2$ pixel loss forces the latent codes and decoder to jointly model all visual information—both task-relevant dynamics and task-irrelevant appearance—because any unreconstructed pixel contributes to the loss. The codes are incentivized to capture texture, lighting, and background details alongside action information, making them environment-specific.

The failure mode. When the LDM is applied to real-world videos (rather than synthetic ones), the reconstructed outputs exhibit significant scene drift and erroneous actions when evaluated in new environments. The paper conjectures that the latent codes "capture irrelevant visual details—such as background motion, lighting changes, textures, and camera displacement—making the model sensitive to environmental changes." In a new environment with different backgrounds or materials, the codes encoding those old appearance features produce outputs that are inconsistent with the new context, causing the generation to drift and fail.

---

The Dynamics-enhanced Latent Dynamic Model (dLDM)—Core Innovation

The dLDM is the paper's central contribution. It modifies the original LDM by replacing the standard decoder with a pretrained Video Diffusion Model (VDM) and introducing specific mechanisms that force the latent codes to focus on dynamics rather than appearance.

Core insight: offloading appearance to the VDM. The VDM (Cosmos DiT 2B) has been pretrained on vast amounts of diverse video data and can generate high-fidelity visual content when provided with appropriate guidance. It does not contain any knowledge of the target task's specific dynamics, but it is exceptionally good at rendering realistic visual details—textures, lighting, backgrounds, object surfaces—once told what should move where. The dLDM exploits this asymmetry: by making the VDM responsible for visual quality, the latent codes can be trained with an objective that penalizes them only for failing to capture information that the VDM cannot produce on its own—the task-relevant dynamics.

Architecture. The dLDM consists of three sub-components, shown in the right panel of Figure 4:

1. Causal VQ-VAE (the LDM encoder + quantizer): Same as the original LDM. Encodes future visual changes into discrete latent codes $z$ using learnable queries and FSQ quantization.

2. Lightweight decoder (VQ-VAE decoder): Same decoder as the original LDM, but repurposed. Instead of producing the final output, it reconstructs low-fidelity, motion-rich frames from the latent codes. These frames are blurry and lack visual detail but preserve coherent object motion—hand positions, object displacements, spatial transformations. The decoder's output is gradient-stopped before being fed forward.

3. Pretrained Video Diffusion Model (VDM): Cosmos DiT 2B, an image-to-video diffusion model. It generates the final high-fidelity future frames using three conditioning signals: the first frame (providing the initial visual context), the coarse decoder output (providing motion guidance via a ControlNet-like branch), and the latent dynamics codes (injected via causal cross-attention, providing the semantic action representation).

Conditioning Pathway 1: Latent codes via causal cross-attention. The latent dynamics codes are fed to the VDM through a projection layer (MLP followed by causal self-attention) and then injected into the VDM's denoising network via causal cross-attention layers. The causal constraint is critical: at denoising timestep $t$, the VDM can only attend to latent codes up to time $t$, preventing information from future codes leaking into earlier frames. This is enforced by the causal attention mask.

The projection layer ensures the latent codes are transformed into a representation compatible with the VDM's internal feature space. The ablation study (Table 3c) shows that both the MLP + self-attention projection and the causal cross-attention are necessary: removing self-attention or using non-causal cross-attention significantly degrades performance.

Conditioning Pathway 2: Coarse motion guidance via ControlNet-like branch. The paper states:

"Directly training the VDM to generate future frames from noise would be extremely slow and prone to incorrect motion, as it has never been trained on the target tasks like long-horizon handicraft making."

To address this, the dLDM reuses the VQ-VAE decoder's output as a coarse motion prior. The decoder reconstructs the latent codes into low-fidelity frames that preserve hand movements and object displacements but lack visual detail. This signal is fed into the VDM via a gradient-stopped, ControlNet-like branch—a parallel network that extracts features from the coarse frames and injects them into the VDM's denoising network.

The gradient-stopping on the decoder output is essential: it prevents gradients from the VDM's denoising loss from flowing back through the decoder to the latent codes. If gradients were allowed to flow, the latent codes would be pressured to encode fine-grained visual details that help the VDM's denoising but are task-irrelevant—exactly the entanglement the paper aims to prevent. The ablation study (Table 3a, rows 3 vs. 4-5) confirms this: removing the gradient stop reduces success rate as the decoder introduces "extraneous noise that degrades latent representation performance."

Conditioning Pathway 3: First frame as initial visual context. The first frame of the clip is provided directly to the VDM as the conditioning image for image-to-video generation. This grounds all generated frames in the correct visual environment—the background, lighting, object colors, and material textures of the test scenario. Since the first frame comes from the test environment, the VDM can render the remaining frames in a visually consistent way, even though the latent codes were learned from a different training environment with different visual properties. This is the mechanism that enables cross-environment generalization: the latent codes specify what actions happen, the first frame provides what the environment looks like, and the VDM combines them.

Training the dLDM. The training process proceeds in two stages:

Stage 1: Warm-up with the original decoder. Before introducing the VDM, the dLDM is trained using only the original $\ell_2$ reconstruction objective on the VQ-VAE decoder—identical to the original LDM training. This enables the latent codes to "rapidly learn to compress visual changes and motion dynamics, while allowing the decoder to reconstruct low-fidelity video clips containing agent motion trajectories based on the initial frame and codes."

This warm-up is critical because the VDM has never seen handicraft or manipulation videos during its pretraining. If the dLDM were trained end-to-end with the VDM from scratch, the latent codes and VDM would need to simultaneously learn dynamics representation and adapt to a new visual domain—a difficult joint optimization. The warm-up disentangles these: the codes first learn to capture motion with a simple decoder, then the VDM learns to render those motions with high visual quality.

Stage 2: Disentangled training with the VDM. After warm-up, the training switches to the full dLDM scheme. The VQ-VAE decoder is kept (for producing coarse motion conditioning) but its reconstruction loss is discarded—the gradient from the VDM's denoising loss is the only signal for training the latent codes. The VDM is fine-tuned (not frozen—see Table 3f for the ablation comparing freeze, LoRA, and full fine-tuning) to adapt its appearance priors to the specific visual domain of the training data.

The total training loss in this stage is the VDM's standard denoising loss (typically a noise prediction or velocity prediction objective in latent space) plus potentially the VQ-VAE's reconstruction loss during warm-up. The key design choice is what gets optimized by each loss: the VDM parameters are optimized by the denoising loss; the VQ-VAE encoder, queries, and quantizer are optimized by the gradient flowing through the causal cross-attention pathway (but NOT through the ControlNet branch due to gradient stopping).

Why this architectural decomposition works. The dLDM creates an information bottleneck with a specific shape. The latent codes are the only pathway through which task-relevant dynamics information can reach the VDM (the coarse decoder output provides motion guidance but is gradient-stopped, so codes cannot be optimized through it). The VDM, having strong appearance priors from pretraining, can generate high-quality visual output without needing the latent codes to specify visual details. Therefore, the latent codes face gradient pressure to capture precisely the information that the VDM cannot produce from the first frame and coarse motion cues alone: the specific task-relevant action semantics.

The result is that latent codes learned in one environment (with particular backgrounds, materials, and lighting) are valid in new environments because they do not encode those appearance features—they encode only the abstract dynamics like "fold the top-left corner to the center." When combined with a first frame from the new environment, the VDM renders these dynamics in the new visual context.

---

Autoregressive Transformer for Policy Learning

After the dLDM is trained, it serves as a tokenizer: every training video is processed through the dLDM encoder to extract a sequence of discrete latent dynamics codes. These codes become the prediction targets for an autoregressive transformer, which learns the policy over the latent action space.

Training data construction. For each video $x_{0:T}$, the dLDM extracts latent codes $\{z_k^n\}_{k=1,n=1}^{K,N}$. These are flattened into a single temporal sequence. The spatial dimension $n$ (the query index) and the temporal dimension $k$ are interleaved, producing a sequence of $K \times N$ tokens per video clip. For the default configuration with 93-frame clips, a stride $s$, and $N=4$ queries, this produces a manageable sequence length that the AR transformer can model.

Transformer architecture and training. The paper uses the NVIDIA Cosmos AR 4B model as the autoregressive transformer, "repurposing its next-token prediction capability to predict our latent codes." The transformer is trained with a standard causal language modeling objective—predict the next latent code given all previous codes, conditioned on the initial frame $x_0$ and the task instruction (a text description of the task being performed).

The training objective is standard cross-entropy loss over the discrete latent code vocabulary:

$\mathcal{L}_{\text{AR}} = -\sum_{t} \log p(z_t | z_{<t}, x_0, \text{instruction})$

where $z_t$ is the ground-truth latent code at position $t$, $z_{<t}$ are all previous codes, $x_0$ is the first frame, and the instruction is the task description.

What it computes: At each position, the transformer's predicted probability distribution over the codebook vocabulary is compared to the true code (a one-hot target, since codes are discrete after quantization). The log probability of the correct code is maximized. This trains the transformer to reproduce the sequence of latent dynamics codes that the dLDM extracted from training videos.

Why this form: Autoregressive training on discrete tokens is the standard approach for sequence modeling and has proven effective for learning long-range dependencies. The conditioning on $x_0$ and the task instruction ensures the transformer learns a policy—given an initial observation and a task description, what sequence of abstract actions (latent codes) leads to task completion. The transformer does not need to model visual appearance because that is handled by the VDM during decoding; it only models the abstract action sequence.

Inference in new environments. At test time (Figure 3, right panel), the process is:

1. A single input frame $x_0$ is captured from the new, unseen environment.
2. The AR transformer autoregressively generates a sequence of latent dynamics codes $z_{1:K, 1:N}$, conditioned on $x_0$ and the task instruction. The generation uses standard autoregressive sampling (likely with temperature sampling or nucleus sampling, though the paper does not specify the exact decoding strategy).
3. The dLDM decoder (specifically, the VDM with its conditioning pathways) converts the predicted latent codes and the initial frame $x_0$ into a sequence of high-fidelity video frames showing the task execution in the new environment.

Because the latent codes do not encode environment-specific visual details, the same predicted code sequence can be decoded by the VDM into visually different outputs depending on $x_0$. A code representing "fold corner to center" produces a different visual result when decoded with a red paper first frame versus a blue paper first frame, but the structural transformation is identical. This is the mechanism of transferability.

The dual role of the AR transformer. As described in the formal problem statement, $\rho$ serves as both world model and policy. During inference, the AR transformer functions as $\pi(\cdot | x_0)$: mapping the initial observation to an action sequence. The action sequence is represented in the latent code space rather than in any human-interpretable action format, making this an implicit, learned policy rather than an explicit, symbolic one. The actions become "executable" only when decoded by the VDM into visual frames—the system's output is a demonstration video, not a sequence of joint angles or gripper commands.

---

Training Procedure, Configurations, and Hyperparameters

The paper provides detailed training configurations in Table 4. Here is a complete accounting of the training setup:

dLDM training configuration:

• Optimizer: AdamW
• Base learning rate: $1 \times 10^{-4}$
• Weight decay: 0.1
• Optimizer momentum: $\beta_1 = 0.9$, $\beta_2 = 0.99$
• Batch size: 128
• Learning rate schedule: WarmupDecayLR
• Warmup iterations: $2 \times 10^3$
• Maximum iterations: $1 \times 10^5$
• Augmentations: None
• Training loss: L2 (for warm-up) + Denoising loss (for VDM stage)
• Training target: Reconstruction (warm-up) + Denoising (VDM stage)

AR transformer training configuration:

• Optimizer: AdamW
• Base learning rate: $3 \times 10^{-4}$
• Weight decay: 0.05
• Optimizer momentum: $\beta_1 = 0.9$, $\beta_2 = 0.98$
• Batch size: 256
• Learning rate schedule: WarmupDecayLR
• Warmup iterations: $5 \times 10^3$
• Maximum iterations: $5 \times 10^4$
• Augmentations: None
• Training loss: Cross-entropy loss
• Training target: Next token prediction

dLDM architectural hyperparameters:

• Default clip length $T = 93$ frames (approximately 5 seconds at 16 fps, matching the Cosmos DiT 2B's maximum context length)
• Default query embedding length $N = 4$
• Codebook size: 1000 (via FSQ levels [8, 5, 5, 5])
• VDM: Cosmos DiT 2B (image-to-video diffusion model, 2 billion parameters)
• AR Transformer: Cosmos AR 4B (autoregressive transformer, 4 billion parameters)

Training strategy and the warm-up phase. The paper describes a two-stage training strategy for the dLDM:

1. Warm-up phase: The dLDM encoder, queries, quantizer, and VQ-VAE decoder are trained jointly with the $\ell_2$ reconstruction loss. The VDM is not involved. This phase enables the latent codes to learn to capture motion dynamics in a setting where the reconstruction objective directly supervises them. The length of this warm-up is not specified numerically in the main text, but the paper states it is "similar to the training strategy of the original VideoWorld LDM."

2. Full dLDM training phase: The VDM is introduced. The decoder's reconstruction loss is discarded (its output is used only as motion conditioning for the VDM via the ControlNet-like branch, with gradients stopped). The latent codes, projection layer, and VDM are trained with the denoising loss. The VQ-VAE decoder may continue to be updated or may be frozen—the paper implies it continues to be used but gradient-stopping prevents its output from affecting the latent codes.

Table 3f ablates different VDM training strategies:

• Random initialization (row 1): "leads to model collapse and fails to generate valid videos." The VDM must be pretrained to provide useful appearance priors.
• Freeze pretrained VDM (row 3): Only VQ-VAE encoder, quantizer, and projection layer are updated. Yields 31.7% paper folding success rate—better than random but far below fine-tuning.
• LoRA fine-tuning (row 4): Low-rank adaptation of the VDM. Yields 50.9% success rate—substantially better than freezing but still below full fine-tuning.
• Full fine-tuning (row 5): The default approach. Yields 68.8% success rate. The authors note this gap exists because "the VDM needs further adaptation to capture the fine-grained manipulation details specific to Video-CraftBench."

Ablation: query embedding length (Table 3b). The number of query embeddings $N$ controls the capacity of the latent code representation. With $N=1$, a single code vector captures all visual changes—achieving 41.9% paper folding success, demonstrating that even a very compact representation can capture useful dynamics. $N=4$ provides the best balance, with 68.8% success. $N=8$ provides slightly better LPIPS (0.195 vs. 0.205) but slightly lower success rate (65.0%), suggesting the extra capacity encodes some noise that interferes with the AR transformer's policy learning.

Ablation: horizon length (Table 3e). The dLDM processes clips of length $T$. At $T=2$ (a CoLA-like setting with only pairwise transitions), performance on the long-horizon paper folding task is poor (19.1% success) due to lack of temporal context. Performance improves as $T$ increases, plateauing at $T=93$, which is the maximum context length of the Cosmos DiT 2B VDM. This ablation demonstrates that modeling multi-step temporal dynamics—rather than pairwise frame transitions—is critical for long-horizon tasks.

Ablation: codebook size (Table 3d). The codebook size (adjusted via FSQ levels) affects the granularity of the latent code discretization. The simpler CALVIN action space benefits from even a very small codebook (size 8 yields reasonable performance), while the complex Video-CraftBench tasks require a larger codebook (size 1000) to achieve significant performance. However, an excessively large codebook (64,000) degrades performance (29.4% success), as it "risks encoding extraneous noise and can hinder dLDM training convergence."

Ablation: VDM-LDM interplay (Table 3c). The interplay mechanism between the LDM and VDM consists of the projection layer (MLP + optional causal self-attention) and the cross-attention type (causal vs. non-causal). Using MLP-only projection with non-causal cross-attention yields 52.0% paper folding success. Adding self-attention to the projection yields minimal improvement (52.3%). Switching to causal cross-attention yields a large improvement (69.8%), and additionally adding self-attention to the projection yields 72.3%. Causal cross-attention is the dominant factor in improving VDM-LDM interaction, preventing the VDM from "cheating" by looking at future latent codes when denoising earlier frames.

4. Key Insights and Innovations

Innovation 1: Appearance-Dynamics Disentanglement as THE Bottleneck for Real-World Video Knowledge Learning

The paper's most fundamental conceptual contribution is its diagnosis of why prior approaches fail on real-world videos, not just the solution. Before this work, the field's understanding of the gap between synthetic and real-world video knowledge learning was fragmented. VideoWorld showed that latent dynamics models work on synthetic domains. Video generation models produced photorealistic outputs but couldn't execute tasks. Latent action models succeeded on short-horizon robotic manipulation but collapsed on longer sequences. Each approach had its own failure narrative, with no unified explanation.

This paper identifies a single root cause: the entanglement of action dynamics and visual appearance in learned representations. The argument is not merely that appearance is a nuisance variable—it is that the very objective functions used to train these models (pixel-level reconstruction losses, denoising objectives) force the learned representations to encode all visual information equally, task-relevant and task-irrelevant alike. The key diagnostic evidence is the contrast between VideoWorld's original LDM and VideoWorld 2's dLDM trained on identical data: the LDM's latent codes, visualized via UMAP in Figure 7 (right panel), show the same action (e.g., "robotic arm moving right") producing different code clusters depending on which environment (Bridge vs. CALVIN) the video came from. The dLDM's codes (Figure 7, left panel) cluster by action semantics, not environment. This is direct evidence that the original LDM's codes captured environment-specific appearance, not transferable dynamics.

What makes this diagnosis intellectually distinctive is its falsifiability and specificity. The paper does not claim that "real-world videos are harder" in some vague sense. It claims a specific mechanism: the reconstruction objective forces latent codes to encode appearance details; these details are environment-specific; therefore the learned codes fail to transfer. If this mechanism is correct, then removing the reconstruction pressure for appearance should solve the problem. That is exactly what the dLDM does by delegating appearance to the VDM, and the performance jump (from VideoWorld's 31.9% final-step success to VideoWorld 2's 72.3% with OpenX pretraining—Table 1) confirms the diagnosis. This is a cleaner scientific result than simply proposing a new architecture and showing it works better: it identifies the causal factor behind prior failures and tests the intervention directly.

The significance extends beyond video knowledge learning. This diagnosis reframes the general problem of learning transferable representations from high-dimensional sensory data as fundamentally a credit assignment problem in representation space: which variation in the input should the representation encode, and which should it abstract away? The paper's answer—use a separately trained, high-capacity model to absorb the unwanted variation—is a general design principle that could apply to other domains where task-relevant and task-irrelevant variation are entangled in raw observations.

Innovation 2: The Pretrained Video Diffusion Model as Appearance Oracle—Repurposing, Not Just Using, Foundation Models

The second conceptual innovation is the inversion of the standard relationship between dynamics models and video diffusion models. The dominant paradigm in both video generation and world modeling treats the video diffusion model as the primary generator, with any conditioning (text, actions, latent codes) serving as auxiliary inputs that guide generation. In approaches like Cosmos, Sora, or UniSim, the VDM is the central intelligence—it must understand the world well enough to produce physically plausible, action-consistent videos from conditioning signals.

VideoWorld 2 inverts this relationship. The VDM is explicitly positioned as an appearance oracle whose ignorance of the target task's dynamics is a feature, not a bug. The paper states: "Although the VDM does not contain any knowledge of the target task's dynamics, it is highly effective at producing realistic visual content once given appropriate dynamics guidance." This reframes the VDM from a world model into a rendering engine—a learned graphics pipeline that can visualize abstract action specifications in arbitrary visual contexts.

This repurposing is fundamentally different from how VDMs are used in prior work:

• In end-to-end video generation (Cosmos, Wan, HunyuanVideo), the VDM is the entire pipeline, and any dynamics learning must happen implicitly within its weights. The paper shows this fails catastrophically on long-horizon tasks: these models succeed on step 1 but collapse by step 4.
• In CoLA, the closest concurrent approach, a VDM is used but is "limited to short 2-frame transitions and ignores the structured temporal cues from coarse VAE outputs." The VDM serves as a decoder, but without the specific architectural mechanisms (multi-frame context, coarse motion conditioning, causal cross-attention) that make it an effective oracle rather than a bottleneck.
• In works like \epsilon-VAE and DiVoT, VDMs perform pixel-level reconstruction, meaning the latent codes must still encode appearance information because the VDM's denoising objective depends on it.

The dLDM's specific design choices—gradient-stopping the coarse decoder output, using causal cross-attention for latent code injection, providing multi-frame motion conditioning via ControlNet—are all implementations of this conceptual inversion. They ensure the VDM cannot "cheat" by extracting dynamics information from sources other than the latent codes, and cannot leak gradient signal through pathways that would pressure the codes to encode appearance. The ablation in Table 3a confirms this: removing the gradient stop reduces success rate by ~20%, and removing the motion conditioning reduces it further by ~20%, because both changes weaken the appearance-dynamics separation that the inversion requires.

This is a fundamental shift in how to think about foundation model reuse. Rather than treating pretrained models as initialization for downstream fine-tuning, VideoWorld 2 treats them as differentiable, domain-general decoders that can be conditionally queried during end-to-end training of task-specific encoders. This is a more modular and potentially more scalable paradigm: as pretrained VDMs improve in visual quality and diversity, the dLDM's task-relevant dynamics should automatically benefit without architectural changes.

Innovation 3: Multi-Step Temporal Modeling as Critical for Long-Horizon Knowledge—Not Just an Implementation Detail

The paper establishes that modeling multi-step temporal dynamics (rather than pairwise frame transitions) is not merely helpful but essential for long-horizon knowledge learning. This is demonstrated most directly in the horizon length ablation (Table 3e), but its conceptual significance goes beyond the raw numbers.

Prior latent action models—LAPA, Moto, AdaWorld, CoLA—operate on short temporal contexts, typically single transitions between consecutive frames or 2-frame windows. This is a natural design choice when the downstream task involves reactive control: given the current observation, predict the next action. For tasks like grasping or toggling, where each action is relatively independent of the long-term plan, short-horizon modeling suffices.

VideoWorld 2 shows that for complex, multi-step tasks like paper folding, this short-horizon assumption is catastrophic. At $T=2$ (the CoLA-like setting), paper folding success is 19.1% despite the dLDM's other architectural advantages. The performance curve (Table 3e) rises monotonically as horizon length increases—$T=9$: 55.4%, $T=49$: 65.3%, $T=93$: 68.8%—demonstrating that the latent codes benefit from seeing longer temporal contexts to disambiguate visually similar but semantically distinct actions. A hand moving toward a piece of paper could indicate the beginning of a fold or a repositioning gesture; only by observing the consequences over multiple future frames can the latent code distinguish these.

The conceptual insight is that action semantics are inherently multi-step. What makes an action "folding the top-left corner to the center" rather than "moving the hand to the top-left" is not the instantaneous motion but its relationship to subsequent states—the paper's shape after the motion completes, the subsequent crease, the next action in the sequence. A representation that only sees one step into the future cannot capture these relational semantics; it can only encode the instantaneous visual change, which is inherently appearance-biased and environment-specific.

This has implications for the broader world model literature, which has largely adopted sequential (one-step) or short-horizon prediction as a design paradigm. The paper's evidence suggests that for tasks with long-range dependencies and visually ambiguous actions, this paradigm is fundamentally inadequate. The multi-step latent code extraction mechanism—where query embeddings attend to the full feature sequence $f_{0:k}$ rather than just adjacent frames—is the specific architectural realization of this insight, but the conceptual point is broader: temporal abstraction may be necessary for learning transferable action representations from video.

Innovation 4: Video-CraftBench as a Stress Test That Exposes Systematic Failure, Not Just Lower Performance

While benchmarks are typically classified as infrastructure rather than scientific contributions, Video-CraftBench provides a diagnostic contribution that the paper leverages to make its core argument. The key insight from this benchmark is not that prior methods perform poorly—it is where and how they fail.

The sequential success rate metric reveals a specific pattern (Table 1): pretrained video generation models (Cosmos AR, DiT, Hunyuan, Wan) achieve high step-1 success (68–82%) but then collapse almost deterministically—by step 4, success rates are ≤10.6%, and by step 5, they are uniformly 0%. This is not a gradual degradation suggesting a capacity limitation; it is a cliff suggesting a qualitative failure mode. The models can reproduce the first action because it is visually distinctive (picking up and folding the paper in half), but cannot chain actions because each subsequent step depends on the precise outcome of prior steps, which the models do not faithfully reproduce.

Similarly, latent action models (Moto, AdaWorld, VideoWorld) show a different failure pattern: they decay more gradually but fail to complete the full sequence, with VideoWorld reaching 6.7% at step 5 (paper folding) and 0% at step 7. The associated visual artifacts (Figure 8)—distorted hands, incorrect object shapes, scene drift—indicate that the models are attempting to execute the right actions but are doing so incorrectly due to appearance entanglement.

The benchmark thus provides differential diagnosis: it distinguishes between models that fail because they cannot plan (the VDM baselines) and models that fail because they cannot generalize their dynamics (the latent action baselines). This distinction is lost in aggregate metrics like average success rate or final-state accuracy, which would obscure the qualitative nature of the failures. For the block-building tasks, which are shorter and visually simpler, the failure pattern is different still—some baselines achieve non-trivial success rates (Wan reaches 39.7% on the block "Human" task), showing that the benchmark's difficulty gradient is calibrated to surface capability boundaries rather than being uniformly impossible.

The conceptual contribution is not the benchmark itself but the evaluation methodology: evaluating long-horizon sequential tasks where each step is gated on all preceding steps provides a more informative signal than independent step evaluation or end-state checking. This methodology could generalize to other domains where procedural knowledge acquisition is the target—surgical procedure learning, assembly instruction following, cooking recipe execution—and provides a template for designing evaluations that surface failure modes rather than just failure rates.

Innovation 5: Implicit Policy Learning Through Latent Code Prediction as an Alternative to Explicit Action Labeling

The final innovation is a methodological insight about representation learning for control: that autoregressive prediction of latent dynamics codes—learned without action labels—can serve as an effective pretraining objective for policy learning, outperforming direct video prediction pretraining.

The CALVIN experiments (Table 2) demonstrate this in two settings. In the in-domain setting (pretraining on CALVIN latent codes, fine-tuning on 2k action labels), VideoWorld 2 achieves 75.8% success on the first sequential task and 1.87 average completed tasks, compared to 50.5% and 1.11 for the baseline trained from scratch on the same 2k labels. This is a ~60% improvement in data efficiency for policy learning, achieved not by a better architecture or optimization procedure but by a better pretraining target—latent dynamics codes rather than ground-truth actions.

The cross-domain setting is even more striking. Pretraining on OpenX latent codes and fine-tuning on CALVIN labels yields 88.5% first-task success and 2.88 average length, surpassing both the oracle trained on 22k CALVIN action labels (80.9%, 2.36) and the video next-token prediction baseline pretrained on OpenX videos (85.9%, 2.46). This means that latent dynamics codes learned from a different embodiment, different tasks, and different visual domain transfer more effectively than either ground-truth actions from the target domain or raw video from the source domain.

The conceptual significance is that this latent code space functions as an embodiment-agnostic action representation. It abstracts away the specifics of joint angles, gripper positions, and camera viewpoints while preserving the semantic content of manipulation actions—moving toward, grasping, lifting, placing. This is a form of unsupervised domain transfer that does not require paired data, shared action spaces, or explicit mappings between embodiments. The latent codes learned from OpenX (a diverse collection of robot demonstrations) capture manipulation primitives that are relevant to CALVIN (a specific simulated environment) because the dLDM's training objective forces codes to represent what changes in a task-relevant way, not what any particular robot's motors are doing.

This reframes the problem of cross-embodiment transfer from one of aligning action spaces (requiring explicit labels or paired trajectories) to one of learning observation-prediction representations that are, by construction, embodiment-invariant. If a folding action involves the same visual transformation regardless of whether it is performed by a Franka arm or a human hand, the dLDM's latent codes will represent it similarly because the visual change is what the codes encode. This is a fundamental shift in perspective with implications for how large-scale robot learning datasets should be constructed and pretrained.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary benchmark is Video-CraftBench, a newly introduced dataset of first-person tutorial videos covering five handicraft tasks: folding a paper airplane, folding a paper boat, building a tower/horse/person using blocks. The dataset contains approximately 7 hours of video split into roughly 9.5k clips, with paper folding tasks lasting 40–80 seconds and block-building tasks lasting 20–30 seconds. The test set comprises approximately 150 separately collected videos with novel backgrounds, paper textures, and block arrangements not seen in training. For robotics, the paper uses the Open-X dataset for training (1.3M demonstration trajectories from diverse robots and environments) and evaluates on the CALVIN benchmark, which features 34 tasks with a long-horizon sequential evaluation protocol requiring models to complete 5-task sequences where success on each task is contingent on all preceding tasks. The CALVIN training set (labeled data) contains 22k trajectories across four task splits (ABCD→D).

• Base model(s). The dLDM uses Cosmos DiT 2B as the pretrained video diffusion model and Cosmos AR 4B as the autoregressive transformer. The Cosmos DiT 2B is a 2-billion-parameter image-to-video diffusion model with strong appearance priors from large-scale pretraining; it processes 93-frame clips at 480px resolution. The Cosmos AR 4B is a 4-billion-parameter autoregressive transformer repurposed to predict latent dynamics codes. The VQ-VAE encoder/decoder components are trained from scratch on the target video data. For baselines, the paper also fine-tunes Wan2.2 14B, HunyuanVideo 13B, and additional Cosmos variants.

• Metrics. The primary metric on Video-CraftBench is sequential task success rate, which measures the fraction of test cases where all key steps of a task are completed in order. For paper folding, 7 key steps are defined (Figure 6), and a step is successful only if all preceding steps are complete—this gated evaluation captures long-horizon knowledge acquisition rather than isolated action execution. A DINOv2-based classifier is trained to detect step completion based on paper shapes and block arrangements, disregarding appearance consistency (texture drift, background changes). The classifier is trained on ~25k frames (15k from training/testing environments, 10k from manually verified generated videos) and achieves 96.1% test accuracy. For block building, a similar classifier verifies only the final generated state. Video quality is measured using LPIPS (lower is better, measuring perceptual similarity) and SSIM (higher is better, measuring structural similarity). On CALVIN, the metric is sequential task success rate across 5 tasks, with the average number of completed tasks per sequence reported alongside per-step success rates.

• Baselines. The paper establishes three categories of baselines:

  • Pre-trained video generation models (4): Cosmos AR 4B, Cosmos DiT 2B, Wan2.2 14B, HunyuanVideo 13B — all fine-tuned on Video-CraftBench with step-by-step textual descriptions provided by Qwen2.5-VL 72B. During inference, these generate videos autoregressively based on their own previous output.
  • Latent action models (5): LAPA (Ye et al., 2024), Moto (Chen et al., 2025b), AdaWorld (Gao et al., 2025), iVideoGPT (Wu et al., 2024), CoLA (Wang et al., 2025a) — methods that compress inter-frame visual changes into latent codes. Their latent extraction methodologies are applied to Video-CraftBench, and an AR transformer is trained to predict their codes. LAPA's decoding into long-horizon sequences causes severe degradation (marked "N.A.").
  • VideoWorld (Ren et al., 2025): The most relevant baseline, using the original latent dynamic model without VDM enhancement.
  • For CALVIN, an additional video next-token prediction baseline pretrained on OpenX videos (not latent codes) is included to evaluate the efficiency of latent versus raw-video pretraining.

• Generation budget / compute accounting. The paper evaluates based on task success and video quality rather than compute FLOPs. The key computational constraint for fair comparison is that all methods have access to the same training data (Video-CraftBench only, or Video-CraftBench + OpenX) and are evaluated under identical test conditions. Latent code baselines use the same AR transformer architecture for code prediction. The dLDM processes 93-frame clips at a time (matching the Cosmos VDM's maximum context), and long videos are generated auto-regressively by extending each segment from the final frame of the previous one. For all methods, 3 video rollouts are generated per test case, and the best is evaluated. The paper does not standardize inference-time compute across methods—the comparison focuses on whether methods can complete tasks at all, not on achieving a particular accuracy-per-FLOP ratio.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. The test set is a fixed set of ~150 separately collected videos for Video-CraftBench and the standard CALVIN evaluation protocol for robotics. The sequential success rate metric is computed per-step across the full test set, with results presented as percentages in Tables 1, 2, and 3. For the CALVIN "10% data" experiments, 2k trajectories are randomly sampled from the ABCD→D training set, but no multiple-seed averaging is reported. This is a notable methodological limitation: the sequential success rate metric aggregates binary outcomes across a fixed set of test cases and is subject to variance, but no estimate of that variance is provided.

Main Quantitative Results

Video-CraftBench: Training on CraftBench Only

The core finding is shown in Table 1 (rows 1–9). VideoWorld 2, trained solely on Video-CraftBench without any external data, achieves 68.8% success rate on the full 7-step paper folding task and up to 81.5% on block stacking. In contrast, all baselines fail to complete the full paper folding sequence. The headline comparisons are:

• Pre-trained video generation models (rows 1–4) achieve strong step-1 performance—68.4% (Cosmos AR), 73.4% (Cosmos DiT), 76.9% (Hunyuan), 81.2% (Wan 2.2)—but degrade rapidly. By step 4, success rates are ≤10.6% for all four models. By step 5 (and all subsequent steps), success rates are uniformly 0.0%. No pretrained video generation model generates a video that completes step 5 or beyond of paper folding under any test case. On block stacking, these models achieve non-trivial but modest success: 10.1–39.7% on the "Human" task, 18.0–42.6% on "Tower," 12.0–34.1% on "Horse."

• Latent action models (rows 5–8) show a different failure pattern. Moto achieves 19.1% step-1 success, decaying to 0.0% by step 4. AdaWorld reaches 43.6% step-1 and 10.8% step-4, but 0.0% by step 5. VideoWorld, the strongest latent-dynamics baseline, achieves 70.3% step-1 and maintains non-trivial success through step 5 (6.7%), but reaches 0.0% at step 7—it cannot complete the full task. On block stacking, VideoWorld achieves 23.8% (Human), 33.9% (Tower), 27.8% (Horse). LAPA is marked "N.A." for sequential evaluation due to structural constraints causing severe degradation in long-horizon decoding.

• VideoWorld 2 (row 9), by contrast, maintains high success throughout: 97.2% step-1, 95.3% step-2, 90.0% step-3, 83.3% step-4, 81.4% step-5, 74.6% step-6, 68.8% step-7. On block stacking, it achieves 70.0% (Human), 81.5% (Tower), 80.9% (Horse). The visual quality metrics are also superior: SSIM 0.770 vs. 0.585–0.719 for baselines, LPIPS 0.205 vs. 0.237–0.394. This represents a ~70% relative improvement in final-step success over the best latent baseline (VideoWorld's 0% → VideoWorld 2's 68.8%), and more dramatically, the difference between total failure and robust success.

The degradation pattern for video generation models is particularly informative. Despite receiving detailed textual descriptions for each step—information a human learner does not need—these models uniformly fail by step 4. The paper interprets this as evidence that they cannot learn the procedural dependency chain: each action depends on the precise outcome of the previous action, and errors in earlier steps (even visually subtle ones) make subsequent steps physically impossible. The models can reproduce individual actions from language descriptions but cannot maintain fidelity across multi-step chains because they lack an internal representation of the object state that tracks how each action transforms the workpiece.

Video-CraftBench: Data Scaling with OpenX

The second major result (Table 1, rows 10–15) evaluates whether pretraining the latent code extraction on a larger, more diverse dataset (OpenX) improves performance on Video-CraftBench. The key finding is that cross-domain latent code pretraining benefits VideoWorld 2 substantially more than any baseline.

The baselines:

• iVideoGPT (row 10): OpenX pretraining provides modest gains over craft-only training (row 5 equivalent in its own category), but still only reaches 23.1% step-1 and 0.0% by step 5 on paper folding.
• Moto (row 11): Improves from craft-only (row 6: 19.1% step-1, complete failure) to 43.1% step-1 with OpenX, but still reaches only 9.7% step-6 and 0.0% step-7. OpenX pretraining helps Moto make progress further in the sequence but not complete it.
• AdaWorld (row 12): Shows the most dramatic improvement among baselines from OpenX pretraining: reaches 49.5% step-1, 41.6% step-2, and achieves 13.0% step-7—the only baseline besides CoLA and VideoWorld to complete the full sequence at all. However, the final-step success remains low (13.0%).
• CoLA (row 13): The closest concurrent approach (also uses VDM but limited to 2-frame transitions), achieves 83.5% step-1, 74.4% step-2, but decays to 40.2% step-7 and 54.1% on block human. This is substantially better than non-VDM baselines but far below VideoWorld 2.
• VideoWorld (row 14): With OpenX pretraining, reaches 91.7% step-1 and 48.2% step-6, but drops to 31.9% at step-7—it can now complete the full sequence (unlike craft-only training), but success is unreliable. Note the SSIM drops to 0.601 from 0.680—OpenX pretraining apparently trades off visual quality for improved dynamics in the original VideoWorld framework.

VideoWorld 2 with OpenX pretraining (row 15) achieves 72.3% step-7 on paper folding and up to 85.8% on block stacking, with SSIM 0.774 and LPIPS 0.193. The step-by-step trajectory is: 98.2% → 96.4% → 90.1% → 86.7% → 83.3% → 81.7% → 72.3%. The OpenX pretraining provides a modest absolute improvement over craft-only training (72.3% vs. 68.8% final-step, 85.8% vs. 80.9% block human), suggesting that VideoWorld 2 already extracts most of the useful dynamics from the craft data alone, but gains additional robustness from exposure to diverse manipulation motions. The most notable difference is in step-6 to step-7 transition: craft-only drops from 74.6% to 68.8% (a 5.8 percentage-point drop), while OpenX-pretrained drops from 81.7% to 72.3% (a 9.4 percentage-point drop). The OpenX model is stronger in absolute terms but shows a similar relative decay pattern, suggesting that the final steps remain challenging regardless of pretraining scale.

The paper also reports qualitative evidence for cross-domain transferability (Figure 5, Section 3.1): similar latent codes in OpenX and Video-CraftBench correspond to videos with similar motion patterns despite differing environments and agents. For instance, a code representing forward reaching motion in a robot arm dataset corresponds to a similar forward hand movement in a handicraft video—the codes abstract away the embodiment-specific appearance and capture the underlying motion primitive.

CALVIN: In-Domain Latent Pretraining

Table 2 (rows 1–4) evaluates whether pretraining the AR transformer on latent dynamics codes from CALVIN videos improves policy learning when fine-tuned on a small fraction of action-labeled data. The setup: pretrain on latent codes from 22k CALVIN trajectories (no action labels), then fine-tune on only 2k ground-truth action labels (10% of the full dataset).

• Oracle trained on full 22k labels (row 1): 80.9% step-1, 24.6% step-5, average length 2.36 completed tasks. This is the upper bound for what policy learning can achieve on this data.
• Oracle trained on 10% labels only (row 2): 50.5% step-1, 0% step-5, average length 1.11. The data-efficiency baseline—training from scratch on limited labels.
• LAPA latent pretraining + 10% labels (row 3): 74.4% step-1, 2.30% step-5, average length 1.49. Latent pretraining substantially improves early-step success but provides limited benefit for long-horizon completion.
• VideoWorld 2 latent pretraining + 10% labels (row 4): 75.8% step-1, 9.70% step-5, average length 1.87. The key advantage over LAPA manifests in later steps: VideoWorld 2 achieves step-5 success of 9.70% vs. LAPA's 2.30%, and step-4 of 20.4% vs. 15.3%. The latent dynamics codes from VideoWorld 2 provide more transferable manipulation priors that help the model complete longer sequences, even though the early-step performance is comparable.

The average length metric (1.87 vs. 1.49 for LAPA, vs. 2.36 for the full-data oracle) quantifies how much of the 5-task sequence is typically completed. VideoWorld 2's latent pretraining recovers approximately 79% of the full-data oracle's average length (1.87/2.36) using only 10% of the action labels, representing a ~10× data efficiency improvement for long-horizon performance.

CALVIN: Cross-Domain Latent Pretraining

Table 2 (rows 5–7) evaluates the most challenging transfer scenario: pretraining on OpenX latent codes (from diverse robots and environments), then fine-tuning on the full 22k CALVIN action labels.

• Video next-token prediction pretraining on OpenX (row 5): 85.9% step-1, 23.0% step-5, average length 2.46. This baseline pretrains on raw OpenX videos using standard next-frame prediction, then fine-tunes on CALVIN actions. It improves over the in-domain oracle (80.9%, 2.36), showing that cross-domain video pretraining transfers positively.
• LAPA latent pretraining on OpenX (row 6): 84.0% step-1, 27.0% step-5, average length 2.51. Latent code pretraining improves step-5 success substantially over raw video pretraining (27.0% vs. 23.0%) despite slightly lower step-1 success, suggesting latent codes better capture manipulation structure that aids long-horizon completion.
• VideoWorld 2 latent pretraining on OpenX (row 7): 88.5% step-1, 30.9% step-5, average length 2.88. This surpasses both the full-data in-domain oracle (80.9%, 24.6%, 2.36) and all cross-domain baselines. Notably, VideoWorld 2's step-1 success exceeds the oracle trained on 22k CALVIN labels—this is a cross-domain improvement over in-domain training with full supervision. The step-5 success (30.9%) represents a 25.6% relative improvement over the video next-token prediction baseline (23.0%) and a 14.4% improvement over LAPA (27.0%).

The average length of 2.88 means VideoWorld 2 completes nearly 3 of the 5 sequential tasks on average, vs. 2.36 for the in-domain oracle and 2.46–2.51 for cross-domain baselines. This demonstrates that latent dynamics codes learned from a different embodiment in a different visual domain transfer more effectively than ground-truth action labels from the target domain—a striking result that supports the paper's central claim about appearance-abstracted, transferable dynamics representations.

Ablation Studies and Robustness Checks

All ablation results are in Table 3, trained on Video-CraftBench only and evaluated on the CALVIN in-domain setting unless otherwise noted.

• Pre-trained VDM introduction (Table 3a, rows 1–2): Removing the VDM entirely (i.e., using the original VideoWorld LDM) yields 0.0% paper folding success and 28.5% block success with LPIPS 0.312. Adding the VDM (but without gradient-stopping or ControlNet) improves to 30.3% paper and 45.2% block with LPIPS 0.297. This confirms that the VDM alone provides a substantial improvement—the paper folding task goes from complete failure to partial success—but additional mechanisms are needed.

• Decoder gradient-stopping (Table 3a, row 2 vs. 3): Removing gradient flow from the VQ-VAE decoder (with VDM active) increases paper folding success from 30.3% to 47.3% (a ~56% relative gain) and block success from 45.2% to 54.7%. LPIPS improves from 0.297 to 0.275. This is the ablation that validates the core "noise injection" hypothesis: the reconstruction decoder, when optimized alongside the latent codes, introduces gradient pressure that encourages codes to encode task-irrelevant appearance details. Eliminating this pressure substantially improves code transferability.

• ControlNet motion conditioning (Table 3a, rows 3 vs. 4–5): Using the coarse decoder output as ControlNet conditioning (row 4) increases paper folding from 47.3% to 51.1%, but the LPIPS improvement is dramatic: 0.275 → 0.213. Adding gradient-stopping on the ControlNet path (row 5, the full dLDM) further increases paper folding to 68.8% and block success to 77.5%, with LPIPS 0.205. The ControlNet conditioning primarily improves video quality (coherent object motion prevents the VDM from hallucinating incorrect movements), while the gradient-stopping ensures the latent codes do not optimize for this quality signal. The benefit is more pronounced on the longer paper folding task (47.3% → 68.8%, a 45% relative gain) than on block stacking (54.7% → 77.5%, a 42% gain), demonstrating that motion conditioning is particularly important for maintaining coherence over longer temporal horizons.

• Query embedding length N (Table 3b): N=1 achieves 41.9% paper folding and 65.0% block with LPIPS 0.221 and average CALVIN length 1.53. N=2 improves to 55.1% paper and 69.5% block (LPIPS 0.210, avg. length 1.64). N=4 (default) achieves 68.8% paper and 77.5% block (LPIPS 0.205, avg. length 1.87). N=8 slightly improves LPIPS to 0.195 but reduces paper folding to 65.0% and increases avg. length to 1.88. The non-monotonic relationship—N=8 degrades task success despite better visual quality—indicates that excess query capacity allows the latent codes to encode spurious information that the AR transformer struggles to model accurately, even though this information helps the VDM produce slightly sharper videos.

• VDM-LDM interplay (Table 3c): The projection layer configuration significantly impacts performance. MLP-only projection with non-causal cross-attention yields 52.0% paper folding and 61.3% block. Adding self-attention to the projection (still non-causal cross-attention) gives 52.3% and 61.8%—negligible improvement. Switching to causal cross-attention (MLP-only projection) dramatically improves to 69.8% paper and 78.6% block. Adding self-attention to the projection with causal cross-attention yields 72.3% paper and 80.9% block (this is the OpenX-pretrained configuration). The dominant factor is causal cross-attention: non-causal attention allows the VDM to "see" future latent codes when denoising earlier frames, providing an unrealistic information pathway during training that does not exist at inference time. This creates a train-test mismatch that degrades autoregressive generation quality.

• Codebook size (Table 3d): Codebook size 8 yields 20.1% paper and 29.9% block with LPIPS 0.258, avg. length 1.65. Size 1000 (default) achieves 68.8% paper and 77.5% block with LPIPS 0.205, avg. length 1.87. Size 4096 drops to 50.4% paper and 59.6% block (LPIPS 0.208, avg. length 1.90). Size 64,000 further drops to 29.4% paper and 36.0% block (LPIPS 0.230, avg. length 1.89). The CALVIN metric (avg. length) is surprisingly robust to codebook size, remaining between 1.65 and 1.90 across all sizes. This suggests the manipulation actions in CALVIN are inherently low-dimensional enough that even coarse discretization captures them. Video-CraftBench, with its more complex deformation and multi-step dependencies, requires sufficient codebook capacity to disambiguate distinct actions, but excessive capacity introduces convergence difficulties and noise encoding.

• Horizon length T (Table 3e): T=2 (pairwise transitions, CoLA-like) yields 19.1% paper folding and 38.7% block, avg. CALVIN length 1.55. Performance improves monotonically: T=9 → 55.4% paper, 68.7% block, avg. length 1.61; T=49 → 65.3% paper, 76.2% block, avg. length 1.80; T=93 (default) → 68.8% paper, 77.5% block, avg. length 1.87. Extending to T=177 yields 69.0% paper and 76.8% block, with a slight drop in avg. length to 1.79. The plateau at T=93 corresponds to the Cosmos VDM's maximum context length; extending beyond provides marginal benefit. The large jump from T=2 to T=9 (19.1% → 55.4% for paper folding) demonstrates that even modest multi-step context dramatically improves dynamics representation compared to pairwise modeling. This ablation provides the strongest empirical evidence for the paper's claim that "multi-step modeling is critical for long-horizon tasks."

• VDM fine-tuning strategy (Table 3f): Randomly initialized VDM (row 1) yields 0.0% paper folding and 0.0% block—complete failure. Freezing the pretrained VDM (row 2) yields 31.7% paper and 40.2% block. LoRA fine-tuning (row 3) yields 50.9% paper and 62.3% block. Full fine-tuning (row 4) yields 68.8% paper and 77.5% block. The large gap between frozen (31.7%) and fully fine-tuned (68.8%) indicates that the VDM's pretrained appearance priors are strong but require substantial adaptation to capture the fine-grained manipulation details specific to handicraft tasks—hand poses, paper deformation, crease formation—that are unlikely to appear in the VDM's original training distribution. The gap between LoRA (50.9%) and full fine-tuning (68.8%) further suggests that the required adaptation is not low-rank; significant capacity in the VDM's weights must be updated.

UMAP visualization (Figure 7): This is not a standard ablation but a qualitative robustness check. The paper samples 4000 trajectories from CALVIN and Bridge (included in OpenX), labels each by robot arm action (up, down, left, right), and visualizes the latent codes via UMAP. With dLDM + VDM (left panel), codes for the same action cluster tightly across environments—"right" movements in Bridge and CALVIN map to overlapping regions. With VideoWorld's original LDM (right panel), codes for the same action show "significant divergence across environments and fail to cluster effectively"—the two environments form largely separate clusters even for identical actions. This visualization provides intuitive evidence for the paper's central claim that VDM-based appearance offloading produces more environment-invariant dynamics representations.

CoLA comparison (Table 1, row 13 vs. 15): CoLA, a concurrent work that also uses VDM for latent action optimization, achieves 40.2% step-7 paper folding with OpenX pretraining vs. VideoWorld 2's 72.3%. The paper attributes this gap to CoLA's two specific limitations: (1) 2-frame transition modeling (vs. dLDM's 93-frame context), and (2) lack of structured temporal cues from coarse VAE outputs. The horizon length ablation (Table 3e) corroborates the first factor—T=2 yields 19.1% craft-only, far below T=93's 68.8%—but the paper does not provide a direct ablation removing ControlNet motion conditioning specifically for CoLA's architecture.

ReSt^EM attempt (discussed in prior sections): The paper mentions that further optimizing the revision/dynamics model with ReSt-style RL training degraded performance. While this ablation is described in the Appendix K reference, the main text does not detail it; the relevant point is that naive application of RL optimization to the latent code learning pipeline is not straightforward.

Critical Assessment

Claim 1: "VideoWorld 2 achieves up to 70% improvement in task success rate" over competing methods.

What was tested: This claim is most directly supported by the paper folding sequential success rate comparison in Table 1. VideoWorld 2 (craft-only) achieves 68.8% final-step success; the best non-CoLa latent baseline (VideoWorld, craft-only) achieves 0.0%. The "70% improvement" is the ~68.8 percentage-point difference, reported as a relative improvement over essentially zero.

Genuine strengths of the evidence: The sequential nature of the metric is rigorous—a method cannot "cheat" by completing step 7 incorrectly because step 7 is only evaluated if steps 1–6 are correct. The collapse of video generation models by step 5 (0.0% for all four) provides a clean contrast that supports the claim. The multiple baseline categories (video generation, latent action, VideoWorld) show that the gap is not an artifact of comparing against weak baselines—each baseline represents a distinct approach that fails in a characteristic pattern.

What the experiments demonstrate versus what they do not: The experiments demonstrate that VideoWorld 2 can complete paper folding sequences in unseen environments with high reliability, while all tested alternatives cannot. However, the "70% improvement" framing is somewhat misleading because it derives from a 0% baseline—any non-zero performance would represent an infinite relative improvement. The absolute metric (68.8% success vs. 0.0%) is more meaningful than the "70%" framing.

Weaknesses and missing evidence:

1. No confidence intervals or error bars. The test set for Video-CraftBench is ~150 videos. The sequential success rate is a cascaded binary metric: a test case either reaches step k or it does not. With ~150 test cases, a 68.8% success rate corresponds to roughly 103 successes. The standard error on this proportion is approximately 3.8 percentage points, meaning the true success rate could reasonably be 61–77%. Without confidence intervals, the comparison with CoLA's 40.2% or VideoWorld + OpenX's 31.9% cannot be rigorously assessed for statistical significance. The sharp drops at certain steps (e.g., craft-only VideoWorld 2 drops from 74.6% step-6 to 68.8% step-7, suggesting ~9 additional failures at the final step) are suggestive but unquantified.
2. 3-rollout evaluation without selection criterion. The paper states "we generate 3 video rollouts per test case." It is unclear whether the reported success rates reflect the best of 3, the average, or some other criterion. If the best of 3 is used, the reported rates are optimistic relative to single-sample performance. If the classifier evaluates all 3, the paper does not specify whether a test case counts as successful if any rollout succeeds or if all rollouts succeed—and these represent very different standards.
3. The difficulty distribution of the test set is not characterized. If the test set happens to contain easier cases (fewer background distractors, more canonical viewpoints), success rates may overestimate real-world generalization. The paper describes test set variations (novel backgrounds, textures, paper appearances, camera viewpoints) but does not report per-variation breakdowns—are failures concentrated on the most visually divergent test cases, or distributed uniformly?
4. Block building results are not universally strong for VideoWorld 2. While the paper emphasizes the block stacking success (70.0–81.5%), the gap between VideoWorld 2 and the best video generation baseline on block "Human" is smaller: 70.0% vs. Wan's 39.7%. The gap is 81.5% vs. 42.6% on "Tower." These are substantial improvements but not the "baseline achieves zero" dynamic found in paper folding. The block tasks are shorter (20–30 seconds vs. 40–80 seconds) and may be inherently easier for video generation models, suggesting that VideoWorld 2's advantage is most pronounced in the long-horizon regime where error accumulation dominates.

Claim 2: "Disentangling action dynamics from visual appearance modeling is critical for effective knowledge acquisition."

What was tested: This claim is supported by multiple converging pieces of evidence: (1) the failure of pretrained VDMs on long-horizon tasks despite strong short-step performance (Table 1, rows 1–4), (2) the ablation showing that adding VDM to VideoWorld improves success (Table 3a, rows 1–2), (3) the improvement from gradient-stopping the decoder (Table 3a, row 3), (4) the UMAP visualization showing VDM-enabled codes cluster by action, not environment (Figure 7), (5) the OpenX-to-CALVIN transfer results where cross-domain latent pretraining outperforms in-domain action-labeled training (Table 2, row 7 vs. 1).

What the experiments demonstrate versus what they do not: The experiments demonstrate that the specific architectural intervention—replacing the LDM decoder with a pretrained VDM plus gradient-stopping and ControlNet conditioning—produces large performance gains across multiple tasks. This is strong evidence that the intervention is effective, but it is not direct evidence for the causal mechanism proposed (appearance-dynamics disentanglement). The UMAP visualization provides qualitative evidence for more environment-invariant codes, but the paper does not provide a quantitative metric of disentanglement (e.g., mutual information between latent codes and environment labels, or a downstream classification task measuring how well codes predict actions vs. environments).

Weaknesses and missing evidence:

1. The disentanglement claim is tested only through performance improvement, not through direct measurement. The paper could have trained an auxiliary classifier to predict environment identity from latent codes—if disentanglement is truly occurring, codes should make this prediction difficult. Alternatively, the paper could have measured the variance in code representations for identical actions across environments and compared this between dLDM and LDM. The UMAP provides qualitative evidence but is subject to visualization artifacts and projection choices.
2. The ablation chain (Table 3a) shows that several mechanisms contribute to the improvement—not just the VDM. Going from "no VDM" (0.0%) to "VDM, no gradient-stop, no ControlNet" (30.3%) to "VDM + gradient-stop" (47.3%) to "VDM + gradient-stop + ControlNet" (68.8%) shows that each mechanism adds value. The paper attributes the full improvement to appearance-dynamics disentanglement, but the gradient-stop specifically prevents the decoder from injecting noise—this is a training stability mechanism that could improve performance even without disentanglement. The ControlNet provides coarse motion guidance that could help the VDM generate better videos even if the latent codes are not perfectly disentangled. The relative contribution of disentanglement versus training stability versus motion guidance is not isolated.
3. The paper does not evaluate whether the dLDM codes are truly appearance-invariant under systematic appearance perturbations. A convincing test would be: take the same action sequence, render it with multiple different backgrounds/textures/lighting conditions (e.g., via synthetic data), and measure whether the dLDM codes remain consistent. The current evaluation relies on naturally occurring environmental variation in the test set, which may be confounded with uncontrolled factors.

Claim 3: "The learned latent codes enable effective cross-domain transfer—from OpenX to CALVIN, and from OpenX to Video-CraftBench."

What was tested: The CALVIN results (Table 2) demonstrate cross-embodiment, cross-domain transfer. VideoWorld 2's latent pretraining on OpenX (diverse robots, environments, tasks) followed by CALVIN fine-tuning achieves 88.5% step-1 and 2.88 average length—outperforming both the in-domain oracle trained on 22k CALVIN labels (80.9%, 2.36) and the video next-token prediction baseline (85.9%, 2.46). The Video-CraftBench results (Table 1, rows 9 vs. 15) show that OpenX pretraining provides a modest additional gain (68.8% → 72.3% paper folding, 77.5% → 85.8% block stacking).

What the experiments demonstrate versus what they do not:

1. The CALVIN result is genuinely strong. Training on data from different robots doing different tasks in different visual environments, and then fine-tuning on CALVIN to outperform a model trained from scratch on full CALVIN supervision, is a compelling demonstration of transfer. The gap between VideoWorld 2's cross-domain result (2.88 avg. length) and LAPA's (2.51) shows the dLDM architecture specifically benefits cross-domain transfer, not just any latent code approach.
2. The Video-CraftBench transfer is more modest. The gain from adding OpenX pretraining (68.8% → 72.3%) is real but incremental, especially compared to the gap between no-pretraining baselines and VideoWorld 2 itself. This suggests that the handicraft domain is sufficiently specific that general manipulation pretraining provides limited additional benefit—most of the "knowledge" is learned from the craft data itself.
3. The paper does not fully disentangle the effect of data scale from the effect of data diversity. The OpenX dataset is both larger (1.3M trajectories vs. 9.5k clips) and more diverse (many robots, environments, tasks) than Video-CraftBench. The improvement in Video-CraftBench from adding OpenX could be due to scale alone, diversity alone, or both. An ablation training on an equivalent volume of craft-only data (if collectable) would isolate the diversity effect.

Weaknesses and missing evidence:

1. Only one cross-domain transfer direction is thoroughly tested (OpenX → CALVIN). The reverse direction (craft → robotics), or transfer between different robot datasets within OpenX, is not evaluated. The paper's claim about "transferable knowledge" would be stronger with evidence that transfer works bidirectionally and across multiple domain boundaries.
2. The paper does not compare against alternative transfer methods. Could the same CALVIN improvement be achieved by simply training a larger transformer on the same OpenX videos (not latent codes) and fine-tuning on CALVIN? The "video next-token prediction" baseline (Table 2, row 5) partially addresses this—it performs worse (2.46 vs. 2.88 avg. length)—but uses a different architecture. A stronger baseline would be: same AR transformer architecture, same OpenX data, but predicting VQ-VAE tokens from raw video instead of dLDM latent codes. This would isolate whether the benefit comes from the dLDM latent space or from the pretraining scale.

Claim 4: "VideoWorld 2 produces coherent long execution videos with high visual quality."

What was tested: SSIM and LPIPS metrics in Table 1 quantify visual quality relative to ground-truth videos. VideoWorld 2 achieves SSIM 0.774 and LPIPS 0.193 with OpenX pretraining, compared to SSIM 0.585–0.719 and LPIPS 0.237–0.394 for baselines. Qualitative results in Figure 2 and Figure 11 show generated frames that appear visually coherent.

What the experiments demonstrate versus what they do not: The metrics and qualitative examples demonstrate that VideoWorld 2 produces videos that are perceptually closer to ground truth than baseline outputs. However, SSIM and LPIPS have well-known limitations: they measure low-level similarity and can reward blurriness (SSIM) or be insensitive to semantically important differences. More critically, the metrics are computed against ground-truth videos from the test environment—this measures similarity to a specific reference execution, not absolute visual quality. A model that generates a visually different but equally valid folding sequence (e.g., folding slightly faster or slower, with a different hand trajectory) would score worse on these metrics even if the task is successfully completed.

Weaknesses and missing evidence:

1. No human evaluation of video quality or task completion. The paper does not report human judgments of generated video quality, naturalness, or task completion accuracy. The DINOv2 classifier provides a proxy for task completion but was trained partly on generated videos and may inherit biases.
2. The VDM's autoregressive extension introduces acknowledged artifacts. The paper notes that "because the VDM's inherent reconstruction noise accumulates over time, visual artifacts such as lighting, texture, or color shifts may gradually appear." These artifacts are visible in the generated sequences but are not quantified. An ideal evaluation would measure artifact severity as a function of generated sequence length.
3. The visual quality comparison is confounded by the task completion gap. Baselines that fail to complete the task generate frames of incorrect actions—these will naturally have worse SSIM/LPIPS because the visual content diverges from ground truth. This means the visual quality metrics partially measure task success, not pure visual fidelity. A fairer visual quality evaluation would compare VideoWorld 2 and baselines on the parts of the sequence they can generate (e.g., steps 1–3 for video generation models), isolating appearance quality from task completion.

Missing experiments that would strengthen the paper:

1. Scaling experiments. How does VideoWorld 2's performance vary with training data volume? The paper uses all available Video-CraftBench data and all OpenX data, but does not show learning curves. This is important for assessing whether the approach would benefit from orders-of-magnitude more internet video data (the motivating vision from the introduction).

2. Ablation on the VDM's pretraining scale. The paper uses Cosmos DiT 2B. Would a smaller VDM (e.g., a 500M parameter model) suffice, or does disentanglement quality scale with VDM capacity? Would a larger VDM (if available) further improve performance?

3. Evaluation on tasks beyond handicraft and block stacking. The paper's claims about "complex, long-horizon tasks" are supported by two task categories. Testing on additional domains—cooking, assembly, repair procedures—would test whether the approach generalizes across task types or is tuned to the specific characteristics of folding and stacking.

4. Controlled appearance perturbation experiments. The paper claims latent codes are appearance-invariant, but this is tested only through natural environmental variation. A controlled experiment that systematically varies backgrounds, lighting, textures, and viewpoints on a fixed set of action sequences would provide direct causal evidence for appearance invariance.

5. Direct measurement of what information the latent codes encode. A probing experiment that trains linear classifiers to predict environment identity, background features, or action semantics from frozen latent codes would quantitatively characterize the degree of disentanglement. The paper shows UMAP clusters but provides no probing accuracy numbers.

6. Comparison with explicit action-labeled training at scale. For Video-CraftBench, the paper does not train a supervised model on manually annotated action sequences (e.g., keypoint trajectories or sub-action labels) to establish how much of the difficulty is "learning from video" versus "the task is inherently hard." If a model trained on explicit action labels also struggles with long-horizon paper folding, the failure mode may not be specific to unlabeled video learning.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Accounted For in the Headline Efficiency Gains

The assumption or constraint. The entire VideoWorld 2 framework—both training and evaluation—assumes that the system has access to a large corpus of demonstration videos from diverse environments. The training of the dLDM, the AR transformer, and the fine-tuning of the VDM all require video data showing correct task execution. This is not acknowledged as a cost, but it is the fundamental resource requirement that determines when the approach is applicable. For a new task, one must first collect hours of first-person tutorial footage (~7 hours for the 5 handicraft tasks in Video-CraftBench, split into ~9.5k clips). The paper does not analyze how performance scales with training data volume—no learning curves, no data-efficiency experiments that subsample the training set—so a practitioner cannot estimate the data collection burden for a new task.

Additionally, the evaluation protocol relies on a task-specific classifier (DINOv2-based, trained on ~25k labeled frames including frames from generated videos) to measure sequential success rates. Training this classifier requires manual annotation of key steps and failed states across training, testing, and generated environments—a non-trivial labeling effort that is task-specific and must be repeated for each new benchmark task. The paper is transparent about this (Section 4.2, Section 9), but the cost of building the evaluation infrastructure is externalized from the model training.

The consequence. For any new task domain, a practitioner must: (1) collect and curate hours of demonstration video covering environmental diversity, (2) train the dLDM from scratch (or with VDM fine-tuning), (3) train the AR transformer on the extracted latent codes, and (4) if quantitative evaluation is desired, annotate key steps and train a classifier. The paper provides no guidance on the minimum viable training set size, the sensitivity of the dLDM's latent code quality to data volume, or whether performance saturates within the ~7 hours collected. A task that is rarer or harder to collect video for—e.g., a specific repair procedure or an uncommon craft—may be infeasible to learn with this approach.

For the CALVIN robotics experiments, the situation is different: the dLDM is pretrained on the large-scale OpenX dataset (1.3M trajectories), and the AR transformer is trained on in-domain CALVIN latent codes (22k trajectories) before fine-tuning on action labels. This leverages existing large-scale datasets. But the headline Video-CraftBench results (68.8% success) use only the 7 hours of collected craft data—the paper does not demonstrate that a smaller craft dataset would suffice, or that the dLDM's latent code quality degrades gracefully with less data.

What evidence exists in the paper. None. The paper does not include any data scaling experiments—no learning curves showing task success rate or latent code reconstruction quality as a function of training video hours. The OpenX pretraining results (Table 1, rows 10–15) show that adding a large, out-of-domain dataset provides only a modest gain on Video-CraftBench (68.8% → 72.3% paper folding), but this tells us about the benefit of additional data beyond ~7 hours, not about the minimum required data. The ablation studies (Table 3) all use the full Video-CraftBench dataset; none subsample the training data. The CALVIN experiments do include a data-efficiency setting (10% of action labels for fine-tuning, Table 2, rows 2–4), but this is about label efficiency for policy fine-tuning, not about the video data requirements for training the dLDM.

Mitigation status. Not addressed. The paper does not suggest future work on data scaling or minimum data requirements. The focus is entirely on architectural innovation (dLDM vs. LDM) and cross-domain transfer, with data quantity treated as a fixed given.

---

Hardest Tasks Remain Essentially Unsolved: Test-Time Performance Depends on the AR Transformer's Planning Horizon

The assumption or constraint. While VideoWorld 2 dramatically outperforms baselines on the full 7-step paper folding sequence (Table 1: 68.8% craft-only, 72.3% with OpenX), there remains a non-trivial failure rate on the longest tasks. The sequential success rate exhibits a consistent decay pattern: 98.2% → 96.4% → 90.1% → 86.7% → 83.3% → 81.7% → 72.3% (OpenX-pretrained). The drop in the final two steps is particularly notable—from step 5 to step 6, the decay is modest (83.3% → 81.7%), but from step 6 to step 7, it accelerates (81.7% → 72.3%). This means that roughly 1 in 4 test cases that successfully complete steps 1–6 fail at the final step. The paper does not analyze why these failures occur—whether the AR transformer plans an incorrect action sequence, whether the VDM fails to faithfully render a correct latent code sequence, or whether the dLDM's latent codes fail to capture the final step's dynamics during training.

The block stacking tasks show a similar pattern. While success rates are higher (74.0–85.8% with OpenX), they are not at ceiling. This means the approach does not fully solve the tasks under study—it achieves strong but imperfect performance, with error rates of 15–28% on block stacking and 28–32% on complete paper folding sequences.

The consequence. For deployment in applications where task completion reliability is critical—e.g., generating tutorial videos for human consumption, or executing actions in a real robotics setting—a 28% failure rate on paper folding is too high for unsupervised operation. The framework as presented cannot detect its own failures: the AR transformer outputs latent codes regardless of whether the planned sequence will succeed, and the VDM renders them into convincing-looking video even when the actions are incorrect. There is no confidence score, no self-verification mechanism, and no recovery or replanning. A user would need to visually inspect each generated video to determine whether the task was executed correctly—defeating much of the automation value.

Additionally, the decay pattern suggests that performance would continue to degrade on even longer tasks. While Video-CraftBench's longest tasks are 40–80 seconds (paper folding), many real-world procedural tasks—cooking a complex meal, performing a repair procedure, executing an assembly sequence—involve dozens or hundreds of steps. The paper provides no evidence about whether the AR transformer's autoregressive planning horizon can be extended beyond the tested sequence lengths, or whether error accumulation in the latent code prediction compounds to make very long tasks infeasible.

What evidence exists in the paper. The sequential success rate curves (Table 1, rows 9 and 15; Table 2, row 7 for CALVIN) provide direct evidence. The consistent monotonic decay in success rate is visible across all task types and training configurations. The paper does not analyze individual failure cases—no confusion matrices showing which specific steps are most error-prone, no qualitative examples of failure modes at step 6 or 7, and no diagnostic analysis of whether failures originate in the AR transformer, the dLDM's code quality, or the VDM's rendering fidelity. The ablation on AR transformer query embedding length (Table 3b, N=8) hints at a potential mechanism—excess latent code capacity leads to worse task success despite better visual quality, suggesting the AR transformer struggles with larger code spaces—but this is not connected to the specific failure analysis.

The CALVIN average length metric (2.88 tasks completed out of 5) reinforces the pattern: even with strong cross-domain pretraining, the approach does not reliably complete all 5 sequential tasks in the benchmark.

Mitigation status. Not addressed. The paper does not discuss failure case analysis, confidence estimation, error detection, or replanning. Section 8 (future work) mentions "continued scaling" but does not specifically address reliability or long-horizon error accumulation.

---

The VDM's Autoregressive Extension Introduces Visual Artifact Accumulation That Is Not Quantified

The assumption or constraint. The VDM (Cosmos DiT 2B) can process only 93 frames at a time (~5 seconds at 16 fps). For longer tasks—paper folding sequences can span thousands of frames (40–80 seconds)—video generation proceeds autoregressively: each 93-frame segment is generated conditioned on the last frame of the previous segment and the corresponding latent codes predicted by the AR transformer. The paper explicitly acknowledges that this introduces accumulating artifacts:

"Because the VDM's inherent reconstruction noise accumulates over time, visual artifacts such as lighting, texture, or color shifts may gradually appear. Nevertheless, the key steps within the generated sequences remain accurate."

This is stated in Section 10 of the appendix but is not quantified anywhere in the paper. The "reconstruction noise" refers to the fact that each autoregressive step conditions on a VDM-generated frame rather than a ground-truth frame. Since VDM outputs are imperfect reconstructions (with subtle blur, color shifts, or texture distortion), these imperfections compound when used as conditioning for the next segment.

The consequence. The visual quality of generated videos degrades as a function of sequence length. For a 40–80 second paper folding video (8–16 autoregressive segments), the final frames may have noticeably different lighting, texture, or color characteristics than the initial frames. This matters for at least two reasons:

1. Task evaluation may be affected at the margins. The DINOv2 classifier used to assess step completion was trained partly on frames from generated videos that may include these artifacts. If the classifier learns to associate certain visual artifacts with task failure, or conversely learns to ignore visual degradation that correlates with real task errors, the reported success rates may be biased.
2. Generated videos may be unsuitable for direct human consumption. A tutorial video that drifts in color and lighting over 80 seconds will appear unnatural, reducing trust and usability. The paper's qualitative examples (Figures 2 and 11) show selected frames—possibly from earlier in the sequence where artifacts are less severe.

The paper's statement that "key steps within the generated sequences remain accurate" is an assertion, not a measured result. There is no evaluation of whether error rate in the classifier's step detection increases with sequence position, which would indicate that artifacts are impacting task evaluation.

What evidence exists in the paper. Only the qualitative acknowledgment in Section 10 and the qualitative visualizations. No quantitative metrics of artifact severity as a function of generated sequence length—e.g., LPIPS or SSIM computed specifically on later segments, or FID measured at different temporal offsets from the conditioning frame. The overall LPIPS (0.193–0.205) and SSIM (0.770–0.774) aggregate across entire sequences and may be dominated by early, higher-quality frames.

The autoregressive extension mechanism itself is described in Section 10: "Long videos are therefore generated auto-regressively by extending each segment from the final frame of the previous one." The paper does not describe any technique to mitigate the compounding error—no noise scheduling adjustment for later segments, no overlap-and-blend strategy, no conditioning refinement.

Mitigation status. Acknowledged qualitatively but not addressed. The paper does not propose solutions (e.g., training the VDM specifically on autoregressively-conditioned data to reduce compounding error, or using a sliding window with temporal overlap), nor does it suggest future work on this specific issue. The "continued scaling" future work (Section 7) may implicitly address this if future VDMs have longer context windows, but this is speculative.

---

Single Task Family, Single VDM Backbone, Single AR Transformer Architecture

The assumption or constraint. All primary results on Video-CraftBench are on two task categories within a single domain: paper folding (airplane, boat) and block building (tower, horse, person). These are both handicraft tasks involving manual manipulation of deformable or discrete objects under first-person viewpoints. The robotics results are on CALVIN, a specific simulated tabletop manipulation environment with a fixed set of 34 tasks. The VDM backbone is Cosmos DiT 2B; the AR transformer is Cosmos AR 4B. The paper provides no evidence that the approach would work with different VDM architectures (e.g., Wan, HunyuanVideo, Sora) or different AR transformer architectures (e.g., other LLM backbones), and no evidence on task domains substantially different from handicrafts or tabletop manipulation.

The paper acknowledges the narrow task scope implicitly by introducing Video-CraftBench as a benchmark for "fine-grained, long-horizon visual reasoning in real-world handicraft tasks" (Section 4), but does not discuss whether the findings are expected to generalize beyond handicrafts. Tasks like cooking (involving liquids, heat, state changes), sports (involving fast motion and complex physics), or navigation (involving environment-scale spatial reasoning) may present fundamentally different challenges for appearance-dynamics disentanglement.

The consequence. A practitioner cannot estimate, from the evidence provided, whether VideoWorld 2 would be effective for their specific task domain. Several factors could limit generalization:

• Task structure: Handicraft tasks have a natural sequential structure with clearly defined intermediate states (creases, folds, block configurations). The DINOv2 classifier can detect these states, and the latent codes can potentially capture them. Tasks where intermediate states are visually ambiguous (e.g., stirring a sauce, kneading dough) may be harder for the dLDM to represent in a compact code space.
• Visual complexity: Handicraft tasks involve relatively constrained visual scenes (hands, desk, paper/blocks). Tasks in cluttered environments (kitchens, workshops, outdoor settings) with many distractor objects may challenge the VDM's ability to focus on task-relevant elements, even with appearance modeling offloaded.
• Motion speed: Handicraft tasks involve slow, deliberate movements. Tasks with rapid motion (sports, tool use) may require different temporal modeling strategies, as the dLDM's 93-frame context at 16 fps (~5.8 seconds) may not capture enough motion information if actions are fast.
• VDM-model coupling: The dLDM's design is closely coupled to the Cosmos VDM's specific architecture—its maximum context length (93 frames), its ControlNet compatibility, its causal cross-attention interface, and the projection layer design. Switching to a different VDM backbone may require architectural modifications that the paper does not explore, and the ablation in Table 3a shows that training strategy (freeze vs. LoRA vs. full fine-tuning) dramatically affects performance (31.7% vs. 50.9% vs. 68.8% paper folding success). This suggests sensitivity to the VDM's adaptation characteristics, which may vary across VDM architectures.

What evidence exists in the paper. None directly. The paper tests multiple baselines across different architectural families (diffusion-based and autoregressive video generation, latent action models with different codec designs), but does not test VideoWorld 2 itself with different backbones. The OpenX-to-CALVIN transfer provides some evidence of domain generalization (different robots, different tasks), but both are tabletop manipulation domains with similar visual structure (robotic arms, grasped objects, desk surfaces). The OpenX-to-CraftBench transfer (Table 1, row 9 vs. 15) shows that adding diverse robot manipulation data provides only modest gains on handicraft tasks (68.8% → 72.3%), suggesting that the benefit of cross-domain pretraining may be limited when domain differences are large—but this is confounded with data scale.

Mitigation status. Not addressed. The paper does not discuss sensitivity to VDM or AR transformer architecture, nor does it propose experiments to test generalization to other task domains. The "continued scaling" future work mentions moving "toward the goal of enabling AI to learn the vast knowledge encapsulated in the real world," but this is aspirational, not a concrete plan for domain diversification.

---

No Latency or Online Applicability Analysis—The System Is Purely Offline

The assumption or constraint. The entire VideoWorld 2 pipeline operates in an offline, batch generation mode. The AR transformer generates a complete sequence of latent codes given an initial frame and task instruction, and the VDM decodes these codes into a full video. There is no mechanism for closed-loop execution where observations after each action inform the next action—the entire plan is fixed at the start. This is an open-loop policy.

For the handicraft setting, this is acceptable because the goal is video generation, not real-world execution. For the robotics setting (CALVIN), however, the evaluation follows an offline protocol: the AR transformer is fine-tuned on action labels after latent pretraining, and the resulting policy is evaluated in the simulator. The paper does not investigate whether the dLDM's latent codes could be used for online, closed-loop control in robotics—predicting the next action given the current observation, executing it, observing the result, and replanning.

Additionally, there is no analysis of inference latency. The generation pipeline involves: (1) AR transformer autoregressive decoding of a latent code sequence, (2) VDM denoising of 93-frame segments (which requires hundreds of denoising steps with a 2B-parameter diffusion model), and (3) autoregressive extension across multiple segments. The total wall-clock time to generate a minute-long video is not reported, nor are GPU memory requirements or throughput. This makes it impossible to assess practicality for applications requiring interactive or real-time performance.

The consequence. For applications that require closed-loop control—robotics, interactive assistants, real-time tutoring systems—the current framework is not directly applicable. The AR transformer would need to be conditioned on intermediate observations, not just the initial frame, and the VDM would need to generate frames on-demand rather than in a pre-planned batch. The paper does not discuss whether the dLDM latent codes are suitable for this—for example, whether codes for a single action can be decoded in isolation, or whether the VDM requires the full multi-frame context for coherent rendering.

For offline applications, the latency matters for throughput. If each minute of generated video requires minutes or hours of GPU computation, the approach may be impractical for large-scale video generation (e.g., generating thousands of tutorial variations). The paper provides no compute budget analysis—no FLOP counts, no GPU-hours per generated video, no comparison of inference cost with baseline methods.

What evidence exists in the paper. None. No latency measurements. No discussion of online vs. offline execution. The only reference to real-time considerations is negative: the paper notes that directly training the VDM to generate future frames from noise "would be extremely slow and prone to incorrect motion" (Section 3.2, regarding CoLA comparison), but does not report actual training or inference times.

Mitigation status. Not addressed. The paper makes no claims about online applicability and does not position VideoWorld 2 as suitable for closed-loop control. The omission is a scope limitation rather than an unacknowledged failure—but it is a critical scope limitation for practitioners considering robotics applications beyond the offline CALVIN evaluation protocol. The future work section (Section 7) mentions "continued scaling" but does not propose online adaptation or latency optimization.

---

The Disentanglement Claim Is Supported by Performance Improvement, Not by Direct Measurement

The assumption or constraint. The paper's central conceptual contribution is that "disentangling action dynamics from visual appearance modeling is critical for effective knowledge acquisition." This claim is supported entirely through indirect evidence: VideoWorld 2 outperforms VideoWorld (which does not disentangle), adding the VDM improves performance, gradient-stopping the decoder helps, and UMAP visualizations show better action clustering. However, none of these directly measure whether the latent codes encode less appearance information or are more invariant to environmental variation in a quantifiable way.

The UMAP visualization (Figure 7) is the closest the paper comes to a direct measurement. It shows that dLDM codes for the same action (robot arm moving right) cluster together across Bridge and CALVIN environments, while VideoWorld's LDM codes do not. But UMAP is a nonlinear dimensionality reduction that can distort distances and exaggerate clusters—it is a qualitative tool, not a quantitative measurement. The paper does not report any quantitative metric of appearance invariance, such as:

• Environment classification accuracy from frozen latent codes—if codes are appearance-invariant, it should be difficult to predict which environment (Bridge vs. CALVIN, or training vs. test craft environment) a code came from.
• Variance ratio between inter-action and intra-action code distances—if codes cluster by action, the ratio should be large; if they cluster by environment, it should be small.
• Correlation between environment-induced visual changes and code-space distances—ideally zero for perfectly disentangled codes.

The consequence. It is difficult to attribute VideoWorld 2's performance improvement specifically to appearance-dynamics disentanglement. The architectural intervention—adding a VDM with gradient-stopped ControlNet conditioning—introduces multiple changes simultaneously: (1) a stronger decoder (VDM vs. lightweight CNN), (2) gradient-stopping that prevents the decoder's reconstruction loss from influencing codes, (3) coarse motion guidance via ControlNet that stabilizes VDM training, (4) fine-tuning the VDM on the target domain data. Any or all of these could contribute to the performance gain, and the disentanglement narrative could be correct without being the primary causal factor. For example, if the principal benefit is that the VDM is simply a better decoder (producing higher-quality video that suffers less from autoregressive error accumulation), then the performance gain is due to improved rendering, not improved representation learning.

The CALVIN UMAP visualization shows that the dLDM codes show better action clustering than LDM codes for a specific set of robot actions—but this does not establish that appearance information is absent from the codes. It shows relative improvement in action clustering, not absolute disentanglement. The codes might still encode environment-specific information (e.g., robot morphology, camera angle) that is simply less dominant than in the LDM codes.

What evidence exists in the paper. Only the UMAP (Figure 7) and the ablation chain (Table 3a). The UMAP suggests better action clustering; the ablation shows that each mechanism (VDM, gradient-stop, ControlNet) adds value. But neither constitutes a direct measurement of appearance information encoded in the latent codes.

Mitigation status. Not addressed. The paper does not acknowledge the gap between the performance-improvement evidence and the disentanglement claim, nor does it propose quantitative disentanglement metrics or probing experiments. The "Further Discussion with Other Works" (Section 6) argues for the novelty of the disentanglement approach relative to prior work, but does not address how to verify that disentanglement is actually occurring. This is a methodological limitation that weakens the paper's conceptual contribution, even though it does not affect the practical utility of the approach—VideoWorld 2 works regardless of precisely why it works.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around video-based knowledge learning from a paradigm where models attempt to learn everything from pixels—appearance, dynamics, task structure—to one where the division of labor between components is the central design question. The shift is not a full paradigm change in the sense of introducing a new problem formulation (autoregressive video prediction remains the objective), but it is a conceptual reframing that changes which research questions the field should prioritize.

Before this work, the dominant approach to learning from unlabeled video was monolithic: a single model (or a tightly coupled encoder-decoder pair) was trained end-to-end to reconstruct or predict frames, with the implicit assumption that task-relevant knowledge would emerge as a byproduct of good video modeling. VideoWorld demonstrated that compressing visual changes into compact latent codes was more effective than generating full pixel sequences, but it still relied on a reconstruction objective that entangled appearance and dynamics. The failure of this approach on real-world videos—and the parallel failure of state-of-the-art video generation models on long-horizon tasks despite strong short-step performance (Table 1, rows 1–4)—revealed that the monolithic assumption is not just suboptimal but qualitatively wrong for complex domains. Models do not gradually improve with scale; they hit a cliff where error accumulation from appearance entanglement makes long-horizon completion impossible.

VideoWorld 2's reframing is that appearance modeling and dynamics modeling are fundamentally separable responsibilities that should be assigned to different components with different training objectives. The pretrained video diffusion model is not a better decoder—it is an appearance oracle whose ignorance of the target task's dynamics is a feature, not a limitation. This inverts the standard relationship: rather than the dynamics model serving the video generator (by providing conditioning), the video generator serves the dynamics model (by absorbing the visual detail burden so the dynamics codes can focus on what matters). This is a specific, testable design principle—delegate task-irrelevant variation to a separately trained, high-capacity model—that generalizes beyond video: it could apply to any domain where task-relevant and task-irrelevant variation are entangled in raw observations (audio, sensor data, multimodal inputs).

The paper also reconciles a latent tension in the world models literature. One line of work (video generation models as world simulators) emphasizes visual fidelity as evidence of physical understanding. Another line (JEPA-style approaches, latent action models) emphasizes abstract representation learning while accepting degraded visual quality. VideoWorld 2 demonstrates that these are not competing objectives—visual quality and task-relevant representation learning can be simultaneously achieved through architectural decomposition. The pretrained VDM handles visual quality; the dLDM handles task-relevant representation. Neither component alone can solve the long-horizon knowledge learning problem, but together they achieve what neither can. This resolution is practically important because it means researchers do not need to choose between approaches—they can compose them, with the pretrained VDM serving as a general-purpose visual decoder for task-specific dynamics models.

Several research directions become more attractive as a consequence:

• Scaling dynamics models independently of visual models. If a large VDM can serve as a universal appearance decoder for many different tasks, then the dLDM and AR transformer—the components that actually learn task knowledge—can be designed specifically for dynamics modeling without needing to represent visual detail. This suggests a future where a single large VDM serves many task-specific dynamics models, with each dynamics model being relatively small and focused.
• Improving VDM robustness to off-policy conditioning. The paper fine-tunes the VDM on the target domain because the pretrained VDM has never seen handicraft videos. As VDMs become more general (trained on ever-larger internet video corpora), the need for fine-tuning may decrease, making the approach more data-efficient on the dynamics side. This creates a natural scaling path: better pretrained VDMs → less domain adaptation needed → dynamics models can focus on a wider range of tasks with less video data per task.
• Cross-embodiment transfer as a latent representation problem. The CALVIN results (Table 2) showing that OpenX latent pretraining outperforms in-domain action-labeled training constitute the strongest evidence in the paper that the learned representations are genuinely embodiment-agnostic. This suggests that building generalist robot policies may be more tractable through latent dynamics pretraining on diverse video data than through explicit action-space alignment—a finding that should redirect effort in the robot learning community toward video-based pretraining objectives rather than action-labeled imitation learning at scale.

Several research directions become less attractive:

• End-to-end video generation for task execution. The uniform failure of four state-of-the-art video generation models on long-horizon tasks (Table 1, rows 1–4) provides strong evidence that scaling monolithic video models—whether diffusion-based or autoregressive—is insufficient for extracting transferable task knowledge. These models achieve excellent short-step performance and can be fine-tuned with detailed language annotations, yet they collapse deterministically by step 4. This does not mean video generation research is unimportant—the VDM remains a critical component in VideoWorld 2—but it suggests that video generation as the primary learning objective for task knowledge is a dead end without explicit appearance-dynamics decomposition.
• Pairwise frame transition models for complex manipulation. The horizon length ablation (Table 3e) demonstrates that short-horizon modeling (T=2, analogous to LAPA, Moto, CoLA) is fundamentally inadequate for long-horizon tasks—paper folding success at T=2 is 19.1% versus 68.8% at T=93. This should discourage further work on latent action models that operate on single frame transitions for tasks requiring multi-step planning, regardless of other architectural improvements.

Follow-Up Research This Work Enables

Quantitative disentanglement metrics for video dynamics representations. The paper's central claim—that the dLDM produces appearance-invariant latent codes—is supported through performance improvement and UMAP visualization, but not through direct measurement. A compelling follow-up would train probing classifiers on frozen dLDM and LDM latent codes to predict environment identity (Bridge vs. CALVIN, or training vs. test craft environment), action type (grasp, fold, place), and appearance features (lighting, texture, background). If disentanglement is genuinely occurring, dLDM codes should yield high action-classification accuracy and low environment-classification accuracy, with a larger gap between these than LDM codes. The paper's existing UMAP (Figure 7) provides qualitative evidence; quantifying this with standard probing methodology (train linear classifier, report accuracy, compare to majority-class baseline) would convert a qualitative observation into a verifiable claim. A strong result—dLDM codes achieving near-chance environment classification while maintaining high action classification—would validate the core mechanism and provide a metric that future work can optimize directly.

Systematic appearance perturbation experiments on controlled synthetic data. The paper evaluates generalization through naturally occurring environmental variation in the test set, which confounds multiple factors (background, lighting, texture, viewpoint). A follow-up could generate a controlled benchmark using a 3D rendering engine (e.g., Blender, MuJoCo with textured objects): record a fixed set of paper folding or block stacking action trajectories, render them with systematically varied backgrounds (10+ textures), lighting conditions (5+ configurations), camera angles (5+ viewpoints), and material appearances (10+ paper/block textures), producing thousands of videos where the ground-truth action sequence is identical but the visual appearance varies. Training the dLDM on a subset of appearance combinations and evaluating on held-out combinations would provide a clean measurement of appearance invariance: the success rate should be near-identical across all appearance conditions because the underlying dynamics are unchanged. The current paper cannot make this claim because test-set variation is uncontrolled; a synthetic benchmark would provide the causal evidence the paper's conceptual argument requires.

Data scaling laws for latent dynamics code quality. The paper trains on all available data (~7 hours for Video-CraftBench, 1.3M trajectories for OpenX) without reporting how performance varies with dataset size. A straightforward but important follow-up would train VideoWorld 2 on random subsets of Video-CraftBench (1%, 5%, 10%, 25%, 50%, 100% of the ~9.5k clips) and CALVIN (subsets of the 22k trajectories) and report sequential success rate as a function of training video hours. This would answer the practical question: how much video data does a new task require? The gentle slope from craft-only to craft+OpenX (Table 1, rows 9 vs. 15: 68.8% → 72.3%) suggests data beyond ~7 hours provides diminishing returns, but the shape of the curve below 7 hours is unknown. If performance saturates at, say, 3 hours, the approach is far more practical than if it requires the full 7. A steep scaling curve would motivate collecting larger datasets; a flat curve would suggest the bottleneck is architectural rather than data-limited.

Combining dLDM latent codes with closed-loop policy execution in real robotics. The current CALVIN evaluation is offline: pretrain on latent codes, fine-tune on action labels to produce an open-loop policy, evaluate in simulation. The paper does not investigate whether dLDM latent codes can be used for closed-loop control—predicting the next latent code given the current observation, executing the corresponding action, observing the outcome, and replanning. A follow-up could modify the AR transformer to operate in a closed-loop regime: at each timestep, condition on the full observation history rather than just the initial frame, predict the next latent code, decode it to an action (via the fine-tuned action head), execute in the simulator, and repeat. This would test whether the dLDM's multi-step temporal modeling (93-frame context) provides benefits over single-step latent action models (LAPA, Moto) in closed-loop settings where the model can observe and correct errors. The paper's finding that VideoWorld 2 outperforms LAPA on CALVIN in the offline setting (Table 2, rows 3 vs. 4: 20.4% vs. 15.3% step-4 success) suggests the multi-step codes capture planning information that could improve closed-loop execution, but this is untested.

Stress-testing the approach with VDM failures on out-of-distribution dynamics. The paper shows that the VDM must be fine-tuned to adapt its appearance priors to handicraft videos (Table 3f: frozen VDM → 31.7%, LoRA → 50.9%, full fine-tuning → 68.8%). A deliberate negative result experiment would test the limits of this adaptation: train VideoWorld 2 on handicraft tasks, then test on tasks with qualitatively different dynamics that the VDM has also never seen (e.g., cooking videos involving liquid pouring and state changes, or sports videos with rapid motion and complex physics). If performance degrades sharply despite the VDM having strong appearance priors, this would define a boundary condition: the approach works when the VDM can adapt its rendering to the target domain's visual style, but fails when the required visual dynamics are too far from the VDM's pretraining distribution. This would inform whether future work should focus on improving VDM generality (larger pretraining corpora, more diverse video sources) or on making the dLDM more robust to VDM limitations (e.g., through architectural changes that reduce dependence on VDM quality).

Alternative VDM backbones and their effect on disentanglement quality. The paper uses Cosmos DiT 2B exclusively. A systematic comparison across VDM backbones (Wan2.2, HunyuanVideo, Sora if available, or smaller VDMs at 500M and 1B parameter scales) would reveal whether disentanglement quality depends on VDM capacity, architecture, or pretraining data distribution. The key measurement would be task success rate on Video-CraftBench with identical dLDM training but different VDM backbones, plus the probing-classifier disentanglement metrics described above. If performance is insensitive to VDM choice (all strong VDMs work equally well because they all successfully offload appearance), the approach is robust and can adopt future VDM improvements without redesign. If performance varies substantially (some VDMs fail to provide clean appearance separation, causing latent codes to re-entangle), then VDM selection is a critical hyperparameter that requires guidelines—and the probing metrics would help predict which VDMs will transfer well before full task training.

Practical Applications and Downstream Use Cases

Automatic generation of task demonstration videos in varied visual contexts. The most direct application of VideoWorld 2 as presented: given a single photograph of a new environment (a specific desk with particular paper and lighting), generate a complete tutorial video showing how to fold a paper airplane or boat in that exact setting. The paper demonstrates this capability qualitatively (Figures 2, 11) and quantitatively (68.8–72.3% sequential success on paper folding, 74.0–85.8% on block stacking). For content creation platforms, e-learning systems, or instructional design, this enables generating thousands of tutorial variations from a single recorded demonstration, each adapted to different visual contexts—different materials, backgrounds, lighting, camera angles. The 68.8% success rate means roughly 2 in 3 generated videos correctly execute the full task; with 3 rollouts per test case (the paper's evaluation protocol), the probability of at least one successful video is approximately 1 – (1 – 0.688)3 ≈ 97% (assuming independent trials), making the system practically useful for batch generation where a human curator selects the successful outputs. The visual quality metrics (LPIPS 0.205, SSIM 0.770) indicate that generated videos are perceptually similar to real recordings, suitable for direct human consumption.

Data-efficient robot policy learning through latent dynamics pretraining. The CALVIN results (Table 2) demonstrate a concrete deployment scenario: pretrain an AR transformer on dLDM latent codes from unlabeled robot demonstration videos (22k CALVIN trajectories or 1.3M OpenX trajectories), then fine-tune on a small number of action-labeled trajectories (2k for the 10% setting, 22k for the full setting). In the most data-efficient regime (pretrain on in-domain latent codes, fine-tune on 2k labels), VideoWorld 2 achieves 75.8% first-task success and 1.87 average completed tasks, compared to 50.5% and 1.11 without pretraining—a ~60% improvement in data efficiency. In the most capable regime (pretrain on OpenX latent codes, fine-tune on 22k CALVIN labels), it achieves 88.5% first-task success and 2.88 average length, surpassing the model trained from scratch on 22k action labels (80.9%, 2.36). For robotics practitioners, this means: if you have a new robot or a new set of tasks, you can significantly reduce the number of teleoperated demonstrations needed by first pretraining on latent codes from available unlabeled video data—either from your own robot doing other tasks, or from other robots in other environments—and then fine-tuning on a modest number of labeled trajectories. The fact that cross-embodiment pretraining (OpenX → CALVIN) outperforms in-domain action-labeled training is the key practical insight: diverse video data from other robots is more valuable for your robot's policy than additional labeled data from your own robot.

Large-scale video data mining for manipulation skill libraries. The finding that similar latent codes emerge from visually different domains (Figure 5: similar codes for "reaching forward" in OpenX robot videos and Video-CraftBench handcraft videos) suggests that VideoWorld 2's dLDM can function as a manipulation skill detector on uncurated internet video. A practical pipeline: crawl a large corpus of first-person video (YouTube tutorials, egocentric video datasets), extract dLDM latent codes from short clips, cluster the codes, and identify clusters that correspond to reusable manipulation primitives (grasping, folding, pushing, stacking). These primitives could serve as a skill library for downstream task learning—when a new task is specified, its video demonstrations can be decomposed into known primitives, and the AR transformer can be pretrained on those primitives before fine-tuning on task-specific sequences. The paper provides initial evidence for this in Figure 5, where clips with similar motion patterns across domains produce similar latent codes, but does not scale this analysis. A production system would require processing millions of video clips and would benefit from the fact that dLDM training requires no labels—only raw video.

When to Prefer This Method

The paper positions VideoWorld 2 against two alternative paradigms—end-to-end video generation models and pairwise latent action models—with clear empirical comparisons in Table 1 and the horizon length ablation (Table 3e). The decision rule that emerges from the results is:

• Prefer VideoWorld 2 over end-to-end video generation models (Cosmos, Wan, HunyuanVideo) when: the task requires multi-step sequential execution where each step depends on the precise outcome of prior steps. Evidence: video generation models achieve 0% success by step 5 of paper folding despite strong step-1 performance (68–82%), while VideoWorld 2 achieves 68.8% at step 7. The failure mode is deterministic collapse, not gradual degradation—indicating a qualitative inability to maintain state across steps rather than a capacity limitation that could be overcome with more scale. If your task involves more than 3–4 causally dependent steps, end-to-end video generation is likely insufficient regardless of model size, based on the evidence from four different models at 2B–14B parameters. The paper also provides this method with step-by-step language descriptions, a capability VideoWorld 2 does not require—so if language annotations are expensive or unavailable for your domain, the preference is even stronger.

• Prefer VideoWorld 2 over pairwise latent action models (LAPA, Moto, AdaWorld, CoLA) when: the task requires long-horizon planning (more than a few steps). Evidence: the horizon length ablation (Table 3e) shows pairwise modeling (T=2) achieves 19.1% paper folding success versus 68.8% with 93-frame context, and LAPA/AdaWorld achieve 0% at step 7 of paper folding (Table 1). The multi-step temporal context of the dLDM's query-based encoder captures relational action semantics that pairwise models cannot—what makes an action "folding the corner to the center" rather than "moving the hand to the corner" depends on subsequent states. If your task can be decomposed into relatively independent short actions (grasping, toggling, single-step pushing), the simpler pairwise models may suffice and have lower computational requirements. If your task requires chaining many actions where each depends on the cumulative outcome of prior actions, the multi-step modeling is necessary, and the performance cliff in Table 3e (19.1% → 55.4% → 68.8%) shows that even modest additional temporal context (T=9) provides large gains over pairwise.

• Prefer VideoWorld 2 for cross-embodiment transfer when: you have access to large-scale video data from different agents or environments but limited labeled data from your target setup. Evidence: pretraining on OpenX latent codes (1.3M trajectories from diverse robots) and fine-tuning on CALVIN action labels (22k trajectories) achieves 88.5% first-task success and 2.88 average length, outperforming an oracle trained from scratch on the same 22k CALVIN labels (80.9%, 2.36). The approach is specifically effective when the source and target domains differ in visual appearance, embodiment, or task specification—the appearance-dynamics disentanglement means the pretrained latent codes abstract away domain-specific visual details and encode manipulation structure that transfers. If you are training a new robot from scratch with no external video data, the approach still provides benefit (in-domain latent pretraining on your own unlabeled video before action fine-tuning, Table 2 row 4 versus row 2), but the largest gains come from cross-embodiment transfer. The video next-token prediction baseline (pretraining on raw OpenX videos, Table 2 row 5) also shows positive transfer but significantly less than latent code pretraining (2.46 vs. 2.88 average length), indicating that the latent representation itself—not just the pretraining scale—drives the transfer benefit.