Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

🎯 Pitch

You can turn a pretrained video generation model into a state-of-the-art robot policy with zero architectural changes—just fine-tune it on robot data and treat actions as extra frames in the video. Cosmos Policy not only reaches 98.5% on LIBERO but also uses its own rollout data to build a world model, enabling test-time planning that boosts hard bimanual tasks by over 12 points.

1. Executive Summary

This paper introduces Cosmos Policy, a method for adapting a large pretrained video generation model (Cosmos-Predict2-2B) into a robot visuomotor policy through a single stage of fine-tuning on target-platform demonstration data, with no architectural modifications whatsoever. The core mechanism is latent frame injection — encoding non-image modalities (robot actions, future proprioception, and state values) directly as latent frames within the video diffusion model's native denoising sequence — enabling the model to jointly serve as policy, world model, and value function under a unified diffusion learning objective. Evaluated across the LIBERO and RoboCasa simulation benchmarks and real-world bimanual ALOHA manipulation tasks, Cosmos Policy achieves a new state-of-the-art 98.5% average success rate on LIBERO (outperforming fine-tuned VLAs including π0.5 and CogVLA), 67.1% on RoboCasa (surpassing prior methods while using 6× fewer demonstrations — 50 versus 300+), and a 93.6% average score in real-world evaluations. Further, by collecting policy rollout data and refining world model and value function predictions, Cosmos Policy enables model-based planning via best-of-N sampling that yields an additional 12.5-point average improvement on the most challenging bimanual tasks, establishing that video foundation model priors provide a strong initialization for control policies even without large-scale robotic action pretraining, and that test-time search over learned dynamics can meaningfully improve task completion rates given a modest amount of on-policy experience data.

2. Context and Motivation

The Core Problem: How Do You Extract Useful Robot Policies from Pretrained Video Models?

The fundamental question this paper tackles is deceptively simple: if you have a large, pretrained video generation model that understands how the physical world evolves over time, how do you turn it into a robot policy that actually works — without destroying what made it useful in the first place? This matters because the robotics field is in an unusual position. On one hand, there is an enormous, rapidly improving class of video foundation models (Cosmos, Wan, CogVideoX, Vidu, HunyuanVideo) that have been trained on millions of Internet videos and demonstrate a remarkable ability to generate physically plausible, temporally coherent sequences — suggesting they have internalized useful knowledge about object permanence, contact dynamics, spatial relationships, and action-consequence patterns. On the other hand, the standard approach in robot learning — training a policy from scratch on a few hundred demonstrations for a specific manipulation task — captures none of this general knowledge.

The gap between these two paradigms is significant for several concrete reasons the paper identifies (Section 1). First, data efficiency: robot demonstrations are expensive to collect. A typical ALOHA bimanual task might provide only 15–80 demonstrations covering 50–100 seconds of interaction each. If a video model's pretrained priors about how the world works can be leveraged, a policy should require fewer demonstrations or achieve higher performance from the same data — a claim the paper directly tests by training on only 50 demonstrations per task in RoboCasa when prior methods used 300+. Second, generalization: video models trained on diverse Internet data should, in principle, generalize better to novel object instances, camera viewpoints, and scene layouts that were never seen in the robot's narrow demonstration distribution. Third, multi-task capability: a single video model can serve as the backbone for many different manipulation skills without per-task architectural redesign, suggesting a path toward general-purpose robot foundation models.

Why Prior Approaches Fall Short

The paper positions itself against three families of existing methods, each with distinct limitations:

Multi-stage video-to-policy pipelines introduce complexity. Several prominent recent works (Video Policy by Liang et al., 2025; FlowVLA by Zhong et al., 2025; Video Prediction Policy by Hu et al., 2024; Genie Envisioner by Liao et al., 2025; UniFoLM-WMA by Unitree, 2025; ViDAR by Feng et al., 2025; RoboEnvision by Yang et al., 2025; Latent Policy Steering by Wang et al., 2025) follow a two-phase recipe: first fine-tune a video model on robot data to generate future frames, then train a separate action module — an action diffuser, an inverse dynamics model mapping video frames back to actions, or a learned action decoder — that sits on top of or alongside the video model. While effective, this approach has several drawbacks the paper flags (Section 2):

"these works often require multiple training stages (e.g., video fine-tuning followed by action module training) and introduce new architectural components, such as separate action diffusers or inverse dynamics models"

Each stage requires its own training pipeline, hyperparameter tuning, and computational budget. More importantly, the separation forces the action module to reconstruct the relationship between visual dynamics and robot commands from scratch, discarding any latent knowledge the video model may have about how actions produce visual changes. The video model becomes a glorified frame predictor rather than a core reasoning engine for action selection.

Unified video-action models sacrifice pretrained priors. An alternative line of work — UVA (Li et al., 2025a) and UWM (Zhu et al., 2025) — trains models that jointly predict future frames and actions in a single architecture, avoiding the multi-stage complexity. However, the paper observes a critical limitation:

"these approaches do not leverage pretrained video models and thus do not benefit from their spatiotemporal priors"

Because these unified architectures are custom-designed from scratch (often with novel tokenization schemes or specialized attention patterns to handle the video-action coupling), they cannot simply load weights from a pretrained Cosmos or Wan model. All parameters must be learned from the available robot demonstration data alone — which, as noted above, is typically in the hundreds of examples rather than the millions that went into video model pretraining. The authors explicitly argue this is a missed opportunity: the video model's core learning algorithm and architecture have already been optimized to represent complex, high-dimensional, multimodal distributions and to generate temporally coherent sequences with hundreds of frames. These capabilities are exactly what a policy needs — why discard them?

Vision-language-action models have different priors. VLAs such as RT-2 (Brohan et al., 2023), OpenVLA (Kim et al., 2024), π0.5 (Intelligence et al., 2025), UniVLA (Bu et al., 2025), and CogVLA (Li et al., 2025b) fine-tune large vision-language models (pretrained on static image-text pairs from the Internet) on robotic interaction data. These models bring strong semantic understanding — they know what a "bowl" looks like and what the instruction "put the candy in the ziploc bag" means — and have achieved impressive results on diverse manipulation benchmarks. However, the paper points out a fundamental difference in the type of prior knowledge:

"they leverage pretrained models that have mostly been trained on static image-text pairs rather than videos. In contrast... we leverage a pretrained video model that has learned spatiotemporal dynamics and implicit physics from predicting future frames"

A static image of a hand reaching toward a ziploc bag tells the model about objects and their identities; a video of the same interaction teaches how the bag's opening deforms under force, how the slider moves relative to the bag body, how the hand's approach angle predicts grasp success, and the temporal causality connecting actions to visual outcomes. The paper hypothesizes that for low-level manipulation control — where millimeter-level precision, contact dynamics, and temporal coherence matter more than open-vocabulary semantic understanding — video model priors may be more useful than VLM priors. This is a substantive claim that goes against the dominant VLA trend in the field.

Where World Models and Value Functions Fit In

The paper also situates itself within the long tradition of model-based reinforcement learning and planning. Works such as Dyna (Sutton, 1991), MBPO (Janner et al., 2019), TD-MPC (Hansen et al., 2022; 2023), and the Dreamer family (Hafner et al., 2019; 2020; 2023) have established that learning a dynamics model — a world model that predicts future states given current states and actions — can dramatically improve sample efficiency and final performance over purely model-free approaches. These methods typically train separate, purpose-built modules for the policy, the dynamics model, and the value/reward function.

More recent work applies these ideas in robotics settings, but the paper identifies a consistent architectural pattern it aims to break:

"In contrast to these prior works that rely on separate modules for the policy, world model, and value function and typically train models from scratch, we use a single unified architecture that serves simultaneously as the policy, world model, and value function and initialize from a pretrained video model"

Specifically, FLARE (Zheng et al., 2025) adds learnable "future tokens" to a diffusion transformer to predict compact future state representations but still requires a separate action prediction mechanism. SAILOR (Jain et al., 2025) combines separate world and reward models with Model Predictive Path Integral (MPPI) optimization to iteratively search for better actions. Latent Policy Steering (Wang et al., 2025) pretrains a world model using optical flow as an embodiment-agnostic action representation, then trains a standalone value function to steer the policy — three separate models in sequence.

The modular approach has advantages (component-level interpretability, independent training schedules), but the paper argues it also has costs: engineering complexity (three models to train, tune, and maintain), the risk of compounding errors when chaining separately trained modules, and — most critically for data-scarce robotics — the inability to share representations across policy, dynamics, and value learning. A unified architecture that learns all three functions simultaneously from the same data can, in principle, learn representations that are useful for all of them, making better use of every demonstration.

How Cosmos Policy Positions Itself

The paper frames Cosmos Policy as a synthesis that addresses the limitations of each prior approach while combining their strengths:

Against multi-stage video pipelines: Cosmos Policy requires a single stage of post-training. There is no separate action module to train, no inverse dynamics model to learn, and no pipeline of fine-tuning-then-action-training. The model is fine-tuned end-to-end on robot demonstration data, and at inference time it directly produces actions (alongside future states and values) from the same denoising process.

Against unified video-action models trained from scratch: Cosmos Policy does leverage a pretrained video model. The entire Cosmos-Predict2-2B checkpoint is loaded as initialization, and all components — the VAE tokenizer, the diffusion transformer backbone, the text conditioning pathway, the EDM noise schedule — are preserved. The paper's key design choice is to avoid any architectural modification that would require discarding pretrained weights.

Against VLAs: Cosmos Policy substitutes the static-image-text prior with a spatiotemporal video prior. The paper does not claim this is universally better — VLAs likely retain advantages for tasks requiring open-vocabulary instruction following or semantic reasoning — but it hypothesizes advantages for the precise, temporally extended manipulation tasks that dominate current benchmarks.

Against modular world-model approaches: Cosmos Policy uses a single architecture for everything. The policy, world model, and value function are literally the same model with the same weights; the only difference during training is which latent frames are provided as clean conditioning versus which are corrupted with noise and need to be denoised. This is the "latent frame injection" mechanism described in the executive summary and detailed in Section 4.1.

The novel contribution is in the simplicity of the adaptation mechanism, not in the idea of using video models for control. The paper is careful to acknowledge that prior work has used video models for robotics (Section 2 catalogs at least eight such works). The contribution is the specific claim that no architectural changes are needed — you can encode actions, proprioception, and scalar values as if they were additional video frames, interleave them with real camera images in the latent diffusion sequence, and fine-tune the model to generate them. This is a strikingly minimalistic approach to a problem that others have solved with substantially more engineering complexity.

The Practical Bet: Simplicity Enables Reliability and Reproducibility

There is a subtext throughout the paper that deserves attention. By avoiding architectural modifications, Cosmos Policy inherits not just the pretrained weights but also the infrastructure around the base model: the efficient training code, the optimized inference pipeline, the hyperparameter defaults, the noise schedule, the data preprocessing. This has practical consequences that the paper's results bear out: the method trains on 50 RoboCasa demos and outperforms methods trained on 300+; it achieves state-of-the-art results on LIBERO while fine-tuned VLAs require large-scale robotic action pretraining; and it enables planning with only a few hundred additional rollout trajectories. The implicit argument is that architectural simplicity — removing degrees of freedom rather than adding them — is itself a design principle that leads to better empirical outcomes when data is scarce.

The paper's positioning can be summarized as: video models already know how to generate coherent sequences conditioned on an initial frame and a text description; if we can express robot control as exactly that problem — with actions, future observations, and values just being more "frames" in the sequence — we get policy learning, world modeling, and value estimation essentially for free, with no new architecture needed. The experiments are designed to test whether this remarkably simple hypothesis actually holds up against the more complex, multi-stage, multi-module alternatives that currently dominate the field.

3. Technical Approach

3.1 Reader Orientation

This paper builds a system that takes a pretrained video generation model and turns it into a robot policy — a program that looks at camera images, reads the robot's joint positions, and outputs the commands that move the robot to accomplish a task — without adding any new neural network components, without retraining the visual tokenizer, and without changing the diffusion training objective. The core problem it solves is how to inject information that video models don't natively handle (robot joint angles, Cartesian actions, scalar success probabilities) into a model that only knows how to generate pixels, doing so in a way that preserves all the spatiotemporal knowledge the model learned from millions of Internet videos.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, all sharing a single set of neural network weights:

The pretrained video backbone (Cosmos-Predict2-2B) — a latent diffusion transformer that takes a starting image and a text description and generates a video by iteratively denoising Gaussian noise in a compressed latent space. It uses the Wan2.1 spatiotemporal VAE to compress images into latents and the EDM (Elucidating Diffusion Models) framework for the denoising objective.
The latent frame injection mechanism — a zero-architecture-change method for representing non-image modalities (robot proprioception, action chunks, scalar values) as if they were additional video frames in the diffusion sequence. Scalar and vector data are normalized, duplicated to fill a latent volume, and overwrite placeholder frames in the sequence.
The joint training scheme — a batch-splitting protocol where 50% of each training batch trains the model as a policy (predict actions, future state, and value from current observations), 25% trains it as a world model (predict future state and value from current observations and actions), and 25% trains it as a value function (predict value from current observations, actions, and future state). Which function is being trained depends solely on which latent frames are provided as clean conditioning and which are noised targets.
The planning deployment — after initial training, the policy is deployed to collect rollouts (both successes and failures), these rollouts are used to fine-tune a separate copy of the model with heavy emphasis on world model and value function accuracy, and at test time the original checkpoint proposes actions while the fine-tuned checkpoint scores them by predicting their outcomes, enabling best-of-N search.

Information flows as follows: multi-view camera images and blank placeholder images are VAE-encoded into a latent sequence → non-image modalities are injected by overwriting the placeholder latents → portions of the sequence are corrupted with noise → the diffusion transformer denoises the corrupted portions conditioned on the clean portions → the denoised latents are extracted and un-normalized to produce actions, future observations, and values.

3.3 Roadmap for the Deep Dive

First, the pretrained Cosmos-Predict2 model — its architecture, latent space, noise distribution, and training objective — because everything in Cosmos Policy inherits from this base model and the design constraint of "no architectural changes" only makes sense relative to what the base model already does.
Second, latent frame injection — the core mechanism that encodes non-image modalities into the latent diffusion sequence — because it is the linchpin that enables the model to handle actions, proprioception, and values without new architectural components.
Third, the joint training objectives — how batch splitting and latent masking train the policy, world model, and value function simultaneously — because this explains how a single model learns to serve three distinct functions.
Fourth, the noise distribution modifications — the change from a log-normal to a hybrid log-normal-uniform noise schedule and the adjusted sampling bounds — because test-time precision for actions requires different noise characteristics than video generation.
Fifth, the planning mechanism — how rollout data collection, fine-tuning, and best-of-N search combine to improve task performance — because this is where the world model and value function predictions become actionable.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that a pretrained video diffusion model can be adapted into a unified policy–world-model–value-function architecture through a single mechanism — encoding all modalities as latent frames — with no architectural changes, enabling strong imitation learning performance and test-time planning.

The Pretrained Cosmos-Predict2 Video Model

Cosmos Policy is initialized from Cosmos-Predict2-2B-Video2World (NVIDIA et al., 2025), a latent video diffusion model. Understanding its architecture and training objective is essential because every design decision in Cosmos Policy — latent frame injection, noise schedule modification, conditioning schemes — is shaped by the constraints and affordances of this base model.

Input-output specification. The base model takes two inputs: a single starting image (the first frame of a video it will generate) and a textual description encoded as T5-XXL embeddings (Raffel et al., 2020). Given these, it generates a short video — a sequence of subsequent frames that continue the scene depicted in the first image in a temporally coherent and text-consistent manner.

Latent space via VAE tokenizer. The model does not operate directly on pixel-space images. Instead, all images are compressed by the Wan2.1 spatiotemporal VAE tokenizer (Wan et al., 2025). This tokenizer maps a video clip of size $(1 + T) \times H \times W \times 3$ (one starting image plus $T$ generated frames, each of spatial resolution $H \times W$ with 3 RGB channels) into a latent sequence of size $(1 + T') \times H' \times W' \times 16$ , where:

$T' = \frac{T}{4}, \quad H' = \frac{H}{8}, \quad W' = \frac{W}{8}$

where $T$ is the number of generated frames, $T'$ is the number of temporally compressed latent groups, $H$ and $W$ are pixel height and width, $H'$ and $W'$ are latent spatial dimensions, and 16 is the channel dimension of each latent frame.

What this computes: a dimensionality reduction that compresses the video along all three axes — 4× temporal compression (every 4 input frames become 1 latent group), 8× spatial compression in height and width, and an expansion from 3 color channels to 16 learned latent channels. The net effect is a massive reduction in the number of tokens the transformer must process while preserving sufficient information for high-quality frame reconstruction.

Why this form: the temporal compression factor of 4 (rather than 1 or 8) balances computational efficiency against the need to represent rapid motion. The first frame undergoes no temporal compression — it is encoded as a standalone latent frame — to allow clean conditioning on a single input image without temporal blur. This detail matters for Cosmos Policy because it creates an asymmetry in the latent sequence structure: the first latent frame corresponds to exactly one image, while subsequent latent frames each correspond to groups of four images. The spatial compression factor of 8 is standard for image VAEs in this parameter regime, providing roughly 192× total compression per frame (8 × 8 × 3 to 16 channels) while preserving fine visual details needed for manipulation tasks.

Diffusion framework (EDM formulation). The model is trained using the EDM denoising score matching formulation (Karras et al., 2022). The core training objective for the denoiser network $D_\theta$ at noise level $\sigma$ is:

$L(D_\theta, \sigma) = \mathbb{E}_{x_0, c, n}\left[ \lVert D_\theta(x_0 + n; \sigma, c) - x_0 \rVert_2^2 \right]$

where $x_0$ is a clean VAE-encoded image sequence (the target the model should produce), $c$ represents the textual description encoded as T5-XXL embeddings and injected via cross-attention, $n \sim \mathcal{N}(0, \sigma^2 I)$ is i.i.d. Gaussian noise used to corrupt $x_0$ , and $D_\theta$ is a diffusion transformer that learns to recover the clean sample $x_0$ given the noise-corrupted version $x_0 + n$ , the noise level $\sigma$ , and the text conditioning $c$ .

What it computes: for a given noise level $\sigma$ , the model is shown a corrupted latent sequence (clean latents plus Gaussian noise scaled by $\sigma$ ) and must predict the clean latents. The loss is the mean squared error between the prediction and the ground-truth clean latents, averaged over training samples, noise realizations, and text conditions. At test time, this denoising capability is used iteratively: starting from pure noise ( $\sigma = \sigma_{\text{max}}$ ), the model progressively removes noise over multiple steps as $\sigma$ decreases toward $\sigma_{\text{min}}$ , eventually producing a clean latent sequence.

Why this form: the EDM formulation parameterizes the denoising objective directly in terms of predicting the clean data $x_0$ rather than predicting the noise $\epsilon$ (as in the DDPM formulation) or the score $\nabla_x \log p(x)$ (as in score-based models). The paper inherits this choice from the base Cosmos-Predict2 model. The $\ell_2$ loss on the clean prediction is standard and well-behaved. The expectation over $x_0$ , $c$ , and $n$ means the model sees a wide range of noise levels during training, learning to denoise at all stages of the diffusion process.

Transformer architecture. $D_\theta$ is a diffusion transformer (Peebles & Xie, 2023). It conditions on the text embeddings $c$ via cross-attention layers interleaved throughout the transformer blocks and on the noise level $\sigma$ via adaptive layer normalization (FiLM conditioning; Perez et al., 2018; Peebles & Xie, 2023) — the noise level modulates the scale and shift parameters of the normalization layers, allowing the model to behave differently at different stages of the denoising process.

Conditioning mask during training. During training, a conditioning mask ensures that the first latent frame (corresponding to the input starting image) remains clean — it is never corrupted with noise — while subsequent frames are corrupted and must be denoised. This teaches the model to generate videos conditioned on a clean first frame, which is the standard video prediction setup. For Cosmos Policy, this conditioning mask mechanism is generalised: during training, different subsets of latent frames are kept clean depending on which function (policy, world model, or value function) is being trained.

Latent Frame Injection: Encoding Non-Image Modalities

This is the paper's central technical mechanism. The base Cosmos-Predict2 model operates on a sequence of latent frames of shape $(1 + T') \times H' \times W' \times 16$ , where each frame originally corresponds to a VAE-compressed image. Cosmos Policy needs to handle additional modalities that are not images: robot proprioception (a vector of joint angles or end-effector poses), action chunks (a sequence of robot commands spanning $K$ future timesteps), and state values (a scalar between 0 and 1 representing expected task completion probability). It also needs to handle multiple camera viewpoints — up to three in the ALOHA setup (one top-down, two wrist-mounted).

The core idea: treat everything as a latent frame. Rather than adding new input heads, output heads, or auxiliary networks, the paper proposes to represent non-image modalities by filling the $H' \times W' \times C'$ (where $C' = 16$ for the Wan2.1 VAE) volume of a latent frame with normalized copies of the modality data, and to insert these synthetic latent frames at specific positions in the diffusion sequence, interleaved with real image latents. The model then learns to condition on and generate these injected latents through the exact same denoising objective it was pretrained with.

Latent injection for a concrete example (two static third-person cameras plus one wrist camera). The full latent sequence contains 11 latent frames, ordered as follows:

A blank (all-zero) placeholder — necessary due to the VAE's temporal compression scheme (explained below).
Robot proprioception at time $t$ (e.g., 14 joint angles for the ALOHA robot).
Wrist camera image at time $t$ .
First third-person camera image at time $t$ .
Second third-person camera image at time $t$ .
Action chunk — the $K$ actions the robot will execute (where $K$ is the action chunk size; $K=50$ for ALOHA, $K=16$ for LIBERO, $K=32$ for RoboCasa).
Future robot proprioception at time $t + K$ (the predicted joint configuration after executing the action chunk).
Future wrist camera image at time $t + K$ .
Future first third-person camera image at time $t + K$ .
Future second third-person camera image at time $t + K$ .
Future state value $V(s_{t+K})$ (a scalar representing expected cumulative reward from the future state).

Among these, positions 2, 6, 7, and 11 are new non-image modalities, while positions 3, 5, 8, and 10 are additional camera views (assuming position 4 — the first third-person camera — is the "primary" camera that the base model was originally designed to handle).

Why this ordering: the sequence represents the tuple $(s, a, s', V(s'))$ , arranged left-to-right. This is a causal ordering: the current state $s$ (observations at time $t$ ) comes first, then the action $a$ (what the robot does between $t$ and $t + K$ ), then the future state $s'$ (observations at time $t + K$ ), and finally the value $V(s')$ (the expected outcome). This left-to-right causal structure enables autoregressive decoding during planning: the model can first generate actions conditioned on current state, then generate future state conditioned on current state and actions, then generate value conditioned on current state, actions, and future state. The interleaving of multiple camera views within both the current and future state groups is handled simply by inserting additional image latent frames at the image sequence level — the VAE tokenizer compresses each view independently.

How the blank placeholder works. The first latent frame (position 1) is a blank frame, not a real observation. This is an implementation detail forced by the Wan2.1 VAE's temporal compression scheme: the VAE encodes the first image of a video alone (no temporal compression) but groups subsequent images into chunks of 4 for temporal compression. The resulting latent sequence therefore has an "unpaired" first latent frame that corresponds to a single image, followed by latent frames that each correspond to groups of 4 images. To ensure that current-timestep observations and future-timestep observations have similarly structured latent representations, the paper places them after this blank first latent frame. Additionally, to have exactly one latent frame per modality and camera viewpoint (rather than the 1:4 mapping that temporal compression would impose on image frames), the paper constructs four identical copies of each image in the input sequence — so that each group of four identical images is compressed into a single latent frame representing exactly one timestep, not four.

The injection operation in detail. Consider injecting an action chunk of shape $K \times d_{\text{act}}$ (50 timesteps × 14 DoF for ALOHA, for example). The procedure is:

Normalize: rescale each action dimension to the range $[-1, +1]$ using dataset statistics.
Flatten: reshape the action chunk from a $(K \times d_{\text{act}})$ matrix to a single vector of length $K \cdot d_{\text{act}}$ .
Duplicate: replicate this vector $\frac{H' \times W' \times 16}{K \cdot d_{\text{act}}}$ times, where $(H', W', 16)$ is the shape of a single latent frame.
Reshape: reshape the duplicated vector into a $(H' \times W' \times 16)$ volume, matching the latent frame dimensions.
Overwrite: replace the target latent frame (position 6 in the sequence above) with this constructed volume.

The same procedure applies to robot proprioception (a vector of joint angles, shape $d_{\text{prop}}$ ) and the value (a scalar, shape 1), with the duplication factor adjusted accordingly. For the value, which is a single floating-point number, the duplication factor is simply $H' \times W' \times 16$ , meaning the scalar is broadcast across the entire latent volume.

Why duplication works. This seems wasteful — why repeat the same information thousands of times in a high-dimensional volume? The answer has to do with how the diffusion transformer processes the latent sequence. Each latent frame is tokenized into a grid of patches (standard for vision transformers), and the self-attention mechanism operates across all patches from all frames. By duplicating the action vector across the spatial dimensions of the latent frame, every patch in that frame carries the same action information, making it available to every attention head regardless of which spatial positions it attends to. The alternative — placing the action vector in a single patch and leaving the rest of the frame blank — would mean most patches carry no information and the model would need to learn to attend specifically to the one patch that does, which is less robust. The duplication approach effectively turns the latent frame into a "broadcast channel" for the non-image modality.

Flexibility across robot platforms. The latent injection scheme is modular: for a robot with only one third-person camera, one would simply remove the latent frames corresponding to additional camera viewpoints (positions 5 and 10 in the example above), resulting in 7 total latent frames instead of 11. For a robot with different proprioception dimensionality or a different action space, only the duplication factors change — the injection mechanism itself remains identical. This means the same pretrained Cosmos-Predict2 checkpoint can be adapted to different embodiments without any model architecture changes, only by adjusting the composition of the latent sequence.

Extraction at inference time. At test time, after the model has generated clean (denoised) latent frames, the non-image modalities must be extracted back into their original form. This is the reverse of the injection process:

Average: for the action chunk, compute the mean value across all $\frac{H' \times W' \times 16}{K \cdot d_{\text{act}}}$ duplicated copies in the latent volume, recovering a single vector of length $K \cdot d_{\text{act}}$ .
Reshape: reshape this vector back into a $(K \times d_{\text{act}})$ matrix.
Un-normalize: rescale from $[-1, +1]$ back to the original action range.

For the value, since it is a scalar, the extraction is simply averaging over the entire latent volume and un-normalizing to $[0, 1]$ . Crucially, extraction of non-image modalities does not require VAE decoding — these elements were directly injected into latent space during training, and the model generates them directly in latent space at inference time. Only the image latents (camera views) need to be passed through the VAE decoder to produce pixel-space images if visualization is desired; for policy execution, only the actions need to be extracted.

Joint Training of Policy, World Model, and Value Function

Now that the latent sequence can represent all required modalities, the next problem is: how do you train a single model to simultaneously serve as a policy (predict actions from observations), a world model (predict future observations from current observations and actions), and a value function (predict expected outcomes from full trajectories)? The answer is a batch-splitting protocol with different conditioning masks.

Data sampling per training step. For each training step, the model samples a batch of $(s, a, s', G_t)$ tuples, where $s$ represents the current observations (robot proprioception and multi-view images), $a$ is the action chunk, $s'$ is the future observations after executing the action chunk, and $G_t$ is the empirical return — the observed terminal reward discounted back to time $t$ . In the paper's sparse reward setting, the terminal reward $R(s_H, a_H) \in [0, 1]$ is provided only at the final timestep of a successful episode, and the return is computed as:

$G_t = \gamma^{H-t} R(s_H, a_H)$

where $\gamma$ is a discount factor, $H$ is the episode horizon, and $t$ is the current timestep.

What it computes: the discounted terminal reward backpropagated from the episode's end to time $t$ . If the episode succeeds ( $R = 1$ ), then $G_t = \gamma^{H-t}$ ; if it fails ( $R = 0$ ), then $G_t = 0$ . Since $\gamma < 1$ , states closer to the episode end receive higher returns.

Why this form: in sparse reward settings, there is no per-step reward signal — the only information is whether the task was completed at the end. Discouting provides a continuous-valued target that smoothly increases as the robot gets closer to task completion, which is more informative for learning than a binary success/failure signal at the final step alone. The paper notes that $G_t$ is labeled " $V(s')$ " throughout the text for notational simplicity, but technically the training target is the Monte Carlo return from the actual rollout, not a bootstrapped value estimate.

Batch splitting ratios. The sampled tuples come from two datasets with a 50/50 split:

50% from the demonstrations dataset — expert trajectories collected via teleoperation. These are used to train the policy: given $s$ , predict $(a, s', V(s'))$ . This subset includes only successful demonstrations (or their filtered versions, since some LIBERO and RoboCasa demos fail when replayed due to teleoperation errors — roughly 10–20% of demonstrations — and the paper filters these out for policy training).
50% from the rollouts dataset — trajectories collected by deploying Cosmos Policy (or, initially, this dataset is simply a superset of the demonstrations dataset that also includes failed demonstrations if they exist). This 50% is further split in half:
- 25% for world model training: given $(s, a)$ , predict $(s', V(s'))$ .
- 25% for value function training: given $(s, a, s')$ , predict $V(s')$ .

So the per-batch composition is: 50% policy, 25% world model, 25% value function, with each split using the exact same total latent sequence but different conditioning masks.

Conditioning masks determine which function is trained. The latent diffusion sequence always has the same structure (ordered $s, a, s', V(s')$ ). The conditioning mask — a binary indicator per latent frame specifying whether it is kept clean (provided as conditioning) or corrupted with noise (serving as the target to denoise) — determines which function is being trained:

Policy training ( $p(a, s', V(s') \mid s)$ ): the latent frames corresponding to $s$ (current observations) are kept clean; the latent frames corresponding to $a$ , $s'$ , and $V(s')$ are corrupted with noise and must be denoised by the model. The model learns to generate actions, future state, and future value conditioned only on the current state.
World model training ( $p(s', V(s') \mid s, a)$ ): the latent frames corresponding to $s$ and $a$ are kept clean; the latent frames corresponding to $s'$ and $V(s')$ are corrupted. The model learns to generate future observations and value conditioned on current observations and the action taken.
Value function training ( $p(V(s') \mid s, a, s')$ ): the latent frames corresponding to $s$ , $a$ , and $s'$ are kept clean; only the latent frame corresponding to $V(s')$ is corrupted. The model learns to predict the value given the full trajectory context.

Why auxiliary targets for policy and world model training. The paper makes a deliberate choice to have the policy predict not just actions ( $p(a \mid s)$ ) but also future state and value ( $p(a, s', V(s') \mid s)$ ). Similarly, the world model predicts not just future state ( $p(s' \mid s, a)$ ) but also value ( $p(s', V(s') \mid s, a)$ ). These are called "auxiliary supervision" — the additional targets provide extra training signal that encourages the model to learn representations that are useful across all three functions. The paper's ablation in Table 4 shows that removing these auxiliary targets reduces average LIBERO success rate from 98.5% to 97.0% (a 1.5-point drop), and a further ablation in RoboCasa (Table 5) shows that removing both world model/value function training and auxiliary supervision for the policy — training a barebones $\pi(a \mid s)$ — causes a catastrophic drop from 67.1% to 44.4% average success rate. The future state prediction target is the most critical component: the model learns what will happen as a result of its actions, even when deployed as a direct policy without planning.

Initial rollouts dataset. Initially, the rollouts dataset is simply a superset of the demonstrations dataset that also includes failed demonstrations, if they exist. In the LIBERO and RoboCasa benchmarks, roughly 10–20% of demonstrations fail when replayed due to human error during teleoperation data collection. These failed trajectories are filtered out for policy training but kept for world model and value function training, since they provide information about what not to do. In environments where teleoperation data is collected carefully (such as the real-world ALOHA setup), failed demonstrations do not exist, so the demonstrations dataset and initial rollouts dataset are identical.

Parallel vs. autoregressive decoding. During direct policy evaluation (no planning), the model generates all target latent frames — $(a, s', V(s'))$ — in parallel in a single denoising pass. This is faster because the model predicts everything at once; the actions can be extracted and deployed while the future state and value predictions are simply discarded (or optionally saved for visualization). During planning, higher-quality predictions are needed, so the model generates autoregressively from left to right: first actions conditioned on $s$ , then future state conditioned on $s$ and the generated actions, then value conditioned on $s$ , generated actions, and generated future state. Autoregressive decoding allows each subsequent prediction to be conditioned on the model's own previous outputs, which can capture dependencies more accurately than joint parallel prediction.

Noise Distribution Modifications for Precision Action Generation

The base Cosmos-Predict2 model uses a log-normal noise distribution during training, following the EDM formulation. The paper finds that this distribution is well-suited for video generation but suboptimal for robot action generation, where predictions must be extremely precise — small errors in predicted joint angles or end-effector positions can cause the robot to miss a grasp, collide with objects, or drop items.

The base model's noise distribution. In the EDM framework, the noise level $\sigma$ is sampled from a log-normal distribution during training:

$\ln(\sigma) \sim \mathcal{N}(P_{\text{mean}}, P_{\text{std}}^2)$

where $P_{\text{mean}} = 1.39$ and $P_{\text{std}} = 1.2$ for Cosmos-Predict2-2B.

What this computes: the natural logarithm of the noise level is drawn from a Gaussian with mean 1.39 and standard deviation 1.2. This means $\sigma$ is log-normally distributed: most training samples have relatively small $\sigma$ values (low noise, easy denoising), with a long tail toward larger $\sigma$ values. The modal $\sigma$ is approximately $e^{1.39} \approx 4.0$ , and the distribution concentrates probability mass at low-to-moderate noise levels.

Why this is problematic for action generation. Diffusion generation begins at $\sigma_{\text{max}} = 80$ with pure noise and iteratively denoises toward $\sigma_{\text{min}} \approx 0$ . At each denoising step, the model predicts the clean data at the current $\sigma$ level. If the model was mostly trained on low- $\sigma$ examples, it has insufficient signal at the high- $\sigma$ regime where generation begins, causing poor initial denoising that cascades into errors in later steps. For image and video generation, small inaccuracies may be imperceptible; for robot actions, they can be catastrophic.

The adjusted hybrid distribution. Cosmos Policy uses a hybrid log-normal-uniform distribution:

With probability 0.7, sample from the original log-normal distribution: $\ln(\sigma) \sim \mathcal{N}(1.39, 1.2^2)$ .
With probability 0.3, sample $\sigma \sim \mathcal{U}[1.0, 85.0]$ (uniform).

What this computes: the log-normal distribution with an extended tail at higher noise levels. The 30% uniform component ensures the model sees substantial training at $\sigma$ values above 40–50, which are in the log-normal tail and rarely sampled under the original distribution. This provides the model with more practice denoising at the high noise levels where the generation process must start.

Why this form: the 0.7/0.3 split is chosen to stay close to the original distribution (preserving the pretrained model's existing knowledge) while extending the high- $\sigma$ tail. These probability values were not tuned — the paper states they "were not tuned" — suggesting the benefit is robust to the exact mixing ratio as long as some high-noise training is provided. The uniform distribution over $[1.0, 85.0]$ was chosen because $85.0$ is slightly above the maximum noise level used at test time ( $\sigma_{\text{max}} = 80$ ), ensuring coverage of the full sampling range.

Adjusted sampling bounds at inference time. The base EDM formulation uses $\sigma_{\text{min}} = 0.002$ (nearly zero) and $\sigma_{\text{max}} = 80$ . Cosmos Policy changes the lower bound:

$\sigma_{\text{min}} = 4 \quad \text{(instead of } 0.002\text{)}$ $\sigma_{\text{max}} = 80 \quad \text{(unchanged)}$

What this computes: the denoising process stops at $\sigma = 4$ rather than continuing to near-zero noise. The model never attempts to produce perfectly clean latents at the final step; instead, the last denoising step produces latents with residual noise at $\sigma = 4$ , which are then used directly as predictions.

Why this form: the paper observes empirically that the final denoising steps at very low $\sigma$ (near zero) are less accurate than earlier steps at moderate $\sigma$ . The likely reason is that at $\sigma \approx 0$ , the signal-to-noise ratio is extremely high — the noisy input is almost identical to the clean target — so the model receives very little information about what to denoise. The residual noise from stopping at $\sigma = 4$ acts as a form of implicit regularization that prevents the model from over-fitting to noise artifacts. The paper reports that this higher lower bound "empirically improves prediction accuracy at inference time for actions, future states, and values, as measured by lower L1 loss on training and validation samples."

The Planning Mechanism: Rollout Collection, Fine-Tuning, and Best-of-N Search

Cosmos Policy can be deployed in two modes: as a direct policy (no planning, just execute the generated actions) or with model-based planning (generate multiple candidate action chunks, predict their outcomes, and execute the best one). The planning mode requires an additional data collection and fine-tuning phase.

Why demonstrations alone are insufficient for planning. When trained only on demonstration data, the world model and value function see a narrow distribution: all trajectories are successful (or near-successful), all actions are expert actions, and all outcomes are positive. At test time, when the model proposes candidate actions that deviate from the expert distribution — which is necessary for planning to improve over the base policy — the world model and value function must generalize to states and actions they have never seen during training. The paper argues that this generalization is unreliable: the world model may predict unrealistic future states, and the value function may assign incorrectly high values to poor actions, defeating the purpose of planning.

Rollout data collection. To address this, the paper collects a rollout dataset by deploying Cosmos Policy (the base checkpoint, without planning) in diverse initial conditions and recording the full trajectory along with the episode outcome (success/fail binary or a fractional score). For the real-world ALOHA experiments, the rollout dataset consists of:

505 rollouts collected during the direct policy evaluations (across all policies evaluated, not just Cosmos Policy).
143 additional rollouts from Cosmos Policy specifically for the "put candy in ziploc bag" task.

Total: 648 rollouts. The authors note that the ziploc bag task requires extra rollouts because training an accurate world model for it is "particularly challenging due to low camera observability from the robot's self-occlusion and highly stochastic environment dynamics where even millimeter differences in control can dictate success or failure."

Fine-tuning the planning model. Given this rollout dataset, the base Cosmos Policy checkpoint is fine-tuned with heavily reweighted objectives:

90% of each training batch is split evenly between world model training (45%) and value function training (45%).
10% of each batch is used for policy training.

The policy training ratio is reduced because the goal is to refine the world model and value function, not to change the policy's behavior (which is already strong). The model resulting from this fine-tuning is called the planning model; the original checkpoint is called the policy model. At deployment time, both are used together: the policy model proposes actions, and the planning model scores them.

Why separate policy and planning models. The policy model (trained primarily on demonstrations) generates high-quality actions that follow the expert distribution. The planning model (fine-tuned primarily on rollout data, including failures) has seen a broader state-action distribution and can more accurately predict the consequences of both good and bad actions. Using the policy model for action generation and the planning model for action evaluation decouples the two requirements: the policy should stay close to the expert data, while the world model and value function should generalize beyond it.

Value function conditioning during fine-tuning. During the fine-tuning phase, the value function is trained with an important modification to its conditioning. Recall that during initial training, the value function is conditioned on the full latent prefix $(s, a, s')$ — the model sees current state, action, and future state before predicting value. During fine-tuning, the paper instead masks out $(s, a)$ so that the value prediction is conditioned only on $s'$ :

$V(s')$ formulation (model-based planning): the value is predicted from future state alone. This requires a world model to first predict $s'$ from $(s, a)$ before the value can be estimated. Planning then consists of: sample candidate action $a$ → predict $s'$ using the world model → predict $V(s')$ from the predicted $s'$ → select action with highest predicted value.
$Q(s, a)$ formulation (model-free planning): the value is predicted from current state and action, with $s'$ masked out. This bypasses the world model entirely: sample candidate action $a$ → directly predict $Q(s, a)$ → select action with highest predicted value.

The paper compares these two formulations in the planning experiments (Figure 7) and finds that $V(s')$ (model-based) outperforms $Q(s, a)$ (model-free), attributing the difference to the world model's ability to leverage learned environment dynamics for more sample-efficient planning given limited rollout data.

Best-of-N search procedure. The planning algorithm at test time proceeds as follows:

Generate candidate actions: using the policy model, sample $N$ action chunk proposals conditioned on the current state $s$ . The paper uses $N = 8$ for real-world experiments, with each candidate generated on a separate H100 GPU in parallel.
Predict future states: using the planning model, for each candidate action $a_i$ , generate an ensemble of 3 future state predictions $s'_{i,1}, s'_{i,2}, s'_{i,3}$ . The ensemble improves robustness to the stochasticity of the world model.
Predict values: for each predicted future state $s'_{i,j}$ , generate an ensemble of 5 value predictions $V_{i,j,1}, \dots, V_{i,j,5}$ , resulting in 15 total value estimates per action proposal (3 future states × 5 value samples).
Aggregate via "majority mean": for each action proposal $a_i$ , determine whether the majority of its 15 value predictions are above a fixed success threshold (indicating predicted success) or below it (indicating predicted failure). Then, average the values only within the majority group. If the majority predict success, the final score for $a_i$ is the mean of the success-predicting values; if the majority predict failure, it is the mean of the failure-predicting values.
Select and execute: choose the action $a_i$ with the highest aggregated value score and execute the full action chunk on the robot (no receding-horizon control — the entire chunk is deployed before requerying the policy).

Why majority mean aggregation. Simple averaging of all 15 value predictions is vulnerable to outliers: if even a few predictions are wildly overconfident (e.g., predicting value 1.0 for an action that will fail), the average can be misleadingly high. The majority-mean approach first determines whether the ensemble consensus is that the action will succeed or fail (binary decision via threshold), then averages within the consensus group only. If the values are bimodal — some predictions strongly indicating success and others strongly indicating failure — the majority vote determines which mode is trusted, and outliers are discarded. This is more robust to the high variance and potential bimodality that the paper observes in value predictions.

Inference cost of planning. The paper reports inference latency for planning with $N = 8$ using 8 H100 GPUs:

Action generation: 10 denoising steps per candidate, parallel across GPUs.
Future state prediction: 5 denoising steps per sample × 3 ensemble members × 8 action candidates = 120 total generations, parallelized across 8 GPUs.
Value prediction: 5 denoising steps per sample × 5 ensemble members × 8 action candidates × 3 future states = 600 total generations, parallelized across 8 GPUs.

Total wall-clock time: approximately 4.9 seconds to generate one action chunk. For comparison, direct policy inference (no planning) takes 0.95 seconds for 10 denoising steps on a single H100 GPU for ALOHA (generating actions for 2 seconds of execution). The planning overhead is roughly 5× slower but provides the 12.5-point score improvement shown in Figure 7.

Why the full action chunk is deployed without receding-horizon control. The paper states that they "execute the full action chunk (rather than only part of it, as done in receding-horizon control) to avoid further increases in computational cost." In receding-horizon control, the policy generates an action chunk but executes only the first few actions, then replans. This would multiply the already-expensive planning cost by the number of replanning steps per chunk. Executing the full chunk amortizes the planning cost over more timesteps.

4. Key Insights and Innovations

Innovation 1: Latent Frame Injection as a Zero-Architecture-Change Adaptation Strategy

The paper's most intellectually distinctive contribution is not the idea of using video models for robot control — prior work has done this extensively — but rather the specific claim that no architectural modifications whatsoever are needed. Cosmos Policy adds no new layers, no separate action heads, no inverse dynamics decoders, no auxiliary training objectives beyond the original diffusion loss, and no new input or output pathways. All adaptation happens through a single mechanism: encoding non-image modalities as latent frames that are injected into the video model's native diffusion sequence, then using the exact same denoising objective the model was pretrained with.

What makes this a genuine conceptual advance rather than an engineering trick. The dominant assumption in the video-to-policy literature — spanning Video Policy (Liang et al., 2025), FlowVLA (Zhong et al., 2025), Video Prediction Policy (Hu et al., 2024), Genie Envisioner (Liao et al., 2025), ViDAR (Feng et al., 2025), RoboEnvision (Yang et al., 2025), and Latent Policy Steering (Wang et al., 2025) — is that video models need to be augmented for policy learning. These works share a common architecture pattern: the video model generates future frames (leveraging its spatiotemporal priors), but actions are produced by a separate module — an action diffuser, an inverse dynamics model mapping frames back to actions, or a learned action decoder — that is trained in a second stage after the video model is fine-tuned. The implicit assumption is that actions are fundamentally different from pixels, requiring different architectural handling.

Cosmos Policy challenges this assumption directly. By encoding a 14-DoF action chunk as a latent frame — duplicating the vector across the spatial dimensions of a latent volume and interleaving it with real image latents — the paper argues that the video model's core competency (generating coherent sequences conditioned on prior frames) is sufficient for action generation without any specialized machinery. The diffusion transformer's self-attention mechanism, which was designed to capture spatiotemporal dependencies between image patches, can apparently model the relationship between robot commands and visual outcomes when actions are simply another "modality" in the sequence, placed in causal order $(s, a, s', V(s'))$ .

This insight is diagnostic rather than prescriptive: it tells us something about what video models have already learned. The fact that full fine-tuning of the pretrained checkpoint on robot data — with no architectural changes — outperforms methods that carefully design separate action modules (Table 1: 98.5% vs. 94.2% for π0; Table 2: 67.1% vs. 66.4% for FLARE) suggests that the representation space learned during video pretraining already supports action-relevant computation, and that adding architectural components may introduce unnecessary capacity that must be learned from scarce robot data, actually hindering performance.

Distinction from unified video-action models like UVA and UWM. Those methods also avoid multi-stage pipelines by jointly predicting frames and actions — but they achieve this through custom architectures designed from scratch for the video-action coupling (novel tokenization schemes, specialized attention patterns). Because these architectures differ from any pretrained video model, they cannot load pretrained weights and must learn everything from the available robot demonstration data. Cosmos Policy's key move is to show that a generic video model architecture — one designed purely for video generation, with no affordances for action prediction — can be adapted to produce actions through a data representation trick alone, preserving all pretrained weights in the process. The ablation in Table 4 quantifies the value of this: training Cosmos Policy from scratch (same architecture, no pretrained weights) drops LIBERO average success rate from 98.5% to 94.6%, a 3.9-point decrease. This gap — the "pretraining dividend" — measures what the video model contributes beyond the architecture.

Why this is fundamental, not incremental. The field has been converging on a norm where domain-specific pretrained models (VLMs for semantic reasoning, video models for dynamics understanding) are adapted to robotics through carefully engineered interfaces — action heads, multimodal fusion layers, temporal aggregation modules. Cosmos Policy demonstrates that, at least for video models, the interface can be zero-width: the model's native I/O format is sufficiently expressive to absorb all required robotics modalities without modification. If this finding generalizes to other video models (Wan, CogVideoX) and other robotics domains (navigation, dexterous manipulation), it substantially simplifies the engineering roadmap for video-model-based policies and shifts research attention from architecture design to data representation and training protocol design.

Innovation 2: Video Model Priors as a Viable Alternative to Large-Scale Robot Action Pretraining

Cosmos Policy achieves state-of-the-art results on LIBERO and ALOHA despite having no pretraining on robot action data whatsoever, while its primary competitors — the fine-tuned VLAs π0.5, OpenVLA-OFT+, CogVLA, and UniVLA — have been pretrained on massive corpora of robotic interaction data (Table 1). This is a striking empirical finding that challenges the emerging consensus that large-scale robotic action pretraining is a prerequisite for high-performance manipulation policies.

The implicit bet made by VLA research. The VLA paradigm (RT-2, OpenVLA, π0, π0.5) builds on a two-stage premise: (1) pretrain a vision-language model on Internet-scale static image-text pairs to learn semantic concepts, object identities, and instruction following; then (2) fine-tune on large datasets of robot interaction data (thousands to millions of episodes across diverse embodiments) to learn the mapping from visual observations and language instructions to robot actions. The second stage — the "action" part of VLA — is considered essential because VLMs trained on static images have no concept of temporal dynamics, contact physics, or the causal relationship between motor commands and visual outcomes. The field has invested heavily in scaling this action pretraining: π0.5 is trained on a mixture of OXE (Open X-Embodiment) data spanning hundreds of tasks and dozens of robot platforms, and its strong performance is typically attributed to this breadth of action experience.

Cosmos Policy demonstrates that comparable — and in several benchmarks, superior — performance is achievable by substituting video model pretraining on Internet videos for VLA pretraining on robot action data. The video model has never seen a robot action, a joint angle, or an end-effector pose during pretraining. It has only seen pixels evolving over time in the natural world. Yet when fine-tuned on as few as 50 demonstrations per task (RoboCasa) or 185 total demonstrations across four tasks (ALOHA), it matches or exceeds VLAs that have seen orders of magnitude more task-relevant data.

What this implies about the nature of the learned priors. This result suggests that the spatiotemporal knowledge acquired by predicting future frames in Internet videos — object permanence, the effects of forces on deformable objects, the visual signatures of successful vs. failed grasps, the temporal coherence of multi-step procedures — transfers to robot control more directly than the semantic knowledge acquired by VLMs from image-text pairs. A video model trained to predict "what happens next" in millions of kitchen, assembly, and manipulation videos has, in some sense, already learned a world model of physical interactions that approximates what a robot needs to know to act. The 50–185 demonstrations then serve not to teach physics from scratch but to align this general physical knowledge with the specific embodiment's action space and camera viewpoint.

This is not a claim that video models universally outperform VLAs — the paper explicitly acknowledges that VLAs likely retain advantages for open-vocabulary instruction following and semantic reasoning about novel object categories (Section 2). Rather, it is a claim that for the low-level, temporally precise manipulation skills that dominate current benchmarks, spatiotemporal priors may be more valuable than semantic priors, and that this value can be extracted without architectural complexity.

Evidence and boundary conditions. The claim is anchored in Tables 1 and 2 (state-of-the-art LIBERO and RoboCasa results vs. fine-tuned VLAs) and Figure 4 (highest average ALOHA score of 93.6% vs. 88.6% for π0.5). However, the paper also provides a boundary condition: on OOD generalization scenarios in ALOHA (Table 3), π0.5 achieves a slightly higher OOD average score (92.5% vs. 89.3%), suggesting that VLAs' broader semantic pretraining may provide an edge specifically when test conditions deviate substantially from the training distribution. This nuance — video models excel at precise execution, VLAs excel at generalization — points toward a future where the two pretraining paradigms are combined rather than treated as competitors.

Innovation 3: Redefining "World Model" as a Conditioning Mask Rather Than a Separate Module

The paper makes a conceptual move that reframes how we think about world models in robotics: rather than training a separate dynamics model alongside the policy, Cosmos Policy demonstrates that a single diffusion model can serve simultaneously as policy, world model, and value function, with the distinction between these functions determined entirely by which latent frames are provided as conditioning and which are treated as targets to denoise. This collapses three traditionally separate training pipelines — imitation learning for the policy, dynamics prediction for the world model, and temporal-difference or Monte Carlo learning for the value function — into a single unified training objective with batch-split conditioning.

The traditional modular approach. In the lineage from Dyna (Sutton, 1991) through MBPO (Janner et al., 2019) to TD-MPC (Hansen et al., 2022) and Dreamer (Hafner et al., 2019; 2020; 2023), world models are architecturally distinct from policies. They may share a representation backbone (as in Dreamer, where a shared recurrent state space model serves both), but they are trained with separate objectives (reconstruction loss for the world model, reward prediction loss for the value function, actor-critic losses for the policy) and often require careful balancing of loss weights, gradient clipping, and target network updates. More recent robotics-specific approaches — FLARE (Zheng et al., 2025), SAILOR (Jain et al., 2025), Latent Policy Steering (Wang et al., 2025) — maintain this modularity, training separate models for dynamics prediction, reward/value estimation, and action generation, then orchestrating them at test time.

Cosmos Policy's insight is that in a diffusion model with a causally ordered latent sequence $(s, a, s', V(s'))$ , the conditioning mask is the only thing that distinguishes the policy from the world model from the value function. If you condition on $s$ and generate $(a, s', V(s'))$ , you're training a policy. If you condition on $(s, a)$ and generate $(s', V(s'))$ , you're training a world model. If you condition on $(s, a, s')$ and generate $V(s')$ , you're training a value function. The architecture, the weights, the denoising objective, and the latent space are identical across all three modes. During training, a batch is split 50/25/25 across these three conditioning masks (Section 4.2), and the model learns to handle all of them simultaneously.

Why this matters beyond engineering convenience. The significance is not merely that one model replaces three — it's that the three functions share a representation space that is jointly optimized. When the policy learns to predict $p(a \mid s)$ , it develops representations that are useful for action selection. When those same weights are also trained to predict $p(s' \mid s, a)$ , they must additionally learn the causal relationship between actions and outcomes — which in turn makes the action representations better, because they're now shaped by downstream consequences. The auxiliary supervision that proved critical in ablations (Table 4: removing it drops LIBERO from 98.5% to 97.0%; Table 5: removing future state prediction drops RoboCasa from 67.1% to 44.4%) is evidence that the representations are genuinely shared and mutually reinforcing, not just coexisting in the same parameter space.

This is similar in spirit to multitask learning or auxiliary task approaches, but with a diffusion-specific twist: the auxiliaries are not separate loss terms with weighting coefficients — they are the same loss (denoising score matching) applied to different subsets of the latent sequence. There are no hyperparameters to tune for loss balancing, no gradient conflicts to manage, and no risk of one objective dominating the others. The only design choice is the batch split ratio (50/25/25), which the paper does not ablate extensively.

The planning model as a refinement, not a replacement. The paper's approach to planning — fine-tuning a separate copy of the checkpoint on policy rollout data with 90% world model / value function weighting and 10% policy weighting — is conceptually elegant: it preserves the strong policy learned from demonstrations while using on-policy data (including failures) to improve the dynamics and value estimates. This decoupling (policy model proposes, planning model scores) is similar in spirit to actor-critic separation but achieved through separate fine-tuning rather than architectural separation. The result (Figure 7: 12.5-point improvement on challenging ALOHA tasks) demonstrates that world model accuracy, not policy quality, is the bottleneck for planning — and that modest amounts of on-policy data (648 rollouts) can substantially improve it.

Why this is fundamental, not incremental. The conditioning-mask-as-function-identity concept changes how researchers should think about building unified robot learning systems. Rather than asking "what architecture should I use for the policy, the world model, and the value function?", the question becomes "what is the causal ordering of information in my task, and how do I express that as a latent sequence with different conditioning masks?" This is a shift from architecture-centric to data-representation-centric design, and it is enabled specifically by the diffusion formulation's natural handling of conditional generation with masked inputs.

Innovation 4: Verifier Over-Optimization Is Not the Primary Bottleneck for Robot Planning (Yet)

The paper's planning results (Figure 7) demonstrate a 12.5-point average improvement from model-based best-of-N search over direct policy execution on the two most challenging ALOHA tasks, achieving this gain with only 648 rollout trajectories and 8-way search ( $N = 8$ ). What is intellectually significant here is not the absolute magnitude of the gain — 12.5 points is substantial but not transformative — but rather what this result implies about the scaling prospects for model-based planning in robotics. Specifically, it suggests that in the low-data regime typical of real-world robot learning, the bottleneck for planning is not over-optimization of the value function (a concern that dominates discussions of test-time search in language models) but rather the quality of the world model and value function predictions themselves.

The contrast with LLM test-time compute scaling. In language model reasoning, a well-documented phenomenon is that aggressive search against a reward model or verifier eventually leads to over-optimization: the search finds outputs that score highly under the verifier but are actually incorrect, causing performance to degrade at high search budgets. The Cosmos paper on test-time compute scaling (the reference example provided) demonstrates that beam search degrades easy-problem performance at high budgets due to this exact phenomenon. An implicit concern when applying model-based planning to robotics is that a similar over-optimization dynamic might occur: as the number of candidate actions $N$ increases, the planning model's value predictions might become unreliable, selecting actions that look good under the learned value function but fail in the real world.

Cosmos Policy's planning results (with $N = 8$ ) do not exhibit over-optimization — performance improves monotonically over the base policy, and the improvement comes from genuine world model accuracy gains (Figure 6 shows that the fine-tuned world model predicts failures like losing grasp of the ziploc bag slider that the base model misses). The paper's "majority mean" aggregation method (Section 4.3) — which classifies predictions into success/failure bins via a threshold before averaging — is explicitly designed to provide robustness against outlier predictions, but the fact that such a simple method works suggests that the planning model is not yet at the point where search optimizes against its errors in a harmful way. The limiting factor is that the world model and value function need more data to become accurate, not that they are accurate enough to be exploited.

Why the $V(s')$ formulation outperforms $Q(s, a)$ . The comparison between model-based planning (predict $s'$ , then $V(s')$ ) and model-free planning (directly predict $Q(s, a)$ ) in Figure 7 is informative. Model-based planning achieves higher performance, and the paper attributes this to the world model's ability to "leverage learned environment dynamics for more effective and sample-efficient planning." In the limited-data regime (648 rollouts), learning a direct $Q$ -function from $(s, a)$ pairs — which must implicitly capture all the environment dynamics that determine whether an action leads to success — is harder than learning to predict the immediate next state $s'$ and then evaluating it with $V(s')$ , which separates the dynamics modeling from the value estimation. The world model can learn from every transition (predicting $s'$ ), while the $Q$ -function only gets a single scalar label per transition. This is a classic argument for model-based over model-free methods, but it's validated here in a modern setting with a unified diffusion architecture and real-world manipulation tasks.

The implication for scaling. The paper's finding that 648 rollouts produce a 12.5-point gain — and that this gain depends critically on per-task data collection (143 additional rollouts for the ziploc bag task) — suggests that planning improvements scale with rollout data quantity and quality, not with search breadth. This shifts the research priority from developing more sophisticated search algorithms (the focus in LLM test-time compute optimization) to developing better methods for collecting and leveraging on-policy rollout data to improve world model accuracy. The paper's acknowledgment that "effective planning requires substantial rollout data" and that "learning from fewer rollouts would increase the accessibility of our approach" (Section 6) frames this as both a finding and an open problem.

Why this is a fundamental observation, not just an empirical result. The over-optimization vs. under-modeling distinction — whether search fails because the verifier is good enough to be exploited or because it's not good enough to be useful — is a diagnostic concept that generalizes beyond this paper. For any system that uses learned models to guide search (LLM reasoning, robotics planning, drug discovery, architecture search), understanding which regime you're in determines the correct intervention: better search algorithms vs. better models. Cosmos Policy provides evidence that in the current real-world robot learning regime, we are firmly in the under-modeling regime, and the returns to better world models (through more data, better architectures, or more effective fine-tuning) are likely larger than the returns to more sophisticated planning algorithms.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three distinct task suites: (1) LIBERO (Liu et al., 2024) — a simulation benchmark featuring a single Franka Emika Panda arm across four suites (LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long/10), each with 10 tasks and 50 demonstrations per task (500 total demos per suite). Evaluation is conducted over 500 trials per suite (10 tasks × 50 episodes) across 3 random seeds, for 6000 trials total. (2) RoboCasa (Nasiriany et al., 2024) — a simulation benchmark with 24 static kitchen manipulation tasks and a single Franka arm. Following the protocol of prior works, each task is evaluated over 50 trials across five evaluation scenes with different floor plans and styles (10 trials per scene, including two unseen styles), and the average success rate is computed over 3 random seeds (3600 trials total). Cosmos Policy is trained on only 50 human-teleoperated demonstrations per task (not the 300+ used by prior methods). (3) Real-world ALOHA (Zhao et al., 2023) — four bimanual manipulation tasks ("put X on plate": 80 demos, "fold shirt": 15 demos, "put candies in bowl": 45 demos, "put candy in ziploc bag": 45 demos) using a dual ViperX 300 S robot arm setup with three cameras (one top-down, two wrist-mounted). Evaluations span 101 trials total across both in-distribution and out-of-distribution initial conditions, using the same fixed set of initial states for all compared methods.
Base model(s). All Cosmos Policy variants are fine-tuned from Cosmos-Predict2-2B-Video2World (NVIDIA et al., 2025), a 2-billion-parameter latent video diffusion transformer using the EDM formulation and the Wan2.1 spatiotemporal VAE tokenizer. This model was chosen because it represents a state-of-the-art video foundation model with strong spatiotemporal priors from training on millions of Internet videos, providing a testbed for the paper's central hypothesis that video model pretraining transfers effectively to robot control. For ablation experiments, a randomly initialized version with the same architecture is trained from scratch to quantify the pretraining dividend. For the FLOPs-matched and planning comparisons, a second fine-tuned copy of the same base checkpoint serves as the planning model.
Metrics. Three primary metrics are used across the evaluation suites. LIBERO and RoboCasa report average success rate (SR), defined as the fraction of evaluation trials in which the task is completed successfully (as determined by the benchmark's built-in success detectors), averaged across all tasks and random seeds. Real-world ALOHA evaluations use score (0–100), a continuous metric representing the percentage of task completion — for example, in "put candies in bowl," 20 points are awarded for each of 5 candies successfully grasped and placed into the bowl; in "put candy in ziploc bag," 20 points are awarded for each of 5 sequential stages (grasp opener with right arm, grasp bag corner with left arm, open bag at least halfway, grasp candy, place candy inside bag). Score is used instead of binary success because it captures partial progress more informatively. Additional training diagnostics include L1 loss on actions, future proprioception, future image latents, and value predictions (reported in Appendices A.2.2–A.2.4). L1 loss magnitudes are not directly comparable across modalities due to different latent volume sizes and normalization ranges.
Baselines. In LIBERO (Table 1), the paper compares against: Diffusion Policy (Chi et al., 2023) — trained from scratch; Dita (Hou et al., 2025) — diffusion transformer policy trained from scratch; π0 and π0.5 (Black et al., 2024; Intelligence et al., 2025) — fine-tuned VLA models; UVA (Li et al., 2025a) — unified video-action model trained from scratch; UniVLA (Bu et al., 2025) — fine-tuned VLA; Video Policy (Liang et al., 2025) — multi-stage video-to-policy method; OpenVLA-OFT (Kim et al., 2025) — fine-tuned VLA; CogVLA (Li et al., 2025b) — fine-tuned VLA. In RoboCasa (Table 2), comparisons include: GR00T-N1 and GR00T-N1.5 (Bjorck et al., 2025); UVA; DP-VLA (Han et al., 2024); GR00T-N1 + DreamGen (Jang et al., 2025); GR00T-N1 + DUST (Won et al., 2025); UWM (Zhu et al., 2025); π0; Video Policy; FLARE (Zheng et al., 2025); GR00T-N1.5 + HAMLET (Koo et al., 2025). In real-world ALOHA (Figure 4, Table 3), the paper compares against: Diffusion Policy, OpenVLA-OFT+, π0, and π0.5 — a competitive subset chosen for demonstrated strong performance in bimanual manipulation. For fair comparison, all methods on ALOHA are trained with the same computational budget (48 hours on 8 H100 GPUs for the large models; 48 hours on 1 H100 GPU for the smaller Diffusion Policy), though differences in training throughput lead to substantially different numbers of gradient steps (e.g., π0.5 completes 400K steps at batch size 256, while OpenVLA-OFT+ completes 32K steps at batch size 96).
Generation budget / compute accounting. The paper does not report a standardized FLOPs budget across methods — this is not a compute-matched comparison. Instead, fairness is ensured through: (1) identical training data (same demonstration datasets for all methods on each benchmark); (2) identical evaluation conditions (same initial states for ALOHA real-world trials); (3) identical wall-clock training time for ALOHA experiments (48 hours on 8 H100 GPUs). For the planning experiments, compute is reported as inference wall-clock time: 0.61 seconds per action chunk with 5 denoising steps on 1 H100 GPU (LIBERO and RoboCasa), 0.95 seconds with 10 denoising steps (ALOHA), and 4.9 seconds for best-of-N planning with $N = 8$ on 8 H100 GPUs. The planning cost includes: action generation (10 denoising steps per candidate), future state prediction (3 ensemble members × 5 denoising steps each), and value prediction (5 ensemble members × 5 denoising steps each × 3 future states), all parallelized across GPUs. The number of denoising steps is the primary compute-relevant hyperparameter, with 5 used for LIBERO and RoboCasa (achieving 67.1% on RoboCasa; with 1 step achieving 66.4%, a 0.7-point drop for a ~4× speedup to 0.16 seconds) and 10 used for ALOHA. No compute cost is reported for difficulty estimation (unlike the Cosmos test-time compute paper), since the planning approach does not require per-problem difficulty binning — it uniformly applies best-of-N search.
Cross-validation / statistical protocol. Three random seeds are used for all simulation evaluations (LIBERO: 6000 trials total; RoboCasa: 3600 trials total), with average success rates reported. For real-world ALOHA evaluations, a single set of 101 trials across four tasks is used per method, with the same fixed initial states for every method to ensure comparability. No confidence intervals, standard errors, or statistical significance tests are reported for any results. The ALOHA evaluation's 101 trials are split across both in-distribution and out-of-distribution conditions (Table 3 provides the breakdown), but the per-condition sample sizes are small (e.g., 12 in-distribution and 8 OOD trials for "fold shirt"), and no domain randomization or systematic difficulty stratification is applied. For the planning experiments on the two hardest ALOHA tasks, the rollouts dataset contains 648 trajectories (505 from all policy evaluations plus 143 additional Cosmos Policy rollouts for the ziploc bag task), and fine-tuning is performed once on this fixed dataset — there is no cross-validation over different rollout subsets, meaning the reported planning improvements could be sensitive to the specific rollouts collected.

Main Quantitative Results

Direct Policy Results: Simulation Benchmarks

LIBERO (Table 1). Cosmos Policy achieves an average success rate of 98.5% across the four LIBERO suites, establishing a new state of the art. The per-suite breakdown is: Spatial 98.1%, Object 100.0%, Goal 98.2%, Long 97.6%. The closest competitor is CogVLA at 97.4% average, with OpenVLA-OFT at 97.1%, π0.5 at 96.9%, and UniVLA at 95.2%. The largest gap appears on LIBERO-Long, where Cosmos Policy's 97.6% exceeds CogVLA (95.4%) by 2.2 points and OpenVLA-OFT (94.5%) by 3.1 points. On LIBERO-Object, Cosmos Policy achieves perfect 100.0% success. The absolute improvement over the previous best (CogVLA) is 1.1 percentage points on average. Over Diffusion Policy (trained from scratch, no pretrained backbone), the improvement is 26.1 points (98.5% vs. 72.4%), quantifying the combined benefit of the video model architecture and pretrained weights.

RoboCasa (Table 2). Cosmos Policy achieves an average success rate of 67.1% across the 24 RoboCasa tasks, a new state of the art, while using only 50 training demonstrations per task — 6× fewer than most prior methods (which used 300–3000). The closest competitors are FLARE (66.4% with 300 demos), GR00T-N1.5 + HAMLET (66.4% with 300 demos), and Video Policy (66.0% with 300 demos). The absolute improvement over FLARE is 0.7 percentage points, but the demos-per-task advantage is substantial: Cosmos Policy achieves higher success with 50 demos than FLARE achieves with 300. UVA, which also uses 50 demos, achieves only 50.0%, indicating a 17.1-point gap between Cosmos Policy and another unified video-action approach in the same low-data regime. Methods that use orders of magnitude more data — DP-VLA (3000 demos, 57.3%), GR00T-N1 + DreamGen (300 + 10,000 synthetic demos, 57.6%) — are substantially outperformed by Cosmos Policy with only 50 demos. The paper notes that Cosmos Policy was not trained on the additional 1000 MimicGen-generated demonstrations that most prior RoboCasa methods use, deliberately assessing data efficiency.

Interpretation of the simulation results. The LIBERO results (Table 1) demonstrate that Cosmos Policy's video model prior plus latent frame injection approach can match or exceed fine-tuned VLAs that have benefited from large-scale robotic action pretraining (π0.5, OpenVLA-OFT, CogVLA). The RoboCasa results (Table 2) make a stronger claim: in a low-data regime (50 demos), Cosmos Policy substantially outperforms methods that require 6–60× more data, suggesting that the spatiotemporal priors from video pretraining provide a larger effective data multiplier than the semantic priors from VLM pretraining, at least for these manipulation tasks. However, the RoboCasa comparison is not purely controlled — prior methods differ in architecture, training procedure, and pretraining data, not just in training demonstration count. The 50-demo Cosmos Policy vs. 300-demo baselines comparison confounds data quantity with model quality and training methodology.

Direct Policy Results: Real-World ALOHA Tasks

Overall performance (Figure 4, Table 3). Cosmos Policy achieves a 93.6% average score across the four ALOHA tasks, the highest among all compared methods. The per-task scores are: "put X on plate" 100.0%, "fold shirt" 99.5%, "put candies in bowl" 89.6%, "put candy in ziploc bag" 85.4%. The closest competitor is π0.5 at 88.6% average (5.0 points lower), followed by π0 at 77.9%, OpenVLA-OFT+ at 62.0%, and Diffusion Policy at 33.6%. The gap is largest on "put candy in ziploc bag" (Cosmos Policy 85.4% vs. π0.5 61.5%, a 23.9-point difference) and "put candies in bowl" (89.6% vs. 95.2% for π0.5 — notably, π0.5 achieves a higher score on this task, 95.2 vs. 89.6, a 5.6-point advantage). On the two easier tasks ("put X on plate" and "fold shirt"), both Cosmos Policy and π0.5 perform near ceiling (98–100%), suggesting these tasks are saturated and do not discriminate among strong methods.

In-distribution vs. OOD breakdown (Table 3). The paper provides a detailed split of in-distribution and out-of-distribution scores. On in-distribution trials, Cosmos Policy achieves 96.3% average vs. 87.8% for π0.5 (8.5-point gap). On OOD trials, Cosmos Policy achieves 89.3% vs. 92.5% for π0.5 — a reversal where π0.5 leads by 3.2 points. The OOD advantage for π0.5 is driven primarily by "put candies in bowl" (π0.5 90.0% vs. Cosmos Policy 74.0%, a 16-point gap) and "put X on plate" (both achieve 100%). Cosmos Policy retains an OOD advantage on "put candy in ziploc bag" (83.3% vs. 80.0%). This pattern suggests that VLAs may have a generalization advantage specifically when test conditions deviate substantially from the training distribution in ways that benefit from semantic understanding (e.g., recognizing an unseen smaller bowl or handling uneven candy distributions), while Cosmos Policy's advantage is strongest when precise manipulation execution matters more than generalization to novel semantic conditions.

Qualitative failure mode analysis (Figure 5). The paper provides qualitative observations of failure modes for π0.5 and OpenVLA-OFT+ on the two hardest tasks. π0.5 "struggles to execute a high-precision grasp and loses grip of the ziploc bag," and often "misses the initial grasp of the slider with the right arm or does not grasp the left side of the bag securely enough with the left arm." OpenVLA-OFT+ "reaches between two candies rather than towards one," which the paper hypothesizes is because "its L1 regression of actions leads to inaccurate modeling of the action distribution in tasks with high multimodality" — essentially, regressing to the mean of a multimodal distribution produces actions that are between modes rather than committing to one. These qualitative observations are consistent with the quantitative results (Cosmos Policy substantially outperforms both methods on these tasks) but are not systematically verified across many trials or through quantitative multimodality metrics.

Model-Based Planning Results

Planning on challenging ALOHA tasks (Figure 7). On the two most challenging ALOHA tasks ("put candies in bowl" and "put candy in ziploc bag"), evaluated on a more challenging set of initial conditions (difficult in-distribution and OOD states), the base Cosmos Policy without planning achieves a certain baseline score (exact numbers not reported in the figure; the figure uses a bar chart format without per-bar numerical labels in the text). Model-based planning with the $V(s')$ formulation achieves a 12.5-point higher average score than the base policy, while model-free planning with the $Q(s, a)$ formulation achieves a smaller improvement. The paper attributes the advantage of $V(s')$ over $Q(s, a)$ to the world model's ability to "leverage learned environment dynamics for more effective and sample-efficient planning" — with only 648 rollouts, learning an accurate Q-function directly from $(s, a)$ pairs is harder than separating dynamics prediction ( $s'$ from $s, a$ ) from value prediction ( $V$ from $s'$ ), because the world model learns from every transition's rich future state signal while the Q-function receives only a scalar return label per transition.

World model prediction quality (Figure 6). The paper provides qualitative evidence that the fine-tuned planning model produces more accurate future state predictions than the base model. In the shown example, the base Cosmos Policy's world model "may fail to predict errors such as losing grasp of the ziploc bag slider," while the fine-tuned model "more accurately predicts the resulting state, enabling more effective planning and eventual episode success." This is presented as visual evidence (predicted future images side-by-side with actual outcomes), but no quantitative metric of world model prediction accuracy (e.g., MSE on future image latents, success rate of predicted vs. actual task outcomes) is reported. The 12.5-point planning improvement is the quantitative evidence that the world model and value function have become more reliable after fine-tuning on rollout data.

Rollout data composition. The 648-rollout dataset used for planning model fine-tuning consists of 505 rollouts from all evaluated policies (not just Cosmos Policy) plus 143 additional Cosmos Policy rollouts for the ziploc bag task. The paper notes that the extra ziploc bag rollouts were necessary because "training an accurate world model for [that task] is particularly challenging due to low camera observability from the robot's self-occlusion and highly stochastic environment dynamics where even millimeter differences in control can dictate success or failure." This implies that the planning improvement is task-dependent and may require task-specific data collection effort. No experiment varies the amount of rollout data to characterize the scaling behavior of planning performance.

Inference Speed and Planning Overhead

Direct policy inference latency (Appendix A.4.2). Cosmos Policy generates action, future state, and value predictions in parallel in 0.61 seconds on 1 H100 GPU with 5 denoising steps (LIBERO and RoboCasa). This decreases to 0.16 seconds with 1 denoising step, achieving a 66.4% RoboCasa success rate (only 0.7 points below the 67.1% achieved with 5 steps) — a nearly 4× speedup for a minimal accuracy cost. For ALOHA, 10 denoising steps take 0.95 seconds. At the ALOHA control frequency of 25 Hz, an action chunk spans 50 timesteps (2 seconds), so the 0.95-second inference latency is well within the execution window (the robot pauses for inference, then executes for 2 seconds).

Planning inference latency (Appendix A.4.2). Best-of-N planning with $N = 8$ on 8 H100 GPUs takes 4.9 seconds total: the model generates 8 candidate action chunks (10 denoising steps each), 3 future state predictions per candidate (5 denoising steps each, total 24 predictions), and 5 value predictions per future state (5 denoising steps each, total 120 predictions), with all computations parallelized across the 8 GPUs. This is approximately 5× slower than direct policy inference but still within a reasonable range for non-dynamic tasks. The paper acknowledges that the slower inference speed "may limit applicability to dynamic tasks" and identifies speed optimization as future work.

Ablation Studies and Robustness Checks

Auxiliary supervision during joint training (Table 4, LIBERO). Removing the auxiliary targets from policy and world model training — so the policy learns only $p(a \mid s)$ instead of $p(a, s', V(s') \mid s)$ , and the world model learns only $p(s' \mid s, a)$ instead of $p(s', V(s') \mid s, a)$ — reduces average LIBERO success rate from 98.5% to 97.0%, a 1.5-point absolute drop. The per-suite decreases are: Spatial 98.1% → 97.6% (−0.5), Object 100.0% → 99.8% (−0.2), Goal 98.2% → 96.7% (−1.5), Long 97.6% → 94.0% (−3.6). The largest degradation occurs on LIBERO-Long, the most challenging suite, suggesting that auxiliary future state and value prediction provides greater benefit when tasks are longer-horizon and more difficult. This ablation does not remove the world model and value function training samples from the batch — it only removes the auxiliary targets within the policy and world model conditioning schemes. The value function continues to receive its 25% of training samples as before.

Pretrained weights vs. training from scratch (Table 4, LIBERO). Training the Cosmos Policy architecture from randomly initialized weights (no pretrained video model) reduces average LIBERO success rate from 98.5% to 94.6%, a 3.9-point drop. The per-suite decreases are: Spatial 98.1% → 94.7% (−3.4), Object 100.0% → 98.9% (−1.1), Goal 98.2% → 96.3% (−1.9), Long 97.6% → 88.6% (−9.0). Again, LIBERO-Long shows the largest degradation (9.0 points), reinforcing that pretrained spatiotemporal priors are most valuable for long-horizon tasks. The Object suite (100.0% → 98.9%) shows the smallest drop, suggesting that object identification and simple pick-and-place skills may be learnable from scratch even without video pretraining. The from-scratch variant also exhibits "jerky motions that may damage the robot over prolonged deployment" in real-world ALOHA tests (achieving only 80.8 score on "fold shirt" vs. 99.5 for the full model — an 18.7-point gap), leading the authors to halt further real-world evaluation of this variant for safety reasons.

Detailed ablation chain in RoboCasa (Table 5). The paper reports a progressive ablation on RoboCasa, removing components one at a time from the full training scheme:

(1) Without value function training samples: The 25% of training batches dedicated to value function training ( $p(V(s') \mid s, a, s')$ ) is removed; those samples are reallocated to policy training (now 50% policy, 50% world model). Success rate drops from 67.1% to 66.6%, a 0.5-point decrease. This small drop suggests that explicit value function training provides minimal benefit when the policy and world model already receive auxiliary value supervision.
(2) Without world model and value function training samples: Both the world model (25%) and value function (25%) training samples are removed; 100% of each batch is used for policy training only (the policy still predicts $p(a, s', V(s') \mid s)$ with auxiliary targets). Success rate drops to 64.0%, a 3.1-point decrease from full. This demonstrates that explicit world model and value function training — even when the policy receives auxiliary supervision on the same targets — contributes non-trivially to performance.
(3) Without world model and value function training samples AND without auxiliary value supervision: In addition to (2), the value target is removed from the policy's auxiliary objectives; the policy now learns $p(a, s' \mid s)$ (actions and future state, but not value). Success rate drops to 62.5%, a further 1.5-point decrease. The value auxiliary adds a small but measurable benefit beyond what future state prediction alone provides.
(4) Without world model and value function training samples AND without auxiliary future state and value supervision (barebones policy): The policy now learns only $p(a \mid s)$ — no auxiliary targets, no separate world model or value function training. Success rate collapses to 44.4%, a 22.7-point decrease from the full model. This is the largest single ablation effect, indicating that future state prediction is the most critical component of Cosmos Policy's training scheme. The model that only predicts actions — despite having the same architecture, pretrained initialization, and training data — performs dramatically worse than the model that jointly predicts actions and future observations.

Number of denoising steps (Table 5, RoboCasa). Reducing inference-time denoising steps from 5 to 1 yields a RoboCasa success rate of 66.4% (vs. 67.1% with 5 steps), a 0.7-point drop with a ~4× inference speedup (0.16 seconds vs. 0.61 seconds per action chunk on 1 H100 GPU). This suggests that Cosmos Policy's action predictions are robust to the number of denoising steps in the low-step regime, which is practically important for latency-sensitive applications.

Model-based vs. model-free planning (Figure 7). The comparison between planning with $V(s')$ (requiring world model rollouts) and planning with $Q(s, a)$ (direct value prediction without future state prediction) shows that the model-based variant achieves "higher performance" (exact numbers not reported in text; visible in bar chart). The paper attributes this to the sample efficiency advantage of separating dynamics learning from value learning, particularly when rollout data is limited (648 trajectories). The Q-function must implicitly learn environment dynamics from scalar reward labels, while the world model learns from the rich signal of predicting full future observations. No intermediate variants (e.g., ensembling fewer world model samples, varying the number of value samples) are ablated.

Ensemble size for planning (Appendix A.3.1). The paper uses an ensemble of 3 future state predictions and 5 value predictions per future state (15 total value estimates per action proposal) for the planning experiments. No ablation varying these ensemble sizes is reported, so the sensitivity of planning performance to the ensemble configuration is unknown. The "majority mean" aggregation method is described but not ablated against simpler alternatives (e.g., naive mean, median, or best-of-N weighted selection without threshold-based binning).

Real-world from-scratch comparison (Section 5.2). The from-scratch Cosmos Policy variant achieves a score of 80.8 on the "fold shirt" task (vs. 99.5 for the full model, an 18.7-point drop), and the paper notes that it "exhibits jerky motions that may damage the robot over prolonged deployment, so we halt further evaluations with it." This is both a quantitative result (18.7-point gap) and a qualitative safety observation (jerky motions). The from-scratch comparison is limited to this single task, so the generalizability of the 18.7-point gap across other ALOHA tasks is unknown.

Noise distribution modification (Appendix A.2.1). The change from the base model's log-normal noise distribution to the hybrid log-normal-uniform distribution is described as empirically improving action prediction accuracy, but no ablation comparing the two distributions is reported with quantitative results (e.g., success rate with original noise distribution vs. hybrid). The adjusted $\sigma_{\text{min}} = 4$ (vs. the original 0.002) is similarly motivated by observed L1 loss improvements on training and validation samples, but no success-rate-level ablation of this choice is provided.

Critical Assessment

Claim 1: Cosmos Policy achieves state-of-the-art performance on LIBERO and RoboCasa while requiring no architectural modifications and fewer demonstrations. The evidence for this claim is strong but has important caveats. The LIBERO result (98.5%, Table 1) does establish a new state of the art, but the absolute improvement over the previous best (CogVLA at 97.4%) is only 1.1 percentage points. On a 6000-trial evaluation, this difference may not be statistically significant — no confidence intervals are reported. The LIBERO benchmark has been heavily saturated by recent methods (the top six methods all exceed 95%), so the practical significance of a 1.1-point improvement in this near-ceiling regime is debatable. The more compelling LIBERO evidence is the comparison against methods that do not use large-scale robot action pretraining — but these methods (Diffusion Policy, Dita) are trained from scratch and are not directly comparable to a model with 2B pretrained parameters.

The RoboCasa result (67.1%, Table 2) is more impressive in absolute terms: it beats FLARE and GR00T-N1.5 + HAMLET (both 66.4%) by 0.7 points while using 6× fewer demonstrations. However, the "fewer demonstrations" claim requires scrutiny. Cosmos Policy uses 50 human-teleoperated demonstrations per task. Prior methods use 300+ demonstrations, but these include both human demos (50) and MimicGen-generated demos (250+). MimicGen demos are synthetically generated by applying the human demonstrations to varied initial conditions in simulation — they are substantially noisier and lower-quality than human demos. It is not obvious that 300 mixed-quality demos should outperform 50 high-quality demos, so the "6× data efficiency" framing may overstate the advantage. A fairer comparison would be to train prior methods on the same 50 human demos as Cosmos Policy, which the paper does not do (with the partial exception of UVA, which also uses 50 demos and achieves 50.0%).

The "no architectural modifications" claim is technically accurate but deserves qualification. While no new layers or heads are added, the paper does modify the training protocol substantially: a new noise distribution (hybrid log-normal-uniform), a new $\sigma_{\text{min}}$ bound (4 instead of 0.002), and a new batch-splitting scheme with three different conditioning masks. Whether these count as "modifications" is a matter of definition — they are changes to the training procedure, not the architecture, but they were necessary for the approach to work and were discovered through experimentation, suggesting that simply fine-tuning the base model on robot data without these adjustments would not have achieved the reported results. The paper does not report the performance of Cosmos Policy with the original noise distribution and sampling bounds, so the contribution of these procedural modifications to the final performance is unknown.

Claim 2: Video model pretraining provides a strong initialization for control policies without requiring large-scale robot action data, and this matches or exceeds fine-tuned VLAs that have benefited from such data. This claim is supported by the comparison between Cosmos Policy and π0.5 / OpenVLA-OFT+ (Tables 1, 3; Figure 4) but has several important limitations. First, the comparison is not controlled for model scale, architecture, or pretraining data quantity. Cosmos-Predict2-2B has 2 billion parameters and was trained on millions of Internet videos. π0.5 and OpenVLA have different architectures (flow matching vs. autoregressive), different scales (7B+ parameters for many VLAs), and different pretraining data (static images and text for VLAs vs. videos for Cosmos). The paper demonstrates that Cosmos Policy outperforms these VLAs on specific benchmarks, but this does not cleanly isolate "video model pretraining" as the causal factor — it could be that Cosmos-Predict2 is simply a better-architected model, or that its specific pretraining data mixture is more relevant to manipulation, or that its 2B scale is in a sweet spot relative to the available fine-tuning data.

Second, the claim that video model priors "substitute" for robot action pretraining is not directly tested. The paper does not compare Cosmos Policy against a version of itself that was pretrained on robot action data (which does not exist, since Cosmos-Predict2 was only pretrained on Internet videos). The claim is therefore based on cross-model comparisons that confound pretraining domain with model architecture, scale, and training methodology. A stronger test would be to compare Cosmos Policy to a VLA with identical architecture but different pretraining data (static images vs. videos), or to compare the same Cosmos-Predict2 architecture pretrained on Internet videos vs. pretrained on robot action videos — neither of which is feasible with current infrastructure.

Third, the OOD generalization results (Table 3) complicate the story: π0.5 achieves a higher OOD average score than Cosmos Policy (92.5% vs. 89.3%), driven primarily by a large advantage on the "put candies in bowl" task (90.0% vs. 74.0%). This suggests that while video model priors may excel at precise execution of skills similar to those demonstrated, VLAs may have a generalization advantage when test conditions require recognizing novel semantic concepts (unseen bowls, unusual object arrangements). The paper's qualitative hypothesis that VLAs benefit from "semantic understanding" while video models benefit from "spatiotemporal priors" is plausible but untested — there are no experiments that systematically vary the semantic vs. spatiotemporal demands of evaluation tasks.

Claim 3: Incorporating policy rollout data to refine world model and value function predictions enables effective model-based planning that yields substantial improvements on challenging tasks. The evidence (Figure 7: 12.5-point average improvement) supports this claim but with significant scope limitations. The planning experiments are conducted on only two of the four ALOHA tasks — the ones where the base policy has the most room for improvement. This is a sensible choice for demonstrating the method's potential, but it means the claim's generality is unknown: would planning help on the easier tasks ("put X on plate," "fold shirt") where the base policy is already near 100%? Would it help on LIBERO and RoboCasa, where the base policy achieves 98.5% and 67.1% respectively? No simulation planning results are reported.

The planning improvement is demonstrated with a specific rollout dataset (648 trajectories) collected in a specific way (mixture of on-policy and off-policy rollouts). The paper does not ablate the amount of rollout data, the on-policy vs. off-policy composition, or the task-specific vs. task-agnostic nature of the collected data. The 12.5-point figure is therefore a point estimate that may not generalize to different data collection strategies or different tasks.

The inference latency of planning (4.9 seconds for an 8-way search on 8 H100 GPUs) is a practical concern that the paper acknowledges. For tasks requiring faster reactive control (e.g., dynamic object catching, contact-rich assembly with force feedback), this latency would be prohibitive. The 12.5-point improvement comes with a 5× inference time cost and 8× GPU requirement relative to direct policy execution. The paper does not explore whether similar improvements could be achieved with fewer GPUs and a correspondingly slower search, or whether alternative planning methods (trajectory optimization, MPPI with fewer samples) could achieve better accuracy-latency tradeoffs.

Missing experiments that would strengthen the paper:

Scaling with demonstration count: The paper emphasizes Cosmos Policy's data efficiency (50 demos in RoboCasa vs. 300+ for prior work) but never varies the number of demonstrations to characterize the scaling behavior. How does Cosmos Policy perform with 10 demos? With 100? Does the advantage over from-scratch training grow or shrink as more data becomes available? This is a critical missing scaling analysis, especially since the paper's core claim is about leveraging pretrained priors to improve data efficiency.
Scaling with rollout data for planning: The planning experiments use exactly 648 rollouts, with no ablation of this number. A learning curve showing planning improvement as a function of rollout dataset size would characterize the data efficiency of the world model / value function refinement and guide practitioners on how much data to collect.
Full planning latency-accuracy tradeoff: The paper reports planning with $N = 8$ and ensemble sizes of 3 and 5, but does not ablate $N$ or the ensemble sizes. A sweep over $N \in \{1, 2, 4, 8, 16\}$ would characterize how planning performance scales with search breadth and at what point diminishing returns or over-optimization sets in.
Simulation planning results: Since simulation enables large-scale evaluation, running planning experiments in LIBERO or RoboCasa (where hundreds of trials per configuration are feasible) would provide more statistically reliable planning improvement estimates than the 25–26 real-world trials per task used in the ALOHA planning evaluation.
Ablation of noise distribution and $\sigma_{\text{min}}$ changes: The paper presents the hybrid noise distribution and adjusted $\sigma_{\text{min}}$ as empirically motivated but does not report success rates with the original distribution. Given that these changes are described as important for action prediction accuracy, quantifying their contribution to final task performance would help readers understand which aspects of the adaptation recipe are load-bearing.
Cross-embodiment generalization: Cosmos Policy is trained separately for each robot platform (Franka in simulation, ViperX in real-world). The paper's latent injection mechanism is described as flexible across embodiments, but no experiment tests whether a Cosmos Policy trained on one embodiment (e.g., Franka in LIBERO) transfers to another (e.g., Franka in RoboCasa, or simulated Franka to real Franka). Such experiments would directly test the claim that the video model priors capture embodiment-agnostic physical knowledge.
Comparison to the base Cosmos-Predict2 model without fine-tuning: The paper never reports what happens if you take the frozen pretrained Cosmos-Predict2 and use it zero-shot for action generation after latent frame injection (i.e., without any fine-tuning on robot data). This baseline would calibrate how much the video model priors contribute without any task-specific adaptation.

Strengths of the experimental design:

The multi-domain evaluation (two simulators, one real-world platform, both single-arm and bimanual tasks) provides stronger evidence of generality than single-benchmark studies.
The systematic ablation chain in RoboCasa (Table 5), progressively stripping components down to a barebones $p(a \mid s)$ policy, cleanly isolates the contribution of each training objective. The 22.7-point drop from the full model to the action-only model is a compelling demonstration that the joint training scheme is not merely incidental — it is load-bearing.
The OOD generalization evaluation (Table 3) adds nuance to the headline results by revealing that the advantage over VLAs is not uniform across distribution shifts.
The real-world from-scratch comparison (80.8 vs. 99.5 on "fold shirt") provides a vivid, if limited, demonstration of the pretraining dividend in a physical setting.
The fair computational budget for ALOHA training (48 hours on 8 H100 GPUs for all large models) controls for one source of confounding in cross-method comparison, though differences in training throughput complicate the interpretation.

Weaknesses of the experimental design:

Sample sizes for real-world evaluations are small: 101 total trials across four tasks, with as few as 8 OOD trials for some task-condition combinations. The 12.5-point planning improvement is based on an unknown number of trials (the paper does not specify how many planning trials were run, only that they use "challenging initial states"). Small sample sizes make the results noisy and potentially sensitive to the specific initial conditions chosen.
No statistical significance testing, standard errors, or confidence intervals are reported for any result, making it difficult to assess whether observed differences (e.g., 98.5% vs. 97.4% on LIBERO) are reliable.
The RoboCasa comparison confounds demonstration count with demonstration quality (human vs. MimicGen-generated), weakening the "data efficiency" claim.
The planning experiments are limited to two tasks on one robot platform, with no simulation results to enable larger-scale or more systematic evaluation.
Critical ablations are missing: noise distribution, $\sigma_{\text{min}}$ bound, ensemble sizes for planning, and scaling of planning performance with rollout data quantity.
The paper does not compare against the simplest possible baseline that uses the same pretrained backbone: fine-tune Cosmos-Predict2 to output actions only ( $p(a \mid s)$ ), without world model, value function, or auxiliary targets. The ablation chain in Table 5 provides this comparison in RoboCasa (44.4% for the barebones policy), but no such ablation is reported for LIBERO or ALOHA. The closest is the "without auxiliary losses" ablation in Table 4 (97.0%), but this variant retains world model and value function training — only the auxiliary targets within the policy loss are removed.

6. Limitations and Trade-offs

6.1 Planning Relies on Substantial Per-Task Rollout Data Despite Modest-Gain Demonstrations

The assumption or constraint. Cosmos Policy's model-based planning improvements depend on collecting hundreds of policy rollouts — including failures — specifically for the tasks where planning will be deployed. The paper acknowledges this explicitly: "effective planning requires substantial rollout data to achieve accurate predictions beyond the demonstration distribution. Learning from fewer rollouts would increase the accessibility of our approach" (Section 6). For the two challenging ALOHA tasks where planning is evaluated, the rollout dataset consists of 648 trajectories: 505 from all evaluated policies plus 143 additional Cosmos Policy rollouts specifically for the ziploc bag task.

The consequence. This creates an uncomfortable asymmetry in the method's value proposition. The headline result — 67.1% on RoboCasa using only 50 demonstrations — emphasizes extreme demonstration efficiency for the direct policy. But if a practitioner wants the additional 12.5-point planning gain, they must deploy the policy in the real world, collect hundreds of trials (many of which will fail, potentially damaging the workspace or requiring manual resets), and then fine-tune. The total data requirement for planning-enabled Cosmos Policy (50 demos + 300+ rollouts per task) may exceed the data budget of methods that achieve comparable performance through pure imitation learning with larger demonstration sets. The paper provides no learning curve showing how planning performance improves with rollout dataset size, so practitioners cannot estimate how much rollout data is "enough" for their task. The ziploc bag task required 143 additional rollouts specifically because "low camera observability from the robot's self-occlusion and highly stochastic environment dynamics" made world model learning difficult (Section 5.3), suggesting that the rollout data requirement is itself task-dependent in unpredictable ways.

What evidence exists in the paper. The 12.5-point planning improvement (Figure 7) is demonstrated using exactly 648 rollouts, with no ablation varying this quantity. The paper reports that the ziploc bag task needed extra rollouts (Section 5.3) but does not quantify how much the planning gain would shrink with fewer rollouts. The planning experiments are limited to two of four ALOHA tasks (the two with most room for improvement), and no simulation planning results are reported, so the data-efficiency characteristics of the planning pipeline are essentially uncharacterized.

Mitigation status. The paper identifies this as a limitation and suggests "learning from fewer rollouts" as future work (Section 6). No technical mitigation (e.g., data augmentation, model-based rollout generation, selective data collection) is proposed or tested.

6.2 4.9-Second Planning Latency Restricts Applicability to Quasi-Static Tasks

The assumption or constraint. Best-of-N planning with $N = 8$ on 8 H100 GPUs requires 4.9 seconds to produce one action chunk (Appendix A.4.2). This includes generating 8 candidate action chunks (10 denoising steps each), 24 future state predictions (3 ensemble members × 8 candidates, 5 denoising steps each), and 120 value predictions (5 ensemble members × 24 future states, 5 denoising steps each). The paper uses no receding-horizon control — the full action chunk is deployed before requerying — to avoid multiplying this cost by replanning frequency. The 8-GPU requirement means planning is infeasible on single-GPU edge deployment setups.

The consequence. The method is limited to quasi-static manipulation tasks where the environment does not change significantly during the 4.9-second planning window. Tasks requiring reactive control — dynamic object catching, contact-rich assembly with force feedback, collaborative tasks with human partners, or any scenario where objects move independently of the robot — are effectively out of scope. Even for the static tasks evaluated, the 4.9-second pause between action chunks (which each span 2 seconds of execution at 25 Hz) means the robot is idle during planning for over twice as long as it spends executing. For applications where throughput matters (e.g., warehouse picking, manufacturing), this idle time represents a direct productivity cost. The paper does not characterize how performance degrades with fewer GPUs (and correspondingly longer latency) or with fewer denoising steps during planning (which could trade accuracy for speed).

What evidence exists in the paper. The latency measurements are reported in Appendix A.4.2: 0.61 seconds for direct policy inference (5 denoising steps, 1 H100 GPU for RoboCasa), 0.16 seconds with 1 denoising step, and 4.9 seconds for planning (8 H100 GPUs). No latency-accuracy tradeoff curves are presented. The paper demonstrates that direct policy inference with 1 denoising step achieves 66.4% RoboCasa success rate vs. 67.1% with 5 steps (Table 5, 0.7-point drop for a ~4× speedup), but no analogous experiment is done for the planning pipeline. The planning experiments are conducted only on static bimanual manipulation tasks in a controlled lab setting.

Mitigation status. The paper acknowledges that slower inference "may limit applicability to dynamic tasks" and identifies "how to speed up the search" as "an important direction for future study" (Section 6). No concrete speedup strategies are evaluated (e.g., reducing ensemble sizes, sharing computation across candidates, knowledge distillation from the planning model into the policy).

6.3 No Evidence That the Method Works Beyond Manipulation with Rigid Objects in Controlled Settings

The assumption or constraint. All experiments — LIBERO (single-arm tabletop manipulation), RoboCasa (single-arm kitchen manipulation), and ALOHA (bimanual tabletop manipulation) — involve quasi-static tasks with rigid or semi-rigid objects (blocks, bowls, utensils, shirts, candies, ziploc bags) in environments where lighting, camera positions, and background are either fixed (simulation) or carefully controlled (lab). The robot platforms are all position-controlled arms with parallel-jaw grippers or simple end-effectors. The paper does not evaluate on: (a) mobile manipulation or navigation, (b) tasks with deformable objects beyond shirt-folding (which is itself highly structured), (c) tasks requiring force control or tactile feedback, (d) dynamic tasks with moving objects or time pressure, (e) multi-agent or human-robot collaboration, (f) tasks with substantial visual diversity (outdoor settings, varying lighting, cluttered backgrounds), or (g) any non-manipulation robotics domain.

The consequence. The paper's central claim — that pretrained video model priors transfer effectively to robot control — is validated only for a narrow slice of the robotics problem space. It is plausible that video model pretraining on Internet videos is specifically well-suited to tabletop object manipulation (since many Internet videos depict hands manipulating objects on tables) but would provide less benefit for tasks where the embodiment, dynamics, or visual appearance differ substantially from typical Internet video content. The latent frame injection mechanism is described as flexible across embodiments (Section 4.1), but this flexibility is never tested across different robot morphologies, action spaces, or sensor configurations. A practitioner working on, say, quadruped locomotion or drone navigation cannot infer from this paper whether Cosmos Policy would help their application. Similarly, the absence of force/torque sensing or tactile feedback in the evaluated platforms means the method's ability to incorporate high-frequency contact information is unknown.

What evidence exists in the paper. The three evaluation domains are all tabletop manipulation with position-controlled arms. The paper's generalization claims are supported by OOD evaluations within the ALOHA tasks (Table 3: unseen shirts, unseen bowls, uneven candy distributions), not by cross-domain or cross-embodiment transfer experiments. The RoboCasa results do include "scenes with different floor plans and styles" and "unseen object instances" (Section 5.1), so there is evidence of visual generalization within the kitchen manipulation domain. However, no experiment tests whether a Cosmos Policy trained on one robot morphology transfers to another, or whether the method handles fundamentally different control modalities (velocity control, impedance control, whole-body control).

Mitigation status. Not addressed as a limitation. The paper's scope claims are descriptive rather than prescriptive — it does not claim to work beyond manipulation, but it also does not discuss the narrowness of the evaluated domains as a limitation that future work should address. The flexibility of latent frame injection for different embodiments is mentioned (Section 4.1) but not tested.

6.4 The Comparison to VLAs Does Not Isolate the Effect of Video Pretraining vs. Architecture

The assumption or constraint. Cosmos Policy's claim to match or exceed fine-tuned VLAs — "despite not having benefited from similar large-scale action supervision" (Section 5.2) — rests on cross-model comparisons that confound pretraining domain (videos vs. static images), model architecture (diffusion transformer vs. various VLM architectures), model scale (2B for Cosmos-Predict2 vs. varying scales for VLAs), pretraining data quantity and composition, and fine-tuning protocol. The paper does not control for any of these variables. Specifically, there is no experiment where the same underlying architecture is pretrained on videos vs. static images vs. robot action data and then fine-tuned on the same manipulation tasks — which would be the clean causal test of the "video pretraining > VLA pretraining" hypothesis.

The consequence. The reported performance advantage over VLAs (e.g., 93.6% vs. 88.6% average ALOHA score vs. π0.5; Figure 4) could be attributable to factors other than the video pretraining domain. Cosmos-Predict2's diffusion transformer architecture may simply be more effective at modeling the action distribution for these particular tasks than π0.5's flow-matching architecture or OpenVLA's autoregressive discretization. The 2B parameter scale may sit in a favorable regime relative to the available fine-tuning data (185 ALOHA demos) compared to the 7B+ scales of some VLAs. The Cosmos-Predict2 pretraining data mixture (millions of Internet videos) may, by fortunate accident, contain content highly relevant to the specific ALOHA tasks (e.g., videos of people folding shirts, opening ziploc bags, picking up candy). The observed performance differences therefore do not constitute evidence that any video model will outperform any VLA — they constitute evidence that this specific video model, fine-tuned with this specific protocol, outperforms these specific VLAs on these specific benchmarks. The paper's language sometimes implies a stronger conclusion (e.g., "This finding suggests that video model priors provide a strong initialization for control policies without requiring additional action-labeled robot data"; Section 5.2), but the experimental design does not support causal attribution to the video pretraining domain.

What evidence exists in the paper. The LIBERO results (Table 1) show Cosmos Policy at 98.5% vs. CogVLA at 97.4%, OpenVLA-OFT at 97.1%, and π0.5 at 96.9%. The ALOHA results (Figure 4, Table 3) show Cosmos Policy at 93.6% vs. π0.5 at 88.6%. The from-scratch ablation (Table 4) isolates the value of Cosmos-Predict2's pretrained weights within the same architecture (98.5% → 94.6%), but this does not isolate the video pretraining domain — it could be that any large-scale pretraining (on any temporally structured data, or even on static images) would provide similar benefits, and the video domain is incidental. The paper does not compare against a version of Cosmos-Predict2 pretrained on a non-video dataset (e.g., static images) of comparable scale, which would begin to isolate the domain effect.

Mitigation status. Not acknowledged as a limitation. The paper's qualitative discussion (Section 5.2) explicitly attributes performance differences to video model priors vs. VLM priors, but the experimental design does not support causal attribution. The paper does not discuss the confounds or propose controlled experiments to address them.

6.5 Real-World Evaluation Sample Sizes Are Too Small to Reliably Rank Top-Performing Methods

The assumption or constraint. The ALOHA robot evaluations consist of 101 total trials per method across four tasks: 30 for "put X on plate," 20 for "fold shirt," 25 for "put candies in bowl," and 26 for "put candy in ziploc bag." These are further split into in-distribution and OOD conditions, yielding as few as 8 OOD trials for some task-condition combinations (e.g., "fold shirt" OOD: 8 trials; Table 3). No confidence intervals, standard errors, or statistical significance tests are reported for any result. The planning experiments (Section 5.3, Figure 7) use an unspecified number of "challenging initial states" across the two hardest tasks. The paper does not describe a protocol for selecting these initial states, nor does it randomize over state variations beyond what is described for each task.

The consequence. With these sample sizes, the reported performance differences between top methods — particularly between Cosmos Policy (93.6%) and π0.5 (88.6%) on ALOHA — may not be statistically reliable. A difference of 5 percentage points based on 101 trials corresponds to roughly 5 more successful task completions. Given the stochasticity inherent in real-world manipulation (minor variations in object placement, lighting, robot calibration, gripper wear), a 5-trial difference could arise from sampling noise rather than genuine policy superiority. The reversal in OOD performance (π0.5 at 92.5% vs. Cosmos Policy at 89.3%, a 3.2-point gap in the opposite direction) further suggests that the rankings are fragile and may not replicate with different trial sets. For the planning experiments, the absence of reported trial counts and the use of "challenging initial states" (which may be hand-selected to demonstrate improvement) make the 12.5-point gain uninterpretable as an expected improvement on a randomly sampled test distribution. A practitioner choosing between Cosmos Policy and π0.5 for their own bimanual manipulation task cannot determine from these results whether the 5-point average difference is real or noise.

What evidence exists in the paper. Table 3 provides the per-task, per-condition breakdown, confirming small sample sizes. No statistical measures are reported. The planning results in Figure 7 are displayed as a bar chart without numerical labels in the text, and the associated trial counts are not stated. The paper states that "We ensure fair comparison between methods by using the same fixed set of initial states for each method" (Section 5.1), which controls for one source of variance (state sampling) but does not address the fundamental issue that the fixed set itself may not be representative, and that small sample sizes amplify the influence of any particular initial state.

Mitigation status. Not addressed as a limitation. The paper reports aggregate scores and in-distribution/OOD breakdowns but does not discuss the statistical reliability of the comparisons or acknowledge the sample-size limitation. The simulation benchmarks (LIBERO: 6000 trials; RoboCasa: 3600 trials) do not suffer from this issue, but the real-world results — which are arguably the most impactful and the ones most emphasized in qualitative discussion — are based on an order of magnitude fewer trials.

6.6 The Method Has Not Been Demonstrated With Any Video Model Other Than Cosmos-Predict2-2B

The assumption or constraint. Every experiment in the paper uses Cosmos-Predict2-2B-Video2World as the pretrained backbone. The paper claims that "the video model's core learning algorithm and architecture" are what enable the transfer to robot control (Section 1, Section 4), and that latent frame injection is applicable "for any particular robot setup" (Section 4.1). However, the approach has never been tested with any other video generation model — not with other Cosmos variants, not with Wan, not with CogVideoX, not with HunyuanVideo, not with Vidu, and not with any smaller or larger model from any family.

The consequence. It is unknown whether Cosmos Policy's performance is specific to Cosmos-Predict2-2B — its particular architecture, pretraining data, training recipe, or scale — or whether the latent frame injection approach would work with comparable effectiveness on any video diffusion model of sufficient quality. This matters because Cosmos-Predict2 is a proprietary model with specific licensing, compute requirements, and availability constraints. A practitioner who cannot access Cosmos-Predict2 (or who wants to use a more permissively licensed alternative like Wan or Open-Sora) cannot assume the approach will transfer. Additionally, the 2B parameter scale is a specific choice that may interact with the amount of available robot demonstration data — a 7B or 20B video model might overfit the small demonstration sets, while a 300M model might lack sufficient capacity. The paper provides no guidance on how model scale affects performance, making it difficult for practitioners to select an appropriately sized backbone.

What evidence exists in the paper. Only Cosmos-Predict2-2B is used. The from-scratch ablation (Table 4) shows that the architecture without pretrained weights achieves 94.6% on LIBERO, which indicates that the architecture itself is capable, but it does not test whether pretrained weights from a different video model would provide similar benefits. No experiment compares multiple video model backbones, or even multiple sizes of the Cosmos-Predict2 family. The paper's claims about the generality of the approach (e.g., "we hypothesize that their learning algorithms are well-suited for representing actions alongside other modalities"; Section 1) are not tested.

Mitigation status. Not addressed as a limitation. The paper's title and framing ("Cosmos Policy") appropriately scope the contribution to the Cosmos model family, but the technical claims about the approach's generality are not accompanied by multi-model evidence. The release of code and models may enable the community to test with other backbones, but the paper itself provides no such evidence.

7. Implications and Future Directions

How This Work Changes the Landscape

Cosmos Policy represents a methodological reframing rather than a paradigm shift. Its primary contribution is not a new class of algorithms but a demonstration that a remarkably simple adaptation strategy—encoding non-image modalities as latent frames within a video diffusion model's native sequence, with no architectural modifications—matches or exceeds the performance of substantially more complex, multi-stage, multi-module approaches that have become the norm in video-model-based robotics and VLA research.

The shift is from architecture-centric to data-representation-centric design. Prior work on adapting video models for robot control (Video Policy, FlowVLA, ViDAR, RoboEnvision, Latent Policy Steering) has converged on a shared assumption: video models need architectural augmentation—separate action heads, inverse dynamics models, action diffusers trained in a second stage—to produce robot commands. Cosmos Policy challenges this assumption by showing that a generic video diffusion model, with no robotics-specific design affordances, can learn to generate high-precision action chunks, future observations, and scalar value estimates when these modalities are simply interleaved as additional "frames" in the latent sequence, using the exact same denoising objective the model was pretrained with. The ablation chain in Table 5 is the most direct evidence for this reframing: stripping away world model training, value function training, and auxiliary targets progressively degrades performance from 67.1% to 44.4% on RoboCasa, but the architecture remains identical throughout. The representational richness of the training scheme—what the model is asked to predict and condition on—matters far more than architectural customization.

This reframing reconciles a tension in the literature that the paper identifies but does not fully resolve. On one side, multi-stage video-to-policy methods achieve strong results by carefully engineering separate action modules, implying that video models alone are insufficient for control. On the other side, unified video-action models trained from scratch (UVA, UWM) demonstrate that joint video-action prediction is effective, but they sacrifice pretrained priors by requiring custom architectures. Cosmos Policy synthesizes these approaches by showing that a pretrained video model can be adapted for joint video-action prediction through a data representation trick alone, obtaining the benefits of both pretrained priors and unified modeling without the engineering complexity of either prior approach. The key finding is that the Wan2.1 VAE latent space—designed purely for video compression—is sufficiently expressive to encode robot actions, proprioception, and scalar values as if they were visual data, and the diffusion transformer's self-attention mechanism—designed for spatiotemporal coherence across image patches—can learn the causal relationships between these modalities when they are arranged in the causal order $(s, a, s', V(s'))$ .

However, the magnitude of this contribution should be calibrated carefully. The paper does not demonstrate that any video model can be adapted this way—only Cosmos-Predict2-2B is tested. It does not demonstrate that any non-image modality can be injected through latent frames—only low-dimensional vectors (actions, joint angles, scalar values) are tested. It does not demonstrate that latent frame injection works at scales substantially different from 2B parameters, or that the approach transfers across substantially different robot morphologies. The contribution is therefore a specific, well-validated existence proof that zero-architecture-change adaptation of a video model into a policy–world-model–value-function system is possible and effective, but the generality of this finding across models, embodiments, and modalities remains to be established. The paper's framing sometimes implies stronger conclusions (e.g., "video model priors provide a strong initialization for control policies without requiring additional action-labeled robot data"; Section 5.2), but the experimental design supports a more specific claim: Cosmos-Predict2-2B, adapted via latent frame injection with joint training objectives, achieves state-of-the-art performance on three manipulation benchmarks.

The work reshapes research priorities in two specific ways. First, it redirects attention from architecture engineering to training scheme design for video-model-based policies. The paper's most impactful practical finding is not latent frame injection per se—which is essentially a data formatting trick—but the joint training scheme where a single model learns policy, world model, and value function simultaneously through different conditioning masks on the same latent sequence, with the 22.7-point RoboCasa gap between full training and action-only training (Table 5) being the clearest signal that what you train the model to predict dominates how you architect the model. Second, it opens a new axis of comparison between video-model-based policies and VLAs that is not about architecture but about pretraining data domain. The paper's finding that Cosmos Policy outperforms fine-tuned VLAs on precision manipulation tasks while lagging slightly on OOD semantic generalization (Table 3) suggests that video pretraining and VLM pretraining provide complementary capabilities rather than one being strictly superior—a hypothesis that the paper does not test directly but that naturally emerges from its results.

What the work does NOT change. The paper does not establish that video model pretraining is generally preferable to VLA pretraining for robotics—the confounded comparisons (different architectures, scales, pretraining data mixtures) prevent causal attribution. It does not demonstrate that model-based planning in robotics scales with compute in the way that LLM test-time compute scaling does—the planning experiments are limited to two tasks, one dataset size, and one search budget ( $N = 8$ ). It does not solve the latency problem that limits diffusion-based policies for dynamic control—the 4.9-second planning latency on 8 H100 GPUs is a substantial barrier to broader applicability. And it does not provide evidence beyond tabletop manipulation with rigid objects—whether the approach transfers to deformable object manipulation beyond shirt-folding, contact-rich tasks, mobile manipulation, or locomotion is entirely open.

Follow-Up Research This Work Enables

Cross-video-model replication of latent frame injection. The paper demonstrates latent frame injection exclusively with Cosmos-Predict2-2B. The most immediate question is whether the approach transfers to other video diffusion models—Wan2.1 (the VAE used by Cosmos), CogVideoX, HunyuanVideo, Vidu, or Open-Sora—which differ in architecture (DiT vs. 3D-UNet), latent space structure (different VAE compression ratios and channel dimensions), pretraining data composition, and scale (from ~1B to ~20B parameters). A strong follow-up would replicate the RoboCasa evaluation (50 demos per task, 24 tasks) across at least three different video model families, keeping the joint training scheme and latent injection mechanism identical, and report per-model success rates. If performance is consistent across models, it establishes latent frame injection as a general technique rather than a Cosmos-specific trick. If performance varies substantially, it reveals which properties of the video model (architecture, scale, pretraining data, VAE design) are load-bearing, guiding backbone selection. The paper's code and model release makes this feasible by providing a reference implementation of the injection mechanism.

Scaling behavior of planning performance with rollout data quantity. The paper's planning experiments use exactly 648 rollouts for two ALOHA tasks, with no ablation of this number. A critical open question is the shape of the learning curve: does planning improve linearly with rollout data, logarithmically (with diminishing returns), or discontinuously (requiring a minimum threshold before the world model becomes useful)? A strong follow-up would collect rollout datasets of increasing size (e.g., 50, 100, 200, 400, 800, 1600 trajectories) for a fixed set of tasks in simulation (RoboCasa or LIBERO, where large-scale evaluation is feasible), fine-tune the planning model on each dataset size, and measure planning performance (best-of-N with $N = 8$ ) as a function of dataset size. This would produce a planning scaling law analogous to neural scaling laws in pretraining, telling practitioners how much data to collect for a target improvement. It would also reveal whether the 12.5-point ALOHA improvement was near the asymptote (suggesting limited headroom) or on the steep part of the curve (suggesting larger gains with more data). This experiment is currently infeasible in the real world due to data collection cost, but it is directly tractable in simulation using the same Cosmos Policy training pipeline the paper has already established.

Search budget scaling and over-optimization detection for robot planning. The paper uses best-of-N with $N = 8$ and ensemble sizes of 3 (world model) and 5 (value function), for a total of 3 × 5 = 15 value estimates per action proposal. It does not vary $N$ , the ensemble sizes, or the aggregation method. A critical follow-up would sweep $N \in \{1, 2, 4, 8, 16, 32\}$ in both simulation and real-world settings, measuring whether planning performance continues to improve with search breadth or whether over-optimization (analogous to the reward hacking documented in LLM test-time search) causes degradation at high $N$ . This would address the implicit question raised by Innovation 4 (Section 4 of the prior analysis): are we in the under-modeling regime where more search helps, or does there exist a threshold beyond which the value function is exploited? The experiment would be conducted by taking the fine-tuned planning checkpoint from the ALOHA experiments (or a simulation equivalent), running best-of-N at each $N$ level, and plotting success rate vs. $N$ . If success rate monotonically increases and plateaus, we are in the under-modeling regime and the bottleneck is world model accuracy. If it increases then decreases, over-optimization has set in and the research priority shifts to verifier robustness. The "majority mean" aggregation method could be ablated against simpler alternatives (naive mean, median, trimmed mean) to understand whether robust aggregation is load-bearing for larger $N$ .

Combining video model and VLA pretraining. The paper's results (Table 1, Figure 4, Table 3) suggest a complementary pattern: Cosmos Policy (video pretraining) excels at precise execution of demonstrated skills, while fine-tuned VLAs (π0.5) show a slight advantage on OOD semantic generalization. This naturally motivates a combined approach: initialize from a video model that has also been pretrained on static image-text data (or co-train a shared backbone on both video prediction and image-text alignment), then fine-tune with the Cosmos Policy joint training scheme. A concrete experiment would compare three initializations on the ALOHA tasks: (a) Cosmos-Predict2 (video only, existing result), (b) a VLM of comparable scale fine-tuned with the same joint training scheme (testing whether the training scheme transfers), and (c) a model pretrained on both video and image-text data (if available). The dependent variables would be in-distribution success rate (where video priors should help), OOD generalization (where VLM priors should help), and the interaction between them. This would directly test whether the two pretraining paradigms are complementary or competitive.

Continuous and dynamic allocation of test-time compute within a single episode. The paper's planning uses a fixed $N = 8$ search budget uniformly across all timesteps. However, some timesteps in a task are more critical than others—grasping the ziploc bag slider requires more precision than lifting the already-grasped bag, and a policy that could dynamically allocate a larger search budget to critical steps and a smaller budget to routine steps would be more efficient. A strong follow-up would implement an adaptive search budget: at each timestep, generate a small number of action proposals (e.g., $N = 2$ ), use the planning model to estimate the variance of the predicted values (as a proxy for step criticality), and if variance is high (disagreement about outcomes, implying a critical decision), generate additional proposals up to a maximum budget of $N = 16$ . Total compute would be similar to or less than fixed $N = 8$ , but allocated more efficiently. This connects directly to the concept of compute-optimal test-time scaling explored in the reference Cosmos paper on LLM inference scaling, adapted to the robot planning setting. The experiment would compare fixed-budget vs. adaptive-budget planning on the ALOHA tasks, measuring success rate and average inference time.

Stress-testing via adversarial perturbation of initial conditions. The OOD evaluation in Table 3 tests generalization to unseen objects and arrangements, but it does not systematically probe the boundaries of Cosmos Policy's robustness. A strong negative-result experiment would identify what kinds of distribution shift cause the policy to fail. For example: systematically vary lighting conditions (brightness, color temperature, shadow patterns) in simulation, measure success rate as a function of perturbation magnitude, and compare degradation between Cosmos Policy and a fine-tuned VLA. Or: introduce visual distractors (irrelevant objects, background clutter) at varying densities, and measure whether video model priors provide greater robustness to visual noise than VLM priors (since video models were trained on noisy, diverse Internet videos) or greater fragility (since video models may rely more heavily on specific visual features for dynamics prediction). This would characterize the failure modes of video-model-based policies and inform deployment requirements (e.g., "lighting must be controlled within ±X lux of training conditions").

Practical Applications and Downstream Use Cases

Low-data robot programming for small-batch manufacturing. Cosmos Policy's RoboCasa result—67.1% average success rate using only 50 human demonstrations per task, matching or exceeding methods that require 300–3000 demonstrations—has direct implications for small-batch manufacturing where collecting hundreds of demonstrations per new task is economically infeasible. In a scenario where a factory needs to program a robot for 24 kitchen-like manipulation tasks (the RoboCasa setup), Cosmos Policy requires 1,200 total human demonstrations (50 × 24) versus 7,200+ for prior methods, representing an estimated 40+ hours of saved teleoperation time (assuming ~2 minutes per demonstration including resets). The practical workflow would be: (1) deploy Cosmos-Predict2-2B pre-loaded on a workshop GPU cluster, (2) collect 50 teleoperated demonstrations per new task, (3) fine-tune with the joint training scheme (~48 hours on 32 H100 GPUs, per the paper's RoboCasa setup), (4) deploy the resulting policy. For tasks where the direct policy's 67.1% success rate is insufficient, collect an additional few hundred autonomous rollouts (Section 5.3) and enable planning for a projected 5–15 point improvement. The primary practical barrier is GPU cost—the training requires 32 H100 GPUs—but this is a capital expenditure rather than the recurring labor cost of demonstration collection, and GPU time costs decrease over time while human teleoperation costs do not.

Distillation of planning-enhanced policies for latency-sensitive deployment. The paper's planning pipeline achieves a 12.5-point improvement but requires 4.9 seconds and 8 H100 GPUs per action chunk—prohibitive for applications requiring sub-second reaction times or edge deployment. A natural downstream application is to use the planning model to generate a dataset of improved action labels, then distill these back into the direct policy (which runs at 0.95 seconds on 1 H100 GPU) through behavioral cloning. Concretely: (1) deploy Cosmos Policy with planning on a set of initial states, (2) collect the action chunks that were selected through best-of-N search (the "planned" actions), (3) fine-tune the direct policy to predict these planned actions directly (without search) using the same latent frame injection training scheme but with the planned actions as targets, (4) deploy the distilled policy at 0.95-second latency with no planning overhead. The hypothesis is that the distilled policy would retain some fraction of the 12.5-point planning gain while maintaining direct-policy inference speed. The paper already has all the components to test this—the planning model, the rollout collection pipeline, and the training infrastructure—making it a low-risk, high-impact extension.

Simulation-based data augmentation using the world model. Cosmos Policy's world model ( $p(s' \mid s, a)$ ) generates future observations conditioned on current observations and actions. While the paper uses this for planning (predicting outcomes of candidate actions), the world model could also be used for data augmentation: given the 50 human demonstrations, generate synthetic demonstrations by applying the demonstrated actions from perturbed initial states, using the world model to predict the resulting observations, and filtering for trajectories that the value function predicts to be successful. This would create a larger, filtered training dataset that expands the policy's experience beyond the original 50 demonstrations, potentially improving robustness to small variations in initial conditions. The key question is whether the world model's predictions are sufficiently accurate to serve as training targets—the paper's Figure 6 suggests the fine-tuned world model is reasonably accurate, but systematic quantification is needed. A practical workflow: (1) train Cosmos Policy on 50 demos, (2) fine-tune the planning model on policy rollouts (as in Section 5.3), (3) use the planning model to generate synthetic demonstrations from perturbed starting states, (4) retrain the policy on the augmented dataset, (5) measure improvement over the original 50-demo policy. This leverages the same planning model investment for both test-time search and training-time data augmentation.

When to Prefer This Method

The paper implicitly articulates a tradeoff between Cosmos Policy (video model backbone) and fine-tuned VLAs (vision-language model backbone), though it does not formalize it as an explicit decision rule. Based on the empirical patterns in Tables 1–3 and Figure 4, the following conditions emerge:

Prefer Cosmos Policy (or analogous video model fine-tuning) when:

The manipulation tasks require high action precision and exhibit high action multimodality (e.g., the ziploc bag and candy tasks in ALOHA, where Cosmos Policy outperforms π0.5 by 23.9 and −5.6 points respectively).
Demonstration data is severely limited (50 demos per task or fewer), where the paper's RoboCasa results show a 17.1-point advantage over UVA (another unified approach) and substantial advantages over methods requiring 6× more data.
Tasks are long-horizon and require understanding of sequential physical interactions, where the LIBERO-Long results show Cosmos Policy exceeding the next-best method (CogVLA) by 2.2 points and the from-scratch ablation shows a 9.0-point pretraining dividend specific to long-horizon tasks.
A GPU cluster is available for inference (1 H100 GPU for direct policy, 8 for planning), since the method requires specialized hardware for the diffusion sampling process.
Quasi-static manipulation is the target domain, since the 0.95–4.9 second inference latency precludes dynamic control.

Prefer fine-tuned VLAs when:

The evaluation distribution includes substantial semantic novelty (unseen object categories, novel instruction phrasing) that benefits from VLM pretraining on diverse image-text data, as suggested by π0.5's 16-point OOD advantage on "put candies in bowl" (Table 3).
Open-vocabulary instruction following is a primary requirement, since VLAs are explicitly trained on language-conditioned tasks while Cosmos Policy inherits text conditioning from Cosmos-Predict2's video captioning objective (which may be weaker).
Inference latency is critical and sub-second reaction times are required, since VLA inference (typically single forward pass) is faster than iterative diffusion sampling.
The robot platform has a substantially different embodiment or camera configuration than those tested in the paper (Franka, ViperX with wrist and third-person cameras), since Cosmos Policy's latent injection scheme has not been validated across diverse embodiments.

These preferences are hypotheses grounded in the paper's results, not confirmed by controlled experiments. The confounded nature of the cross-model comparisons (different architectures, scales, pretraining data) means the true causal factors behind performance differences—pretraining domain vs. architecture vs. scale vs. training protocol—remain unresolved, and a practitioner's choice between the two paradigms should be informed by task-level validation rather than assumed from benchmark rankings alone.