Cosmos World Foundation Model Platform for Physical AI

ArXiv: 2501.03575

🎯 Pitch

A single pre-trained video generation model can become a digital twin for robots, self-driving cars, and more, achieving up to 95% better 3D consistency than existing methods. By fine-tuning on just a small amount of specialized data, the same base model can control a robotic arm or generate autonomous driving scenes with startling accuracy, finally solving the chicken-and-egg problem of safely training physical AI.

---

1. Executive Summary

This paper introduces the Cosmos World Foundation Model Platform, a comprehensive system for building and customizing world foundation models (WFMs)—digital twins of the physical world that Physical AI systems can safely interact with for policy evaluation, planning, and synthetic data generation. The platform trains transformer-based diffusion and autoregressive WFMs on approximately 100M video clips curated from a 20M-hour collection, using a video curation pipeline, causal video tokenizers (both continuous and discrete), and a pre-training-then-post-training paradigm that produces generalist WFMs and specializes them for downstream tasks. The diffusion-based 14B model generates 3D-consistent videos achieving a 62.6% camera pose estimation success rate on static scenes (versus 4.4% for a VideoLDM baseline) and 4× better multi-view geometric consistency than prior work, while the platform demonstrates that post-trained WFMs can incorporate camera control, robotic action conditioning, and autonomous driving trajectories with state-of-the-art fidelity—achieving, for instance, a camera trajectory rotation error of only 1.646° compared to 8.277° for CamCo, establishing that pre-trained world foundation models transfer effectively to diverse Physical AI setups when fine-tuned on domain-specific data.

2. Context and Motivation

The Core Problem: Physical AI Cannot Safely Scale Through Real-World Interaction Alone

The fundamental challenge this paper tackles is deceptively simple: how do you train a robot, autonomous vehicle, or any Physical AI system to understand and predict the consequences of its actions without letting it cause damage in the real world? Physical AI systems—embodied agents equipped with sensors and actuators that observe and interact with their environment—face a data scaling bottleneck unlike any other AI domain.

The problem manifests concretely in the exploration-exploitation tradeoff that all learning systems face, but with Physical AI the stakes are existential. An LLM generating a bad token during training wastes a few FLOPs; a humanoid robot executing an incorrect action during exploration can destroy itself, damage its surroundings, or injure humans. The paper states this directly in Section 1:

"These actions perturb the physical world and may cause severe damage to the system and the world. This is especially true when the AI is still in its infancy when exploratory actions are essential."

This creates a catch-22 for Physical AI: the system needs diverse, real-world interaction data to learn robust behaviors, but it cannot safely collect that data until it already knows how to behave safely. Unlike language, vision, or code—where billions of training examples exist on the internet—interleaved observation-action sequences for physical tasks are scarce, expensive to collect, and dangerous to obtain at scale.

Why This Problem Matters

The importance extends beyond academic interest into three domains with massive societal and economic implications:

1. Robotics and industrial automation. General-purpose robots capable of operating in unstructured human environments—factories, warehouses, homes, hospitals—remain largely aspirational despite decades of research. The bottleneck is not hardware (we can build dexterous manipulators) nor conceptually intelligent control algorithms; it's that robots lack access to sufficiently diverse training experiences. A WFM that accurately simulates how objects deform under grasping, how liquids slosh during pouring, or how fabrics move during folding would let a robot practice these skills millions of times digitally before executing them once physically. This is the difference between a robot that can handle only pre-programmed motions in structured settings and one that can adapt to novel situations.

2. Autonomous driving. Training self-driving systems requires exposure to rare and dangerous scenarios—children running into the street, vehicles running red lights, sudden weather changes—that are unethical to stage and impossible to collect at scale from real-world driving logs. A multi-view, trajectory-controllable WFM that generates photorealistic, physically consistent driving videos conditioned on specific scenarios could dramatically improve the safety and robustness of autonomous vehicle policies by enabling training and evaluation on millions of edge cases that occur only once per billion miles of real driving.

3. Scientific and engineering simulation. Beyond robotics and driving, accurate world models could accelerate progress in fields ranging from climate science (simulating weather dynamics) to drug discovery (simulating molecular interactions) to civil engineering (simulating structural responses to stress). While this paper focuses on visual WFMs for Physical AI, the concept of learning world simulators from video data generalizes across domains.

The Limitation of Existing World Models

The paper positions itself against two broad categories of prior work: classical model-based approaches and recent learning-based world models trained on limited data.

Classical Physics-Based Simulators

Traditional approaches in robotics and control use explicit mathematical models derived from physics—rigid body dynamics, fluid dynamics, contact mechanics—coded into simulation engines like MuJoCo, Isaac Sim, or PyBullet. While these simulators are physically grounded by construction, they suffer from a fundamental limitation: they cannot capture the full visual complexity and diversity of the real world. A physics engine can simulate a block sliding down an inclined plane with perfect Newtonian fidelity, but it cannot render the varied visual appearances of real blocks—wood grain, paint flecks, scuff marks—nor the infinite variety of real-world backgrounds, lighting conditions, and camera viewpoints that a Physical AI system would encounter during deployment.

More critically, classical simulators require manual specification of asset geometry, material properties, and environmental parameters for every scenario. This makes them non-scalable: you cannot hand-engineer a simulator for every possible kitchen, warehouse, or street corner a robot might encounter. The result is a well-known sim-to-real gap: policies trained in classical simulators often fail when transferred to real-world hardware because the visual and physical details do not match.

Learning-Based World Models

The concept of learned world models dates to the seminal work of Ha and Schmidhuber (2018), who proposed training neural networks to predict future states given current observations and actions. Subsequent work in this lineage—particularly the Dreamer family (Hafner et al., 2019, 2021, 2023) and related approaches—operates in learned latent spaces, where a recurrent or transformer backbone models state evolution in a compact representation produced by an autoencoder.

The paper identifies several critical limitations of these prior learned world models:

1. Scale-limited training. Previous world models have been trained on small, domain-specific datasets—typically hundreds of hours of video from a single environment like Atari games or a specific robotic setup. This produces specialists that cannot generalize: a world model trained on a Franka arm manipulating blocks in a lab bench setup cannot predict observations for a humanoid robot folding laundry in a home environment. The paper explicitly argues that scaling training data is the missing ingredient:

"Data determines the ceiling of an AI model." (Section 1)

The Cosmos platform targets approximately 100M video clips spanning nine distinct categories (driving, hand manipulation, human activity, spatial navigation, first-person POV, nature dynamics, camera movements, synthetic rendering, and others)—roughly three orders of magnitude more diverse than prior work.

2. Latent-space limitation. Many prior world models (Dreamer, DreamerV2, TD-MPC) operate on low-dimensional latent representations optimized for control rather than visual fidelity. They can predict state evolution usefully for policy learning, but they cannot generate photorealistic video observations that a human could evaluate or that could serve as synthetic data for vision-based policies. This limits their utility for debugging, evaluation, and sim-to-real transfer.

3. Absence of foundation model pre-training. Prior world models are trained from scratch on the target domain. There exists no concept of a pre-trained world model—a generalist that captures universal physical dynamics (gravity, collision, object permanence, perspective consistency) from broad video data and can then be fine-tuned on small amounts of domain-specific data. The Cosmos paper explicitly introduces this pre-training-then-post-training paradigm:

"We position a world foundation model as a general-purpose world model that can be fine-tuned into customized world models for downstream applications." (Abstract)

This mirrors the now-standard paradigm in NLP and vision (BERT → fine-tuned BERT, CLIP → fine-tuned CLIP) but applies it to world simulation.

Where Video Generative Models Fall Short as World Models

The paper acknowledges rapid progress in video generative models—Sora, Dream Machine, Kling, Gen-3—that can produce high-quality, photorealistic videos from text prompts. However, it identifies a critical gap: text-to-video generation is not world simulation. The distinction is central to the paper's motivation:

1. Lack of action conditioning. Text-to-video models generate videos from language descriptions, not from actions or perturbations. A Physical AI system needs to predict what happens given that it executes a specific action—e.g., "what will the camera see if the end-effector moves 3cm to the left?" Text descriptions cannot capture the precise, continuous, low-level control signals that robots and vehicles use.

2. Insufficient multi-view and 3D consistency. Most video generative models produce videos from a single, fixed viewpoint. Autonomous vehicles have six or more cameras; robots need to reason about 3D scene geometry for grasping and navigation. A useful world simulator must maintain geometric consistency across views and over time—a capability that text-to-video models are not designed to provide.

3. No explicit model of world state. Video generative models learn a mapping from text to pixels without learning an explicit representation of the underlying physical state. They can produce visually impressive videos that violate physics—objects changing shape, disappearing and reappearing, defying gravity—because they optimize for perceptual quality, not physical fidelity. The Cosmos paper acknowledges this explicitly in its physics alignment evaluation (Section 5.3.2), where even their largest models struggle with rigid-body dynamics, object permanence, and contact-rich interactions.

4. Insufficient evaluation frameworks. The paper notes that evaluating whether a generated video is physically consistent presents a significant challenge that the field has not adequately addressed:

"Defining robust rubrics for humans to evaluate physical fidelity is hard as such assessments are often influenced by personal biases, backgrounds, and other subjective factors." (Section 9)

The Cosmos platform introduces structured evaluations—3D geometric consistency via Sampson error and camera pose estimation, physics alignment via controlled simulation rollouts—that represent a substantive contribution to benchmarking world models beyond visual quality alone.

The Tokenization Bottleneck

A less obvious but critical gap the paper identifies is the lack of high-quality, efficient video tokenizers suitable for world foundation model training. Tokenizers compress raw video into compact representations that make training computationally tractable—the Cosmos diffusion WFM uses an $8 \times 8 \times 8$ compression factor, reducing video data by 512×. The quality of the tokenizer directly bounds the quality of any downstream WFM.

Existing tokenizers, the paper argues, suffer from multiple deficiencies:

• No causal mechanism. Most video tokenizers process all frames simultaneously using non-causal attention or 3D convolutions that peek at future frames. This prevents joint image-and-video training (since a causal tokenizer treating a single image as a one-frame video is also an image tokenizer) and misaligns with the causal nature of Physical AI systems that operate forward in time.

• Limited to narrow resolutions and aspect ratios. Prior tokenizers typically train on fixed resolutions (e.g., 256×256) and square aspect ratios. Real-world video for Physical AI spans diverse aspect ratios (1:1, 9:16 for smartphone video, 16:9 for dashcams) and resolutions up to 4K.

• Poor compression-quality tradeoffs. The paper's TokenBench evaluation (Figure 8) shows that Cosmos tokenizers achieve a ~4 dB PSNR improvement over the next best tokenizer at equivalent compression rates. This improvement is not incremental—it directly enables training larger WFMs with less distortion.

The Cosmos tokenizer suite (Section 4) directly addresses these gaps with causal temporal convolution and attention, wavelet-domain processing, and joint image-video training across multiple compression rates.

How This Paper Positions Itself

The Cosmos platform defines itself as a foundation model platform for world simulation, explicitly drawing parallels to foundation models in language and vision while acknowledging that world modeling is a harder, earlier-stage problem. The key positioning claims are:

1. It is a platform, not a single model. The paper releases a family of models—diffusion-based (7B, 14B) and autoregressive-based (4B, 5B-Video2World, 12B, 13B-Video2World)—with complementary strengths. Diffusion models produce higher visual quality; autoregressive models leverage LLM inference optimization techniques for real-time generation. The platform includes data curation tools, tokenizers, upsamplers, diffusion decoders, and guardrails—components that downstream developers need to build their own customized WFMs.

2. It demonstrates the pre-training-then-post-training paradigm concretely. Rather than merely proposing the concept, the paper provides empirical evidence across three diverse downstream applications: camera-controllable 3D navigation (Section 6.1), instruction-following and action-conditioned robotic manipulation on two different robot platforms (Section 6.2), and multi-view trajectory-conditioned autonomous driving (Section 6.3). In each case, post-training the pre-trained Cosmos WFM on domain-specific data yields substantial improvements over baselines trained from scratch on the same data—demonstrating that the pre-trained world model captures transferable physical knowledge.

3. It is intentionally incomplete. The paper is unusually candid about what it does not achieve. It explicitly states that the post-trained models are "Sample" models—illustrative demonstrations, not production-ready systems—and that the world foundation model problem is "still far from being solved" (Section 1). The limitations section (Section 9) acknowledges issues with object permanence, contact-rich dynamics, instruction following, and physics alignment. This candor signals that the Cosmos platform is an enabling technology for the research community, not a finished product.

4. It introduces structured evaluation for world models. The 3D consistency evaluation (Sampson error, camera pose estimation success rate, Gaussian splatting view synthesis) and physics alignment evaluation (simulation-based rollouts with pixel-level, feature-level, and object-level metrics) represent methodological contributions that can serve as benchmarks for future work. These evaluations probe specific capabilities that matter for Physical AI—3D geometric consistency, adherence to Newtonian physics, object-level tracking accuracy—rather than relying solely on perceptual quality metrics like FID/FVD that dominate video generation evaluation.

5. It is open-weight and permissively licensed. The paper concludes by making pre-trained WFMs and tokenizers available under the NVIDIA Open Model License, positioning the work as infrastructure for the Physical AI research community. This distinguishes it from proprietary video generation models (Sora, Dream Machine, Gen-3) and aligns with the foundation model philosophy of providing pre-trained models that others can build upon.

The Critical Unresolved Tension

The paper identifies but does not resolve a fundamental tension: autoregressive versus diffusion architectures for world modeling. The evaluation results (Section 5.3.1, Section 6.2) show that diffusion-based WFMs currently produce better visual quality and 3D consistency, and can incorporate diverse control signals through fine-tuning. However, autoregressive WFMs hold two compelling advantages: (1) they can leverage pre-trained LLM weights to inherit extensive world knowledge encoded in text, and (2) they support inference optimizations (speculative decoding, key-value caching) that could enable real-time generation for interactive applications. The paper demonstrates real-time autoregressive generation at 10 FPS on 320×512 video (Section 5.2.4), but does not achieve comparable quality to the diffusion family.

This tension—visual quality versus inference speed, generative fidelity versus architectural flexibility—is presented as an open research question rather than a solved problem, with the paper explicitly stating that "exploring hybrid approaches and their trade-offs remains an active and promising area of research" (Section 9).

Summary of the Gap

The paper addresses a four-way gap: (1) Physical AI lacks scalable training environments, (2) existing learned world models are too small and domain-specific to serve as general-purpose simulators, (3) video generative models lack the action-conditioning, multi-view consistency, and physical fidelity required for world simulation, and (4) video tokenizers have not been designed with the requirements of world model training in mind. The Cosmos platform proposes to fill this gap by providing open-weight, pre-trained world foundation models trained at scale on diverse video data, with the infrastructure (data curation, tokenization, post-training recipes) that enables the community to specialize them for downstream Physical AI applications.

3. Technical Approach

3.1 Reader Orientation

The Cosmos World Foundation Model Platform is a multi-component system for building and customizing video-prediction neural networks that simulate the physical world. It solves the problem of Physical AI data scarcity by providing pre-trained models that can generate realistic, physically-grounded future video frames conditioned on past observations and control inputs, which downstream developers can then fine-tune for specific robots, vehicles, or environments. The "shape" of the solution is a multi-stage pipeline: first, curate a massive, diverse video dataset; second, train efficient causal tokenizers to compress raw video into compact representations; third, pre-train large transformer models (both diffusion and autoregressive) on next-frame prediction across this data; fourth, post-train these generalist models on small amounts of domain-specific data to produce specialists that can be controlled by camera poses, robot actions, text instructions, or vehicle trajectories.

3.2 Big-Picture Architecture (Diagram in Words)

The Cosmos platform comprises seven interconnected components, arranged in a pipeline from raw data to deployable world models:

1. Video Curator (Section 3): Ingests ~20M hours of raw video across diverse categories, splits them into shot-consistent clips, filters out low-quality or static content, annotates each clip with a VLM-generated caption, deduplicates semantically similar clips, and shards the result into webdatasets. Output: ~100M training clips for pre-training, ~10M for fine-tuning.

2. Video Tokenizers (Section 4): Learn to compress raw RGB video frames into compact continuous vectors (for diffusion models) or discrete integer indices (for autoregressive models), with spatial compression factors of 8×, 16× and temporal compression of 4× or 8×. Operate causally—processing frames in temporal order without peeking at the future. Trained with reconstruction, perceptual, optical flow, and adversarial losses. Output: compressed token representations at ~512× total compression.

3. Pre-trained Diffusion WFMs (Section 5.1): Transformer-based denoising models trained to predict future video tokens given past frames and text descriptions. The 7B and 14B variants operate in the continuous token space, using 3D patchification, cross-attention to T5-XXL text embeddings, 3D factorized RoPE, and EDM-based denoising score matching. Trained progressively from 512p→720p resolution with joint image-video batches. Output: generalist Text2World and Video2World models.

4. Pre-trained Autoregressive WFMs (Section 5.2): Decoder-only transformers trained for next-token prediction on discrete video tokens. The 4B and 12B base models predict future frames without text; the 5B and 13B Video2World variants add cross-attention to T5 embeddings for text-conditioned generation. Trained in multiple stages with increasing context length, using 3D RoPE with YaRN extension for long video sequences. Output: complementary autoregressive world models.

5. Prompt Upsampler (Section 5.1.5): A fine-tuned Mistral-NeMo-12B-Instruct model that converts short user prompts into the detailed, descriptive style used during WFM training, bridging the training-inference distribution gap. Uses long-to-short data creation (VLM generates short captions from training long prompts) to produce paired data for fine-tuning.

6. Diffusion Decoder (Section 5.2.5): A fine-tuned Cosmos-Predict1-7B-Text2World model that decodes discrete autoregressive tokens into higher-quality continuous tokens by conditioning on the discrete token embedding and denoising through the continuous space. Compensates for blurriness in the discrete tokenizer's aggressive compression ($8 \times 16 \times 16$).

7. Guardrail System (Section 7): A two-stage safety filter—pre-Guard blocks harmful text inputs using keyword matching and the Aegis content safety LLM; post-Guard classifies generated video frames for unsafe content and blurs detected faces larger than 20×20 pixels.

Information flow: Raw videos → Video Curator → training clips with captions → Tokenizer → compressed token sequences → WFM training → pre-trained generalist model → post-training on domain-specific data (camera poses, robot actions, driving trajectories) → specialized world simulator. At inference, user input (text, image/video, control signal) passes through optional prompt upsampling → tokenization → WFM generation → optional diffusion decoding → guardrail filtering → output video.

3.3 Roadmap for the Deep Dive

• First, the data curation pipeline (Video Curator), since data quality and scale determine the ceiling of all downstream models, and the curation choices (shot detection, filtering criteria, annotation strategy, deduplication) directly shape what physics the WFMs can learn.
• Second, the video tokenizers, because both diffusion and autoregressive WFMs operate in token space, and understanding the compression-quality tradeoffs (continuous vs. discrete, spatial vs. spatio-temporal compression ratios) is prerequisite to understanding model design.
• Third, the diffusion-based WFM architecture and training, since it is the higher-fidelity family and demonstrates the most complex conditioning and parallelism strategies.
• Fourth, the autoregressive-based WFM, explaining its complementary design choices, inference optimizations (speculative decoding with Medusa), and the diffusion decoder that compensates for its discrete tokenizer limitations.
• Fifth, the post-training recipes for camera control, robotic manipulation, and autonomous driving, which concretely demonstrate the pre-training-then-post-training paradigm.
• Sixth, the guardrail system and evaluation methodology, since safe deployment and rigorous measurement are platform-level concerns that span all components.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and platform paper whose core idea is that world foundation models—general-purpose video-prediction models trained at scale on diverse data—can serve as a pre-training foundation that, when post-trained on domain-specific data, produces specialized world simulators for Physical AI applications. The technical contribution spans data engineering, tokenizer design, two distinct model architectures trained at unprecedented scale for world simulation, and empirical demonstrations of transfer across three application domains.

---

Data Curation Pipeline

The data curation pipeline transforms approximately 20M hours of raw video (720p to 4K resolution) into ~100M high-quality training clips suitable for learning physical dynamics. The pipeline operates in five sequential stages, each designed to eliminate noise and redundancy while preserving content that teaches the models about object motion, camera movement, scene geometry, and physical interactions.

Raw dataset composition. The curated dataset targets nine video categories with specific percentages of the total (Section 3.1):

• Driving: 11%
• Hand motion and object manipulation: 16%
• Human motion and activity: 10%
• Spatial awareness and navigation: 16%
• First person point-of-view: 8%
• Nature dynamics: 20%
• Dynamic camera movements: 8%
• Synthetically rendered: 4%
• Others: 7%

These categories are chosen to provide broad coverage of visual objects, actions, and physical phenomena relevant to Physical AI applications. The explicit over-representation of hand manipulation, spatial navigation, and nature dynamics (versus, say, talking heads or static scenery) reflects the paper's goal of teaching models about physics rather than just visual appearance.

Stage 1: Splitting.

The splitting stage takes raw videos of arbitrary length and segments them into individual shots—contiguous sequences without abrupt scene changes—to ensure that the training data captures physically plausible transitions rather than edited cuts. Two sub-steps are involved:

Shot detection. The paper evaluates four algorithms on a custom benchmark called ShotBench, which combines existing datasets (BBC Planet Earth, RAI, ClipShots, SHOT) with diverse shot transition types including complex visual effects. The evaluated methods span a spectrum from classical to learned:

• PySceneDetect: thresholding temporal changes in HSV color histograms. F1 scores range from 0.477 (ClipShots) to 0.889 (BBC).
• Panda70M: augments PySceneDetect with CLIP-embedding-based stitching and filtering. F1 scores range from 0.513 to 0.829.
• TransNetV2: a neural network predicting a transition probability per frame from a 100-frame rolling window. F1: 0.726–0.967.
• AutoShot: another learned detector with a similar architecture. F1: 0.711–0.952.

The paper selects TransNetV2 for two reasons beyond its superior F1 scores. First, it handles challenging shot changes (rapid cuts, dissolves, wipes) better than AutoShot in the authors' empirical testing, despite comparable benchmark scores—indicating that the benchmark may not fully capture real-world difficulty. Second, as an end-to-end neural network, it can leverage GPU acceleration without the engineering complexity of hybrid approaches like Panda70M that combine heuristic detection with embedding-based stitching. Clips shorter than 2 seconds are discarded as likely shot transitions or visual effects; clips longer than 60 seconds are further subdivided to a maximum of 60 seconds.

Transcoding. The raw videos use many different codecs and encoding settings, complicating downstream processing. Each detected shot is re-encoded into a consistent, high-quality MP4 format using the h264_nvenc codec with a high bitrate chosen to avoid perceptible degradation even for fast-motion, high-frequency texture content. The paper performs an extensive throughput optimization study (Table 2), comparing hardware configurations (NVIDIA H100 with 28 CPU cores vs. NVIDIA L40S with 1 CPU core), software settings (ffmpeg with libx264 vs. h264_nvenc, single vs. batch processing), and the use of PyNvideoCodec for direct GPU-accelerated video stream transcoding. The final configuration—L40S GPU with PyNvideoCodec for video streams and ffmpeg for audio remixing—achieves a throughput of 0.3702 videos/second, representing a ~6.5× speedup over the baseline configuration (H100, ffmpeg, libx264, no batching: 0.0574 videos/second). This optimization is critical: processing 20M hours of video at the baseline throughput would be infeasible within the project's 3-month compute budget on 10,000 H100 GPUs.

Stage 2: Filtering.

The filtering stage removes clips that are static, visually degraded, or content-inappropriate for world model training, using four sequential filters applied to the output of the splitting stage.

Motion filtering. The goal is two-fold: remove static videos (which teach nothing about dynamics) and remove videos with random, abrupt camera motion (typically from hand-held cameras, which produce jittery, non-smooth trajectories that world models would learn to replicate). Additionally, the filter tags videos with specific camera motion types (pan, zoom, tilt, etc.) for potential use as conditioning signals.

The core component is a lightweight ViT classifier trained on labeled videos. The input is a sequence of motion vectors or optical flow extracted from each clip. The paper experiments with three motion signal sources: H.264 codec motion vectors, the Farneback optical flow algorithm, and a NVIDIA TensorRT-accelerated optical flow estimation network. The TensorRT-accelerated flow produces the highest classification accuracy and is selected. This choice trades compute for quality: optical flow estimation is more expensive than extracting codec motion vectors but captures true pixel-level motion more accurately for the classifier.

Visual quality filtering. Two quality criteria are assessed. First, distortion filtering removes clips with artifacts, noise, blur, low sharpness, overexposure, or underexposure. The paper uses a video quality assessment model based on DOVER, trained on human-rated videos, which produces a perceptual quality score per clip. Clips in the bottom 15th percentile are removed. Second, appearance quality filtering applies an image aesthetic model on sampled frames from each clip. A conservative threshold of 3.5 is used, since "aesthetics are less important for Physical AI" (Section 3.3.2)—the goal is to filter egregiously bad content, not to restrict training to professionally-shot material, because real-world robot cameras will not produce aesthetically pleasing footage.

Text overlay filtering. Many videos contain post-processed text additions (subtitles, watermarks, creator handles) that co-occur with visual effects and editing. Since the goal is to learn physics from raw visual input, not to reproduce post-processing artifacts, these videos are filtered out. Critically, the paper distinguishes between text overlaid in post-processing (which should be removed) and text naturally present in the scene, such as street signs in driving videos (which should be preserved—they are part of the visual world a Physical AI system encounters).

The filter uses an MLP binary classifier trained on video embeddings extracted by InternVideo2. Training labels come from a proprietary VLM prompted to identify post-processed text. The InternVideo2 embeddings are pre-computed during filtering and reused across multiple stages (semantic deduplication later, and video type classification), amortizing their extraction cost.

Video type filtering. To adjust the training data distribution toward content types relevant to physical world simulation, the paper defines a comprehensive taxonomy of video categories and visual styles. A classifier trained on the same InternVideo2 embeddings labels each clip with taxonomy categories. Specific types are excluded: abstract visual patterns, video game footage, animated content, and other categories that could lead to unrealistic dynamics or poor generation quality. The classifier is trained using annotations from a proprietary VLM prompted with 8 uniformly sampled frames per clip and queried for the most appropriate taxonomy label.

The remaining data is then rebalanced by upsampling categories relevant to WFMs (human action, human-object interaction) and downsampling less critical categories (nature, landscape). This is not a binary filtering step but a distribution-shaping step: the paper recognizes that raw video prevalence does not match training utility, and actively reweights the data to emphasize content where physical dynamics are most informative.

Stage 3: Annotation.

Each filtered clip is paired with a text description to provide supervision and conditioning for world model training. The paper uses a 13B-parameter internal VILA model (ViT-based vision-language model) fine-tuned for video captioning, with an enlarged context window supporting up to 5904 input tokens and 256 output tokens. The VILA is prompted with "Elaborate on the visual and narrative elements of the video in detail" and given 8 uniformly sampled frames from each clip. The average output caption is 559 characters or 97 words.

The choice of VILA over alternatives (VFC, Qwen2-VL) is based on a small-scale human evaluation where VILA produced "more accurate descriptions." Inference throughput is a critical concern: captioning ~100M clips requires massive compute. The paper quantizes the VILA model to FP8 using a TensorRT-LLM engine and measures throughput on a single H100 GPU (Table 3). The FP8 TensorRT-LLM configuration achieves 1.96 clips/second or 470.6 tokens/second at batch size 16—a 10× speedup over the PyTorch FP16 baseline (0.21 clips/second, 49.6 tokens/second). At this throughput, captioning 100M clips takes approximately 1.4M GPU-hours, which is feasible within the 10,000-GPU cluster over three months.

An important design choice: the VLM is configured to focus on "material facts and details" rather than subjective or interpretive descriptions. This aligns the caption distribution with the requirements of world model training—the model needs to learn to map "a robotic arm grasping a red cup on a wooden table" to the corresponding visual dynamics, not to generate poetic interpretations.

Stage 4: Deduplication.

The sheer volume of video data means there are necessarily duplicated or near-duplicated samples—the same event captured from slightly different angles, reposted videos, or clips covering very similar content. Training on duplicates wastes compute, biases the data distribution, and increases the risk of memorization.

The paper adopts SemDeDup-style semantic deduplication. Using the InternVideo2 embeddings already computed during filtering, videos are clustered via a multi-node GPU-accelerated k-means implementation with $k = 10,000$ clusters. Within each cluster, pairwise cosine distances are computed to identify duplicates. When duplicates are detected, the video with the highest resolution is retained to preserve quality. To avoid storing the entire pairwise distance matrix in GPU memory (which would be infeasible given the cluster sizes), the upper-triangular matrix and argmax reduction are computed on-the-fly in blocks of 256. Approximately 30% of the training data is removed during deduplication—a substantial fraction that would have wasted significant training compute.

The embeddings and clustering results are also repurposed to build a visual search engine for the training dataset. This supports free-form text and video queries, enabling debugging of data issues and analysis of the gap between pre-training data and downstream application domains.

Stage 5: Sharding.

The final stage packages processed clips into webdatasets directly consumable by the model trainer. Videos are sharded based on resolution, aspect ratio, and length to align with the progressive training curriculum (low-resolution first, then high-resolution; multiple aspect ratio buckets). A separate set of even higher-quality fine-tuning datasets is created by applying stricter thresholds during filtering.

Infrastructure. The entire pipeline runs on geographically distributed clusters using AnyScale Ray for orchestration. The key architectural decision is a streaming pipeline system that decouples data transfer from computation: pipeline stages consume data from remote storage without requiring local caching of the full dataset. Memory requirements scale with pipeline complexity (the number of concurrent stages and their buffer sizes), not dataset size, enabling unbounded stream processing. The scheduler extends the Fragmentation Gradient Descent algorithm to optimize multi-resource allocation—simultaneously balancing network bandwidth for data ingestion, NVDEC units for video decoding, and GPU compute for filtering and annotation—by automatically scaling individual stages to maintain balanced throughput across specialized hardware accelerators.

---

Video Tokenizer Architecture and Training

The tokenizers are the fundamental compression layer that makes training large WFMs computationally tractable. Operating directly on raw RGB video at 720p resolution and 121 frames would require processing approximately $1280 \times 704 \times 3 \times 121 \approx 327$ million floating-point values per video—far too many for transformer models with quadratic self-attention costs. The tokenizer compresses this into a compact latent representation, reducing dimensionality by a factor of up to $8 \times 16 \times 16 = 2048$ in the most aggressive configuration.

The Cosmos tokenizer is an encoder-decoder architecture that maps input video $x_{0:T} \in \mathbb{R}^{(1+T) \times H \times W \times 3}$ to token video $z_{0:T'} \in \mathbb{R}^{(1+T') \times H' \times W' \times C}$, with spatial compression $s_{HW} = H/H' = W/W'$ and temporal compression $s_T = T/T'$:

$\hat{x}_{0:T} = \mathcal{D}\left(\mathcal{E}(x_{0:T})\right)$

where $\mathcal{E}$ is the encoder, $\mathcal{D}$ is the decoder, $\hat{x}_{0:T}$ is the reconstructed video, $T$ is the number of input frames, $H$ and $W$ are the input spatial dimensions, $C$ is the embedding dimension (16 for continuous, 6 for discrete), and $s_{HW}$, $s_T$ are the compression factors.

What it computes: the encoder maps each spatio-temporal group of input pixels to a compact latent representation; the decoder inverts this mapping, reconstructing the original video from the compressed tokens. The reconstruction quality (PSNR, SSIM) measures how much information is preserved through the bottleneck.

Why this form: the encoder-decoder bottleneck forces the model to learn a compact, semantically meaningful representation—similar to how JPEG compression discards high-frequency details invisible to human perception, but learned from data rather than hand-designed using the discrete cosine transform. The reconstruction loss ensures the representation preserves the visual information needed for downstream world model training. Without such compression, training a 14B-parameter transformer on raw video would require thousands of GPUs just for the attention computation.

Causal temporal design. A defining architectural choice is that the tokenizer is temporally causal: each frame's encoding depends only on current and past frames, never on future frames. This is implemented through:

• Causal temporal convolution layers: 1D convolutions along the time axis using left-padding of $k-1$ (where $k$ is the kernel size), so frame $t$'s output depends only on frames $t, t-1, \ldots, t-k+1$.
• Causal temporal attention layers: self-attention applied along the time axis with a causal mask, preventing frame $t$ from attending to frames $> t$.

The causal design serves two purposes. First, it enables joint image-video training: when $T=0$ (a single image), a causal video tokenizer naturally reduces to an image tokenizer, because there are no future frames to condition on. This allows the tokenizer and downstream WFMs to leverage the vast abundance of high-quality, diverse image datasets alongside video data, which is critical because image datasets are orders of magnitude larger and more diverse than video datasets for many visual concepts. Second, it aligns with Physical AI systems that operate in real-time causal environments—a robot's current camera frame cannot depend on future frames.

Wavelet-space processing. Rather than operating directly on raw pixel values, the tokenizer first applies a 2-level 3D Haar wavelet transform that downsamples the input by a factor of 4 along all three dimensions (height, width, time). The transform groups frames as $\{x_0, x_{1:4}, x_{5:8}, \ldots, x_{(T-3):T}\} \rightarrow \{g_0, g_1, g_2, \ldots\}$ and processes these groups causally as $\{g_0, g_{0:1}, g_{0:2}, \ldots\} \rightarrow \{\xi_0, \xi_1, \xi_2, \ldots\}$. The wavelet transform eliminates pixel-level redundancies (neighboring pixels are highly correlated) before the learned compression begins, allowing the encoder and decoder to focus their capacity on more semantic compression rather than decorrelating raw pixels. This is analogous to how JPEG compression first applies a DCT to decorrelate spatial frequencies before quantizing.

Encoder-decoder blocks. After the wavelet transform, the encoder consists of a series of residual blocks interleaved with downsampling blocks. Each residual block uses spatio-temporal factorized 3D convolutions:

• A 2D spatial convolution with kernel size $1 \times k \times k$ captures spatial patterns within each frame independently.
• A 1D temporal convolution with kernel size $k \times 1 \times 1$ captures frame-to-frame dynamics.

This factorization reduces the parameter count from $k^3$ to $2k^2$ (approximately one-third for $k=3$) while still capturing 3D structure, because spatial and temporal correlations can largely be decomposed. For long-range dependencies, spatio-temporal factorized causal self-attention operates with global support along the temporal dimension—by the final encoder block, the attention spans $1+T'$ tokens along time, enabling each token to attend to all preceding tokens.

Normalization and activation. The paper uses Layer Normalization rather than the more common Group Normalization for video models. The justification is empirical: Karras et al. (2020) and Sadat et al. (2024) observed that GroupNorm can produce large magnitude values in specific regions of the latent space or reconstructed outputs. LayerNorm, operating across the full channel dimension at each spatio-temporal position, prevents this. The Swish activation function (a smooth, non-monotonic variant of ReLU: $x \cdot \sigma(x)$ for sigmoid $\sigma$) is used for non-linearity.

Continuous tokenizers. For the continuous variant used by diffusion WFMs, the tokenizer uses a vanilla autoencoder (AE) formulation: no variational prior, no KL divergence loss, no commitment loss. The latent dimension $C = 16$, meaning each token is a 16-dimensional continuous vector. This is in contrast to the VAE formulation used by Stable Diffusion and many other latent diffusion models. The ablation is implicit: the paper found that the vanilla AE produces sufficient regularization through the bottleneck alone, and adding a KL prior loss was unnecessary for reconstruction quality.

Discrete tokenizers. For the autoregressive variant, the tokenizer uses Finite-Scalar-Quantization to discretize the continuous latent vectors into integer indices. FSQ works by quantizing each of the 6 latent dimensions independently into a small number of bins: the levels are $(8, 8, 8, 5, 5, 5)$, producing a total vocabulary size of $8 \times 8 \times 8 \times 5 \times 5 \times 5 = 64,000$ possible discrete tokens. This is the vocabulary that the autoregressive transformer predicts at each position.

The choice of FSQ over VQ-VAE (the more common vector quantization approach) eliminates the commitment loss and codebook collapse issues that plague VQ-VAE training. FSQ bounds each dimension to a small integer range by applying a bounded rounding operation, which is simpler and more stable than learning a codebook of continuous vectors via nearest-neighbor lookup. The specific levels $(8, 8, 8, 5, 5, 5)$ mean that the first three latent dimensions are quantized into 8 bins each (3 bits per dimension), while the last three are quantized into 5 bins each (~2.32 bits per dimension). This asymmetry suggests that the first three dimensions capture higher-variance information that requires finer quantization.

Training objective. The tokenizer is trained in two stages, both using only the final decoder output loss (no auxiliary losses on intermediate latent representations—the commitment and KL losses that would be required for VQ-VAE or VAE formulations are explicitly omitted). The first stage minimizes:

$\mathcal{L}_1 = \|\hat{x}_{0:T} - x_{0:T}\|_1$

and the perceptual loss based on pre-trained VGG-19 features:

$\mathcal{L}_{\text{Perceptual}} = \frac{1}{L} \sum_{l=1}^{L} \sum_{t} \alpha_l \|\text{VGG}_l(\hat{x}_t) - \text{VGG}_l(x_t)\|_1$

where $\text{VGG}_l(\cdot) \in \mathbb{R}^{H \times W \times C}$ is the feature map from the $l$-th layer of a frozen VGG-19 network, $L$ is the number of layers (typically early-to-mid layers that capture texture and structure), and $\alpha_l$ is a per-layer weight.

What it computes: the L1 loss penalizes pixel-level color differences between the input and reconstruction; the perceptual loss penalizes differences in deep feature space, which correlates better with human-perceived similarity than pixel-space metrics (two images with a one-pixel shift have catastrophic L2 loss but nearly identical VGG features).

Why this form: training with only L1 loss produces blurry reconstructions because the model averages over possible fine details (the L1 minimizer for a multimodal distribution is the median, which discards high-frequency texture). The perceptual loss provides a learned similarity metric that encourages the decoder to produce sharp, high-frequency details matching the input's texture statistics, without requiring the adversarial training needed for GAN-based losses.

The second stage fine-tunes with three additional losses:

$\mathcal{L}_{\text{Flow}} = \frac{1}{T} \sum_{t=1}^{T} \|\text{OF}(\hat{x}_t, \hat{x}_{t-1}) - \text{OF}(x_t, x_{t-1})\|_1 + \frac{1}{T} \sum_{t=0}^{T-1} \|\text{OF}(\hat{x}_t, \hat{x}_{t+1}) - \text{OF}(x_t, x_{t+1})\|_1$

where $\text{OF}(\cdot, \cdot)$ computes optical flow between consecutive frames using RAFT (a state-of-the-art flow estimation network). This loss ensures the reconstructed video has temporally consistent motion—that object velocities and trajectories in the reconstruction match those in the original.

$\mathcal{L}_{\text{Gram}} = \frac{1}{L} \sum_{l=1}^{L} \sum_{t} \alpha_l \|\text{GM}_l(\hat{x}_t) - \text{GM}_l(x_t)\|_1$

where $\text{GM}_l(\cdot)$ computes the Gram matrix of VGG features at layer $l$, capturing texture statistics (the correlation between feature channels) independent of spatial arrangement. This enhances sharpness by matching the texture "style" of the reconstruction to the original.

Finally, adversarial loss is added in the fine-tuning stage "to further enhance reconstruction details, particularly at large compression rates" (Section 4.2). The exact adversarial formulation is not specified, but typically involves a discriminator network trained to distinguish real from reconstructed frames, with the tokenizer decoder acting as a generator trying to fool the discriminator.

Training configurations. The tokenizer suite includes multiple variants at different compression rates, trained with different mini-batch configurations:

Continuous image tokenizers (CI):

• CI 8×8: spatial-only compression of 64× (used as baselines for latent diffusion models)
• CI 16×16: spatial-only compression of 256×

Continuous video tokenizers (CV):

• CV 4×8×8: total compression $4 \times 64 = 256\times$
• CV 8×8×8: total compression $8 \times 64 = 512\times$ (used for the main diffusion WFMs)
• CV 8×16×16: total compression $8 \times 256 = 2048\times$

Discrete image tokenizers (DI): DI 8×8, DI 16×16

Discrete video tokenizers (DV): DV 4×8×8, DV 8×8×8, DV 8×16×16 (the latter used for autoregressive WFMs)

Additionally, the tokenizers are split into two generations based on the training video length:

• Cosmos-0.1-Tokenizer: trained with mini-batches of 49 frames for CV, 17 frames for DV
• Cosmos-Tokenize1: trained with 121 frames for CV, 49 frames for DV—the longer sequences enable better temporal coherence at the cost of more GPU memory per batch

The Cosmos-Tokenize1 variants are the production tokenizers used by the WFMs.

Joint image-video training. Training alternates mini-batches of images and videos at a preset frequency. This joint training is enabled by the causal architecture: when the input is a single image ($T=0$), a causal video tokenizer with temporal compression $s_T$ still processes it correctly because there is no temporal dimension to compress. This allows the tokenizer to leverage image datasets (MS-COCO, ImageNet, and internal high-quality image collections) that contain rich appearance information and are orders of magnitude more diverse than video datasets for many object categories.

The encoder and decoder operate on specific input resolutions but generalize to higher resolutions during inference. The paper reports that the tokenizers "generalize well to resolutions they were trained on and maintain strong quality at higher resolutions" (Section 4.2)—a property enabled by the fully convolutional architecture (no position-dependent parameters other than the learned attention, which uses relative positions via RoPE-style mechanisms downstream).

---

Diffusion-Based World Foundation Model

The diffusion-based WFMs are transformer models trained to denoise videos in a learned continuous latent space, enabling controllable generation of future world states. The architecture adapts the DiT design from image generation to 3D video data, incorporating several innovations for efficiency, stability, and conditioning flexibility.

Formulation. The training follows the EDM (Elucidating the Design Space of Diffusion-Based Generative Models) framework. The denoising score matching loss for a denoiser $D_\theta$ at noise level $\sigma$ is:

$\mathcal{L}(D_\theta, \sigma) = \mathbb{E}_{\mathbf{x}_0, \mathbf{n}} \left[ \|D_\theta(\mathbf{x}_0 + \mathbf{n}; \sigma) - \mathbf{x}_0\|_2^2 \right]$

where $\mathbf{x}_0 \sim p_{\text{data}}$ is a clean latent representation of a video (produced by the continuous tokenizer), $\mathbf{n} \sim \mathcal{N}(0, \sigma^2 I)$ is isotropic Gaussian noise, and $D_\theta$ is a noise-conditioned neural network (the denoiser) that predicts the clean latent $\mathbf{x}_0$ from the noised version $\mathbf{x}_0 + \mathbf{n}$.

What it computes: at each training step, the model takes a latent representation, adds random Gaussian noise scaled by $\sigma$, and learns to predict the original clean latent. This is equivalent to learning the score function (gradient of the log probability density) scaled by $\sigma$, which tells the model how to move noisy latents toward higher-density regions of the training distribution.

Why this form: predicting the clean latent directly (rather than the noise as in DDPM) is simpler when the data has complex structure—the clean latent is a fixed target, while predicting the noise requires the model to learn that the target depends on $\sigma$ in a specific way. The denoising formulation also enables iterative refinement at inference: starting from pure noise, the model repeatedly predicts and removes a portion of the noise, gradually revealing a coherent video structure.

The overall training loss is a weighted expectation over noise levels:

$\mathcal{L}(D_\theta) = \mathbb{E}_\sigma \left[ \frac{\lambda(\sigma)}{e^{u(\sigma)}} \mathcal{L}(D_\theta, \sigma) + u(\sigma) \right]$

$\lambda(\sigma) = (\sigma^2 + \sigma_{\text{data}}^2) / (\sigma \cdot \sigma_{\text{data}})^2$

$\ln(\sigma) \sim \mathcal{N}(P_{\text{mean}}, P_{\text{std}}^2)$

where $\sigma$ is the noise level sampled from a log-normal distribution with hyperparameters $P_{\text{mean}}$ and $P_{\text{std}}$, $\sigma_{\text{data}}$ is the standard deviation of the training data (estimated from a batch), $\lambda(\sigma)$ is a weighting function that equalizes the contribution of each noise level at the start of training, and $u(\sigma)$ is a continuous uncertainty function (parameterized by a small MLP) that adaptively reweights noise levels during training.

What $\lambda(\sigma)$ does: at small $\sigma$ (low noise, easy denoising), the score matching loss is naturally smaller; at large $\sigma$ (high noise, harder), the loss is larger. The $\lambda(\sigma)$ factor compensates for this, ensuring the model attends equally to all noise levels initially rather than focusing only on the high-noise regime. The specific form $(\sigma^2 + \sigma_{\text{data}}^2) / (\sigma \cdot \sigma_{\text{data}})^2$ is derived from the relationship between score matching and denoising objectives.

What $u(\sigma)$ does: as training progresses, the balance between noise levels may degrade—some levels become easier than others, and the model would benefit from focusing on the harder levels. The uncertainty weighting treats the optimization over noise levels as multi-task learning, where each $\sigma$ is a separate task. High uncertainty $u(\sigma)$ causes the loss at $\sigma$ to be downweighted (the denominator $e^{u(\sigma)}$ grows), but the $+u(\sigma)$ term penalizes the model for being uncertain, encouraging it to eventually reduce uncertainty. This is an adaptive curriculum over noise levels.

Why this form: the EDM formulation is equivalent to flow matching (used by HunyuanVideo, MovieGen) and DDPM under appropriate reparameterization, but the preconditioning and noise scheduling differ. The paper states that in practice, "we have not encountered any performance limitations with the EDM formulation" (Section 5.1.1), implying the choice between these equivalent frameworks is irrelevant at this scale—what matters is the engineering around it (model architecture, data, parallelism).

Architecture. The denoiser $D_\theta$ is a DiT-style transformer adapted for 3D video data.

3D patchification. The continuous latent representation from the tokenizer has shape $T' \times C \times H' \times W'$, where $C=16$. To prepare inputs for the transformer, non-overlapping 3D cubes of shape $(p_t, p_h, p_w)$ are projected to individual token embeddings via a linear layer. The paper uses $p_t = 1, p_h = p_w = 2$, meaning:

• Each token covers 1 frame in time (no temporal patchification), 2×2 spatial blocks
• This produces a sequence length of $T' H' W' / (1 \cdot 2 \cdot 2)$
• For the high-resolution training configuration (720p, 121 frames): width = 1280, height = 704, compressed by 8× in the tokenizer → $H' = 88, W' = 160$, temporal compression 8× → $T' = (121-1)/8 + 1 = 16$. Patchification yields $16 \times 88/2 \times 160/2 = 56,320$ tokens (Table 12).

The choice of $p_t = 1$ (no temporal patchification) means that each frame's latent is processed at full temporal resolution in the transformer, while spatial information is aggregated in 2×2 blocks. This preserves fine temporal dynamics at the cost of longer sequences.

3D positional embeddings. The model uses a hybrid approach combining relative and absolute positional information.

3D-factorized Rotary Position Embedding (RoPE) encodes relative positions along the temporal, height, and width dimensions independently. The feature dimension is partitioned into three approximately equal chunks, each applying RoPE with positional information along one axis. For temporal RoPE, the frequencies are rescaled based on the training video's Frames Per Second (FPS), allowing the model to distinguish videos with different frame rates—a car moving at 30 FPS has twice the per-frame displacement as the same motion at 60 FPS, and the FPS-aware RoPE compensates for this.

Why factorized 3D RoPE over a single 3D RoPE: factorization allows each axis to be processed independently, which (a) enables efficient implementation by concatenating frequency embeddings and reusing LLM-optimized RoPE kernels, (b) supports generation at arbitrary aspect ratios and frame lengths because each axis's positions are encoded separately, and (c) enables NTK-RoPE extension during progressive training (when resolution or video length increases, the model can rapidly adapt by interpolating the RoPE frequencies).

Learnable absolute positional embeddings are added per transformer block as a complement to the relative RoPE encoding. The paper finds this combination "reduces training loss and reduces morphing artifacts in generated videos" (Section 5.1.2). The learnable embeddings capture absolute coordinate information (e.g., "this token is in the upper-left corner of frame 5") that RoPE's relative encoding cannot directly represent.

Cross-attention for text conditioning. Each transformer block consists of three sequential operations: self-attention (over the spatiotemporal token sequence), cross-attention (to text embeddings), and a feedforward MLP. The cross-attention uses T5-XXL embeddings as keys and values, with the video tokens as queries. The T5 embeddings are zero-padded to a fixed sequence length of 512.

The text conditioning strategy differs from prior work in two ways. First, the paper does not randomly zero out text embeddings during training (unlike the standard classifier-free guidance recipe that drops text conditioning with some probability). Instead, they rely on the model's ability to generate high-quality content even without guidance, which they attribute to high-quality training data: "careful data selection achieves a similar effect" (Section 5.1.3) to the mode-seeking behavior that classifier-free guidance promotes. However, for video generation—where high-quality training data is scarcer compared to images—higher guidance values are needed during inference. Second, they use negative prompts during inference rather than fixed null-text conditioning, which gives users more control over what to exclude.

Query-key normalization. During early training, the authors observed "instability in the growth of attention logits, leading to a collapse of attention entropy" (Section 5.1.2)—the attention distribution becomes a one-hot vector focusing on a single token, preventing the model from learning useful feature combinations. To address this, they apply RMSNorm with learnable scales to both the query $Q$ and key $K$ before the dot-product attention computation, for all self-attention and cross-attention layers. The normalization constrains the logit magnitudes, preventing the softmax from saturating.

AdaLN-LoRA. Adaptive Layer Normalization (AdaLN) layers in standard DiT architectures condition the network on the noise level $\sigma$ by producing scale, shift, and gate parameters for the layer normalization operations. However, these dense linear projections account for a significant portion of the model parameters while contributing negligibly to FLOPs, since they are applied once per block rather than per token. The paper decomposes these projections using Low-Rank Adaptation (LoRA): each weight matrix $W \in \mathbb{R}^{d_{\text{model}} \times 3d_{\text{model}}}$ is replaced with $W = BA$ where $B \in \mathbb{R}^{d_{\text{model}} \times r}$ and $A \in \mathbb{R}^{r \times 3d_{\text{model}}}$ with rank $r = 256$.

For the Cosmos-Predict1-7B model, this reduces parameters from 11B to 7B (a 36% reduction) without measurable performance degradation. The intuition is that AdaLN projections map a 1D noise embedding to high-dimensional modulation parameters; this mapping is inherently low-rank because the noise embedding is low-dimensional, so full-rank dense matrices are wasteful.

Training strategy. The diffusion WFMs are trained in three progressive stages:

Stage 1: Low-resolution pre-training. Videos and images at 512p resolution (640×512), 57 frames, context length 10,240 tokens. This stage builds basic visual and motion understanding at lower computational cost.

Stage 2: High-resolution pre-training. Scaled to the target 720p resolution (1280×704), 121 frames, context length 56,320 tokens. This is the main training stage that consumes the majority of the compute budget.

Stage 3: High-quality fine-tuning. A brief fine-tuning phase ($\mathcal{O}(10^4)$ iterations) on a curated high-quality subset, with a linearly decaying learning rate. This improves perceptual quality without substantially changing the model's learned distribution.

Why progressive training matters: training directly at the final resolution from initialization would be computationally wasteful because early training iterations mostly learn low-level features (edges, textures, basic motion) that are resolution-agnostic. Training at lower resolution first reduces per-iteration cost by roughly $2\times$ in both FLOPs and memory, enabling more iterations within the same compute budget. When switching to higher resolution, NTK-RoPE interpolation enables the model to adapt rapidly—the paper reports achieving "reasonable performance even within 5,000 training steps" after resolution changes.

Joint image-video training. The training alternates between image and video batches. To address the distribution shift between image and video latent representations, domain-specific normalization is applied: image latents and video latents are standardized using independent running estimates of mean and variance. This is motivated by the observation that reducing distributional shift between modalities improves generation quality.

Additionally, video frames contain temporal redundancy (consecutive frames are highly correlated), which results in smaller gradient magnitudes for video batches compared to image batches. The paper scales the noise levels for video batches by the square root of the frame count relative to image batches. For example, if images use noise level $\sigma$ and videos have 57 frames, the video noise level is $\sigma \times \sqrt{57}$. This compensates for the redundancy by effectively increasing the denoising difficulty for videos, balancing the convergence rates.

Multi-aspect training. Data is organized into five aspect ratio buckets: 1:1, 3:4, 4:3, 9:16, and 16:9. During training, each data-parallel process group samples from one bucket, allowing different GPUs to process different aspect ratios simultaneously. The paper uses longest-side resizing—each image or video is resized so its longest side matches the target resolution (e.g., 704 for height or 1280 for width), with the shorter side filled to the aspect ratio using reflection padding. A padding mask is supplied to the diffusion backbone, enabling the model to ignore padded regions during attention.

This approach contrasts with center-cropping used by many prior works: center-cropping guarantees consistent tensor shapes but can discard important content at the edges (e.g., a person's head cropped off in a portrait video). Reflection padding preserves all original content at the cost of some compute waste on padded regions.

Mixed-precision and stability. Training maintains two weight copies: BF16 for forward/backward passes (reducing memory by 2× vs. FP32), and FP32 for parameter updates (maintaining numerical precision for the optimizer). To stabilize training, the denoising loss is scaled by 10×, and the AdamW optimizer uses "lower betas and eps coefficients" (Section 5.1.3) than standard settings—the exact modified values are $\beta_1, \beta_2 = 0.9, 0.99$ and $\epsilon = 10^{-10}$. This choice significantly reduces loss spikes; for the 14B model, "we rarely encountered loss spikes, and there were no non-recoverable loss spikes."

The base learning rates are $2^{-15}$ for the 7B model and $2^{-16}$ for the 14B model, with a linear warmup over 2,500 iterations. Weight decay is 0.1 for 7B and 0.2 for 14B.

Text conditioning at scale. The T5-XXL text encoder processes prompts into embeddings that are zero-padded to a fixed length of 512 tokens. During inference, classifier-free guidance is used: the model generates with and without text conditioning, and the final prediction is $D_{\text{cfg}} = D_{\text{uncond}} + w \cdot (D_{\text{cond}} - D_{\text{uncond}})$, where $w > 1$ amplifies the text influence.

For Text2World models, the conditioning is purely text→video. For Video2World models (fine-tuned from Text2World), additional video conditioning is incorporated: the conditioning frames are concatenated with the generated frames along the temporal dimension in the latent space. Augmented noise is added to the conditioning frames during training (with $\sigma$ sampled from a log-normal distribution with $P_{\text{mean}} = -3.0, P_{\text{std}} = 2.0$) to make the model robust to variations in input quality during inference. A binary mask distinguishes conditioning frames (no loss) from generated frames (loss applied), and the number of conditioning frames is randomly varied during training so the model can handle either a single image or multiple video frames as input.

Scaling up with parallelism. The 14B model trained at 720p with 121-frame videos requires approximately 280 GB for parameters, gradients, and optimizer states, plus approximately 310 GB for activations—far exceeding the 80 GB HBM3 capacity of a single H100 GPU. Two parallelism strategies are combined:

Fully Sharded Data Parallelism (FSDP). Parameters, gradients, and optimizer states are sharded across 64 GPUs (for the 14B model; 32 for 7B). Each GPU stores only its shard during non-compute periods and gathers the full parameters only when needed for its assigned computation. This reduces per-GPU memory for static state from 280 GB to roughly 4.4 GB.

Context Parallelism (CP). The 56,320-token sequence is split across 8 GPUs within NVLink-connected groups. Each GPU processes its chunk of the query $Q$ and iteratively accumulates partial attention outputs using blocks of key-value pairs $(K, V)$ transferred from other GPUs in the CP group. The paper uses the P2P variant from TransformerEngine that overlaps communication and computation: while one $(K, V)$ block is being transferred, the GPU processes another block. With block sizes chosen appropriately, the data transfer latency is entirely hidden, and the attention computation proceeds at near-peak FLOPs.

The combination reduces activation memory from 310 GB to approximately 40 GB per GPU. Cross-attention layers (where the key-value sequence length is much shorter—512 tokens versus 56,320) do not use CP because the computation is too small to mask communication overhead.

Prompt upsampler. A significant distribution gap exists between training prompts (detailed, descriptive VLM-generated captions averaging 97 words) and inference prompts (short, user-written descriptions like "a car driving down a street"). Training a VLM to generate short captions from training long prompts and corresponding videos produces paired data (short user-style, long training-style). This "long-to-short" data creation strategy is effective because it preserves the authentic video content and distribution from the detailed training prompts while ensuring fidelity between the short and long prompts—the pair describes the same video.

The upsampler is a fine-tuned Mistral-NeMo-12B-Instruct model, producing enriched prompts that add detail while maintaining the original key elements, intent, and structure. For the Video2World model, the upsampler uses Pixtral-12B (an open-source VLM) with zero-shot prompt engineering to incorporate both the video conditioning and the user's text into the enriched description. The paper notes Pixtral worked "well out of the box" without fine-tuning.

---

Autoregressive-Based World Foundation Model

The autoregressive WFMs approach world simulation as next-token prediction, directly analogous to language modeling. A video is first encoded into a sequence of discrete tokens using the Cosmos discrete tokenizer. Then, a decoder-only transformer is trained to predict each successive token given all previous tokens, minimizing the negative log-likelihood:

$\mathcal{L}_{\text{NLL}} = \sum_i -\log P(v_i | v_1, v_2, \ldots, v_{i-1}; \Theta)$

where $v_i$ is the $i$-th discrete video token (an integer in $\{0, \ldots, 63999\}$), $P$ is the conditional probability modeled by the transformer with parameters $\Theta$, and the sum runs over all tokens in the sequence.

What it computes: for each position in the token sequence, the model outputs a probability distribution over the 64,000 possible discrete tokens, and is trained to maximize the probability assigned to the ground-truth next token. At inference, tokens are sampled autoregressively from this distribution and decoded back to video frames by the discrete tokenizer's decoder.

Why this form: autoregressive modeling decomposes the intractable joint probability of a video into a product of tractable conditional probabilities, converting a high-dimensional generation problem into a sequence of 64,000-way classification problems. This enables the use of the same architectures, optimization techniques, and scaling laws developed for large language models, potentially allowing autoregressive WFMs to inherit the rapid progress in LLM training and inference.

Architecture modifications for video.

3D positional embeddings. As in the diffusion WFM, the autoregressive model uses a hybrid of relative and absolute positional encodings:

3D factorized RoPE encodes relative positions along temporal, height, and width axes identically to the diffusion model. However, during the multi-stage training where video sequence length increases (from 17 frames to 34 frames), the paper applies YaRN (Yet another RoPE extensioN) scaling specifically along the temporal dimension. YaRN extends the context window of RoPE by interpolating the temporal frequencies, enabling the model to extrapolate to longer sequences than it was trained on without catastrophic degradation. This is analogous to how LLMs extend their context windows from 4K to 32K tokens using NTK-aware scaling.

3D absolute positional embeddings use fixed sinusoidal embeddings (not learnable, unlike the diffusion model) factorized across the three dimensions and added at each transformer block. The paper chooses sinusoidal over learnable for the autoregressive model; they do not elaborate on the reason, but sinusoidal embeddings are known to extrapolate better to unseen positions, which is critical for progressively increasing sequence length.

Vocabulary. The discrete tokenizer produces one integer per latent position from a vocabulary of 64,000 possible tokens (determined by the FSQ levels: $8 \times 8 \times 8 \times 5 \times 5 \times 5$). This is analogous to the sub-word vocabulary of an LLM tokenizer (e.g., GPT-4 uses ~100K tokens), but it represents visual content rather than text. At each autoregressive step, the model predicts one of these 64,000 discrete tokens.

Cross-attention for text conditioning. The base 4B and 12B models are pure video predictors without text input. Text conditioning is added post-hoc by introducing cross-attention layers after every self-attention layer, with keys and values from T5-XXL embeddings. The cross-attention weights are randomly initialized and trained during Stage 2 of the multi-stage curriculum.

Query-key normalization (QKNorm). As in the diffusion model, the query $Q$ and key $K$ are normalized before computing attention logits to prevent saturation of the softmax. Unlike the diffusion model's RMSNorm, the autoregressive model uses a learnable scaling factor $\gamma$ instead of the fixed $1/\sqrt{d_k}$, giving the model more control over attention score magnitudes.

Z-loss. To further stabilize training, a regularization term is added:

$\mathcal{L}_{\text{z-loss}} = \lambda \cdot \sum_i z_i^2$

where $z_i$ is the logit value for token $i$ (the model's output before softmax), and $\lambda = 3 \times 10^{-4}$. The z-loss penalizes large logit magnitudes, which can cause numerical instability or gradient explosions.

What it computes: the squared sum of all raw logits produced by the model. Large logits produce extreme softmax distributions (near one-hot), which have vanishing gradients and can cause loss spikes.

Why this form: the z-loss is shift-invariant—adding a constant to all logits does not change the softmax distribution but does increase the squared sum, so the model learns to center its logits around zero. This prevents logit drift toward extreme positive or negative values, maintaining gradient norms in a healthy range. The authors found this "critical in maintaining gradient norms" when scaling to many GPU nodes.

The full training objective adds this to the negative log-likelihood:

$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \mathcal{L}_{\text{z-loss}}$

Training strategy. The autoregressive WFMs are trained in a multi-stage curriculum:

Stage 1 (pure video prediction): Given the first frame, predict the next 16 frames (context length 17 frames). This teaches the model basic motion dynamics without text conditioning.

Stage 1.1 (extended context): Increase context to 34 frames using YaRN temporal RoPE extension. The model learns longer-term dependencies.

Stage 2 (text conditioning): Add cross-attention layers (randomly initialized) and train with text-conditioned video prediction on 34-frame contexts. Joint image-video training is used, with larger batch sizes for images (since image context length is much smaller—a single frame vs. 34).

All training uses a fixed spatial resolution of 640×1024.

Cooling down. After pre-training, a final phase with high-quality data linearly decays the learning rate to zero over 30,000 iterations. This is analogous to the annealing phase in LLM training that improves final performance with minimal additional compute.

Model configurations. Four autoregressive variants are released:

• Cosmos-Predict1-4B: 16 layers, model dimension 4,096, 32 attention heads, 8 key-value heads, FFN dimension 14,336. Trained through stages 1 and 1.1. No cross-attention.
• Cosmos-Predict1-5B-Video2World: Derived from 4B by adding cross-attention layers and training with Stage 2. The extra cross-attention parameters bring the total to ~5B.
• Cosmos-Predict1-12B: 40 layers, model dimension 5,120, 32 attention heads, 8 key-value heads. Trained through stages 1 and 1.1.
• Cosmos-Predict1-13B-Video2World: Derived from 12B, adding cross-attention for text conditioning.

The base learning rates start at $1 \times 10^{-3}$ for the 4B and 12B models, and $3 \times 10^{-4}$ and $5 \times 10^{-4}$ for the 5B and 13B models during Stage 2 (lower rates for fine-tuning with new cross-attention layers). A linear warmup over 5,000 iterations is used. The activation function is SwiGLU, weight decay is 0.01, and the RoPE base frequency $\theta = 500,000$.

Inference optimization with Medusa speculative decoding. Autoregressive generation of video tokens is sequential—each token must be predicted before the next can be processed. This creates an inference bottleneck: generating $N$ tokens requires $N$ serial forward passes through the transformer. For real-time Physical AI applications (e.g., robot planning at 10 FPS), this sequential decoding can be prohibitively slow.

The paper adapts the Medusa speculative decoding framework to accelerate video generation. Instead of using a separate draft model, Medusa fine-tunes additional "heads" onto the transformer backbone. Each Medusa head is a single-layer FFN (with SiLU activation and residual connection) that predicts one future token, enabling the model to propose multiple subsequent tokens in parallel. These speculated tokens are then verified with a single forward pass of the full model using rejection sampling—if the speculation was correct, multiple tokens are generated in one step; if incorrect, only the correctly predicted prefix is accepted.

The paper's innovation is in how the Medusa heads are trained for video generation:

• Selective fine-tuning: Only the last two transformer layers, the final unembedding layer, and the Medusa heads themselves are fine-tuned; the rest of the backbone remains frozen. Full fine-tuning caused quality degradation; freezing too much produced poor multi-token prediction—the chosen strategy balances both.
• Unified FFN for parallel prediction: The weight matrices of multiple Medusa heads (each a single-layer FFN) are merged into one larger FFN, enabling all heads to predict their respective tokens in a single matrix multiplication.
• Optimal head count: Ablation over 3, 6, 9, and 12 heads (Table 15) shows that 9 heads provides the best tradeoff—more heads further reduce the number of forward passes but slow down token throughput due to the overhead of processing the additional predictions.

On the 5B model, 9 Medusa heads achieve 982.77 tokens/second (3.2× speedup over no Medusa) and require only 1,799 forward passes (5.7× reduction, since 9+1 tokens are potentially generated per iteration). The speedup on the 4B model is slightly lower (894.67 tokens/second, 2.0× speedup) because the base model is already smaller and faster, so the Medusa overhead is relatively larger.

Real-time generation. To achieve real-time performance, the model is adapted to lower spatial resolution (320×512, from the original 640×1024) in three steps: (1) fine-tune the discrete video tokenizer on 320p videos from the target Physical AI domain, (2) fine-tune the autoregressive WFM with this low-resolution tokenizer, (3) add Medusa heads. At 10 FPS with 320×512 input, the system generates 10 frames in less than 1 second on 8×H100 GPUs (Table 17: 806.61 tokens/second).

Diffusion decoder. The discrete tokenizer's aggressive compression ($8 \times 16 \times 16$, or 2048× total) causes blurriness and visible artifacts in generated videos. The diffusion decoder addresses this by learning a higher-capacity mapping from discrete to continuous latent space, essentially "upsampling" the compressed representation before final RGB decoding.

The decoder is built by fine-tuning Cosmos-Predict1-7B-Text2World. During training, each video is tokenized twice:

1. Through Cosmos-Tokenize1-DV8×16×16-720p → discrete token video (conditioning input).
2. Through Cosmos-Tokenize1-CV8×8×8-720p → continuous token video (higher-quality target).

The discrete tokens are embedded into 16-dimensional vectors via a learnable vocabulary embedding, then upsampled 2× along the spatial dimensions to match the continuous token resolution. These are concatenated with the noised continuous latent along the channel dimension, and the 7B denoiser learns to recover the clean continuous latent, leveraging the coarse information from the discrete tokens as guidance.

At inference, the autoregressive WFM produces discrete tokens → diffusion decoder generates higher-quality continuous tokens → CV8×8×8 decoder produces the final RGB video. This two-step decoding adds inference cost but substantially improves visual quality (Figure 18 shows the autoregressive output is blurry; the diffusion decoder output is sharp while preserving the same content and motion).

---

Post-Training for Downstream Applications

The pre-trained WFMs are generalists; post-training specializes them for specific Physical AI setups by fine-tuning on small domain-specific datasets. The paper demonstrates three application domains.

Camera control (Section 6.1). The goal is to generate temporally coherent, 3D-consistent video from a single input image and a specified camera trajectory.

Dataset: DL3DV-10K, a large-scale static-scene video dataset. Videos are chunked into 256-frame clips, and structure-from-motion (GLOMAP) computes relative camera poses for all frames. The first frame's camera pose is set to identity, and all subsequent poses are relative. A VLM captions each video describing it as a static scene.

Fine-tuning: Camera control is integrated by concatenating Plücker embeddings with the latent features. Plücker coordinates represent each ray from the camera through a pixel as:

$\mathbf{r} = (\mathbf{d}, \mathbf{m}) \in \mathbb{R}^6, \quad \text{where } \mathbf{m} = \mathbf{c} \times \mathbf{d}$

where $\mathbf{c}$ is the camera center location and $\mathbf{d}$ is the unit ray direction for each latent pixel (treating the latent as a downsampled image). The Plücker coordinates are concatenated with the corresponding latent features along the channel dimension.

What it computes: for each pixel in each latent frame, the Plücker embedding encodes the viewing ray's direction and its moment relative to the camera center. Two rays that intersect the same 3D point from different viewpoints will have consistent Plücker coordinates, providing a geometric signal that the model can use to maintain 3D consistency across camera motions.

Why this form: Plücker coordinates are a standard representation for lines in 3D, parameterizing both direction and position in a 6D vector that linearizes the ray geometry. Alternative representations—providing the camera pose as a flat vector or as a 4×4 transformation matrix—would require the model to learn the relationship between pose changes and pixel displacements from scratch, whereas Plücker coordinates make this relationship locally linear.

Since the tokenizer has a temporal compression rate of 8× (every 8 input frames produce 1 latent frame), the Plücker embedding at the 4th frame of each group is used for the corresponding latent. The model generates 57 frames at a time and can be run autoregressively by conditioning on the last generated frame.

Evaluation: Compared against CamCo, a state-of-the-art camera-controllable video generation model, both fine-tuned on DL3DV-10K and evaluated on RealEstate10K (a distribution-shifted test set). The Cosmos post-trained model achieves:

• 82.0% camera pose estimation success rate (vs. 43.0% for CamCo)
• Rotation error: 1.646° (vs. 8.277°)
• Translation error: 0.038 (vs. 0.185)
• FID: 14.30 (vs. 57.49), FVD: 120.49 (vs. 433.24)

The improvements are attributed to the pre-trained WFM's generalization ability: both models see the same fine-tuning data (DL3DV-10K) and are tested on different data (RealEstate10K), but the Cosmos model transfers its general 3D consistency knowledge while CamCo overfits to the training domain.

Robotic manipulation (Section 6.2). Two tasks are demonstrated:

Instruction-based video prediction: Given a current video frame and a text instruction, predict a video of the robot executing the instruction. Fine-tuned on the Cosmos-1X dataset (~12,000 episodes from the EVE humanoid robot). In human evaluation on 23 test episodes, Cosmos-Predict1-7B-Video2World-Sample-Instruction achieves 78.3% overall preference versus 13.0% for a VideoLDM-Instruction baseline (8.7% ties), evaluated across instruction following, object permanence, verity (absence of imagined objects), and overall reasonableness for robot planning.

Action-based next-frame prediction: Given the current frame and a 7-dimensional action vector ($\Delta x, \Delta y, \Delta z, \Delta\theta_r, \Delta\theta_p, \Delta\theta_y, \Delta\text{Gripper}$ in gripper coordinate space), predict the next frame. Fine-tuned on the Bridge dataset (~20,000 episodes of a robot arm in a kitchen environment). For the diffusion-based model, action vectors are encoded via an action embedder MLP and added to the DiT timestamp embedding. The fine-tuned 7B Video2World model achieves PSNR 21.14, SSIM 0.82, FVD 190 versus the IRASim-Action baseline (PSNR 19.13, SSIM 0.64, FVD 593).

Autonomous driving (Section 6.3). The goal is multi-view, trajectory-conditioned video generation for training and evaluating autonomous vehicle policies.

Dataset: Real Driving Scene (RDS) dataset, approximately 3.6M 20-second clips (20,000 hours) from six synchronized cameras (front, left, right, rear, rear-left, rear-right), with ego-motion data providing trajectory information. The dataset is curated to balance attributes including traffic density, weather, illumination, ego speed, road type, and rare structures (tollbooths, tunnels, etc.).

Architecture modifications for multi-view generation:

• View-independent positional embedding: 3D FPS-aware RoPE is applied independently to each view, rather than extending to a 4D representation. This means each camera view's tokens are encoded with their own spatial and temporal positions, without explicit encoding of which view they belong to.
• View embedding: A learnable embedding per camera view is added to the denoising function's conditioning inputs, distinguishing views globally rather than positionally.
• View-dependent cross-attention: Each view's cross-attention attends only to the text description for that specific view (since each camera has a separate caption describing its field of view). This prevents the model from mixing content across views (e.g., confusing a front-view car with a rear-view car).

Trajectory control: A trajectory is a sequence of 64 3D points representing the ego vehicle's translation from the initial position at 0.1-second intervals. The trajectory embedding is computed by a separate encoder and incorporated as additional conditioning into the denoiser.

Multi-view generation pipeline: Cosmos-Predict1-7B-Text2World-Sample-MultiView generates 57 frames (all 6 views) from text. This output is then extended to 201 frames by Cosmos-Predict1-7B-Video2World-Sample-MultiView, which takes the previous video frames as conditioning and generates the continuation autoregressively.

Evaluation:

• Generation quality: FID 32.16, FVD 210.23 (versus VideoLDM-MultiView baseline: FID 60.84, FVD 884.46)
• Multi-view geometric consistency (Table 24): Temporal Sampson Error 0.68 (vs. 1.24 for baseline, 0.69 for real videos), Cross-view Sampson Error 2.11 (vs. 6.48 for baseline, 1.71 for real videos). The Cosmos model's cross-view error approaches real-video quality, indicating near-photorealistic 3D consistency.
• Trajectory consistency (Table 25): Trajectory Following Error of 20.20 cm versus the ground-truth oracle's 13.49 cm—the generated videos follow the specified trajectory within ~7 cm of the physical trajectory, sufficient for autonomous driving simulation.

---

Evaluation Methodology for Pre-trained WFMs

The paper evaluates pre-trained WFMs on two dimensions not captured by standard video generation metrics (FID, FVD): 3D consistency and physics alignment.

3D consistency evaluation (Section 5.3.1). The test set comprises 500 videos from RealEstate10K, captioned as static scenes. Two metrics are used:

Geometric consistency: The Sampson error measures how well epipolar geometry constraints are satisfied between pairs of frames:

$\epsilon_{\text{samp}} = \frac{1}{N} \sum_{i=1}^{N} \frac{|\bar{\mathbf{y}}_i^\top \mathbf{F} \bar{\mathbf{x}}_i|}{\sqrt{\|\mathbf{S} \mathbf{F} \bar{\mathbf{x}}_i\|_2^2 + \|\mathbf{S} \mathbf{F}^\top \bar{\mathbf{y}}_i\|_2^2}}, \quad \mathbf{S} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{bmatrix}$

where $\bar{\mathbf{x}}_i, \bar{\mathbf{y}}_i$ are $N$ keypoint correspondences (in homogeneous coordinates) between a frame pair, matched using SuperPoint and LightGlue, and $\mathbf{F}$ is the fundamental matrix estimated via OpenCV's 8-point RANSAC.

What it computes: for each correspondence, the numerator $|\bar{\mathbf{y}}_i^\top \mathbf{F} \bar{\mathbf{x}}_i|$ is the epipolar distance—how far the point in the second frame is from the epipolar line corresponding to the point in the first frame. If the generated video is perfectly 3D-consistent, corresponding points should lie exactly on their epipolar lines, making this distance zero. The denominator normalizes by the gradient of the epipolar constraint, converting the algebraic error to an approximate geometric distance in pixels.

Why this form: the Sampson error is the standard first-order approximation of geometric reprojection error that avoids the iterative optimization required for exact reprojection error. It correctly handles the scaling ambiguity of the fundamental matrix (multiplying $\mathbf{F}$ by a scalar does not change the epipolar constraint, and the normalization makes the error scale-invariant).

Additionally, the camera pose estimation success rate measures the fraction of generated videos where COLMAP can successfully estimate camera poses from the generated frames. A high success rate indicates that the frames are geometrically self-consistent—they could plausibly be 2D projections of a rigid 3D scene.

View synthesis consistency: Every 8th frame is held out, and a 3D Gaussian Splatting model is fit to the remaining frames using Nerfstudio defaults. The held-out frames are then rendered from the fitted model and compared to the generated frames via PSNR, SSIM, and LPIPS.

What it computes: if the generated video is 3D-consistent, it should be possible to reconstruct the underlying 3D scene from a subset of frames and re-render the held-out views with high fidelity. High PSNR/SSIM and low LPIPS indicate that the generated frames are consistent projections of a shared 3D structure.

The Cosmos-7B-Text2World achieves Sampson error 0.355 (vs. 0.841 for VideoLDM), pose estimation success rate 62.6% (vs. 4.4%), and view synthesis PSNR 33.02 (vs. 26.23). Remarkably, the pose estimation success rate exceeds even real videos (56.4%), suggesting the generated videos may be more geometrically regular than real ones—possibly because the model learns a strong 3D prior that eliminates the slight camera calibration imperfections present in real footage.

Physics alignment evaluation (Section 5.3.2). A more direct test of physical understanding: can the WFM predict physically correct object motion?

Synthetic data generation: Eight 3D scenarios are designed using PhysX and Isaac Sim to test specific physical principles:

1. Free-falling objects (gravity, collision)
2. Tilted planar slope (gravity, moment of inertia)
3. U-shaped slope (potential/kinetic energy conservation)
4. Stable stack (balanced forces)
5. Unstable stack (gravity, collision chains)
6. Dominoes (momentum transfer)
7. Seesaw (torque, rotational inertia)
8. Gyroscope (angular momentum, precession)

For each scenario, object types, textures, sizes, and backgrounds are randomized. 800 videos of 100 frames at 1080p are rendered from 4 static camera views each.

WFM conditioning: The WFM is conditioned on either 1 or 9 initial frames from the simulation and (when applicable) a text prompt captioned from the conditioning frames, focusing on the kinematic state of the objects. The model then generates the remaining frames autoregressively.

Metrics:

• Pixel-level: PSNR and SSIM between generated and ground-truth frames (excluding conditioning frames).
• Feature-level: DreamSim similarity, a learned perceptual metric.
• Object-level: Using ground-truth instance segmentation masks and SAMURAI tracking, the Intersection-over-Union (IoU) between predicted and ground-truth object masks is computed for each dynamic object across frames. This isolates object motion accuracy from background quality.

Key results (Table 20): On 9-frame conditioning, the diffusion-based 7B Video2World achieves PSNR 21.06, SSIM 0.691, DreamSim 0.859, Avg. IoU 0.592. With only 1 conditioning frame, performance drops substantially (PSNR 17.34, IoU 0.332), confirming that multi-frame context is essential for inferring velocities and accelerations. The autoregressive models achieve lower pixel-level metrics (e.g., 12B model: PSNR 18.22 at 9 frames) but comparable object-level IoU (0.487).

Notably, larger models do not significantly improve physics alignment—the 14B diffusion model achieves similar IoU (0.598) to the 7B (0.592), and the 13B autoregressive model (IoU 0.482) is comparable to the 4B (0.481). The paper attributes this to a fundamental limitation: "all the WFMs equally struggle with physics adherence and require better data curation and model design" (Section 5.3.2). The evaluation reveals specific failure modes: objects spontaneously appearing or disappearing, shape deformation, implausible kinematics, and gravity violations.

---

Guardrail System

The two-stage guardrail protects users from harmful content in both input prompts and generated videos.

Pre-Guard (text-domain):

• Keyword blocking: Input words are lemmatized using WordNetLemmatizer to extract root words, then compared against a "comprehensive" hard-coded blocklist. Any match blocks the entire prompt.
• Aegis guardrail: Aegis-AI-Content-Safety-LlamaGuard-LLM-Defensive-1.0, a fine-tuned version of Llama-Guard, classifies prompts against NVIDIA's taxonomy of 13 safety risk categories. The defensive version uses tighter boundaries than the permissive version. Prompts in categories including violence, sexual, criminal planning, weapons, substance abuse, suicide, CSAM, hatred, harassment, threat, and profanity are blocked.

Post-Guard (vision-domain):

• Video content safety filter: A per-frame multi-class classifier trained on a combination of (a) sampled frames from the training dataset labeled by a VLM, (b) synthetic videos generated with targeted prompts to cover corner cases, and (c) human-annotated "gold standard" labels. SigLIP embeddings are extracted per frame, and a simple MLP classifier labels each frame as safe or unsafe. If any frame in a generated video is unsafe, the entire video is flagged.
• Face blur filter: RetinaFace detects facial regions with high confidence. Any face larger than 20×20 pixels is pixelated—obscuring identity while preserving scene composition for Physical AI applications (where precise facial detail is rarely needed). The 20×20 threshold prevents blurring distant, unidentifiable faces while protecting privacy for recognizable individuals.

Red teaming. Over 10,000 distinct prompt-video pairs have been tested by expert annotators, providing high-quality annotations with start/end frames for unsafe content detection, and probing each guardrail component independently with adversarial examples.

4. Key Insights and Innovations

Innovation 1: The Pre-Training-Then-Post-Training Paradigm for World Models

The dominant assumption in learned world modeling, from Ha and Schmidhuber (2018) through the Dreamer family (Hafner et al., 2019, 2021, 2023) to recent video-generation-as-simulation work (Valevski et al., 2024; Yang et al., 2023), has been that each world model is trained from scratch on data from a specific environment. You want a world model for a Franka arm in a lab? Train on Franka-arm-in-a-lab data. You want one for autonomous driving? Train on driving data. This has a direct consequence: the world model is only as general as its training set, which in practice means it is highly domain-specific. There is no transfer of physical knowledge across environments because there is no mechanism for it—the model starts with random weights every time.

The Cosmos platform introduces the pre-training-then-post-training paradigm to world modeling, which is at once conceptually straightforward and structurally transformative. It directly imports the foundation model philosophy from NLP and vision—BERT pre-trained on internet text then fine-tuned for sentiment analysis, CLIP pre-trained on image-text pairs then fine-tuned for classification—into a domain where no such generalist pre-training had been demonstrated. The concept itself is not novel in AI broadly, but its execution and validation for world simulation constitute the paper's most significant intellectual contribution.

What makes this move distinctive is that it addresses the data scaling bottleneck of Physical AI not by collecting more physical interaction data, but by decoupling the problem into two parts. First, learn what the visual world looks like and how it moves from readily available internet video at scale (~100M clips, nine categories covering driving, manipulation, human activity, nature, and synthetic footage). Second, learn how a specific system interacts with that world from a small amount of domain-specific data (thousands of episodes rather than millions). The pre-trained model provides a rich prior over visual dynamics—gravity, object permanence, perspective consistency, common motion patterns—that the post-training stage only needs to adapt to a particular embodiment and task, not learn from scratch.

The evidence for the paradigm's effectiveness is distributed across all three post-training demonstrations. In camera control (Table 22), the Cosmos post-trained model achieves 82.0% camera pose estimation success rate versus 43.0% for CamCo, despite both models being fine-tuned on identical data (DL3DV-10K) and tested on distribution-shifted data (RealEstate10K). The 39-point gap is not attributable to architecture or training tricks—it reflects the pre-trained model's transfer of general 3D consistency knowledge that CamCo, trained from scratch on the same fine-tuning set, cannot acquire. In instruction-based robotics (Figure 24), the Cosmos 7B Video2World model achieves 78.3% overall human preference versus 13.0% for a VideoLDM baseline fine-tuned on the same Cosmos-1X dataset. In action-conditioned robotics (Table 23), the Cosmos models outperform IRASim-Action on the Bridge dataset by 2–3 dB PSNR and cut the FVD by more than half. The consistent pattern—Cosmos post-trained models substantially outperforming baselines trained from scratch on identical fine-tuning data—provides convergent evidence that the pre-trained generalist captures transferable physical knowledge.

An important subtlety: the paper does not claim that pre-training on internet video teaches physics in any rigorous sense. The physics alignment evaluation (Section 5.3.2, Table 20) shows that even the largest models (14B diffusion, 13B autoregressive) struggle with rigid-body dynamics—they achieve object-level IoU of ~0.48–0.60, far from ground truth. What the pre-training captures is not Newtonian mechanics but visual-physical correlations: what falling objects look like, how shadows move, how fabric deforms when handled. These correlations are sufficient to bootstrap domain-specific fine-tuning but insufficient for reliable physical simulation. This diagnostic distinction—pre-training provides visual-physical priors, not physics understanding—is itself an important clarification for the field, tempering both hype ("WFMs understand physics!") and dismissal ("WFMs are just video generators").

This is a fundamental shift in approach rather than an incremental refinement. Prior to Cosmos, the world model community operated with a specialist mindset: each environment, each embodiment, each task deserved its own model trained from scratch. After Cosmos, the question shifts to: what is the minimal amount of domain-specific data needed to specialize a generalist pre-trained WFM, and how does this trade off with training a specialist from scratch? This reframing has direct practical consequences for how Physical AI developers allocate their data collection budgets.

---

Innovation 2: Difficulty-Aware Video Data Curation as a Pre-Training Dataloader Design Problem

The paper gives an unusual amount of attention—an entire Section 3, roughly 15% of the paper's length—to its video data curation pipeline. This emphasis is not merely thorough engineering documentation; it reflects a conceptual insight: the quality and distribution of pre-training video data determines the ceiling of world foundation model capabilities in ways that are not captured by standard dataset scaling narratives. The innovation is not any individual curation step (shot detection, filtering, annotation, deduplication), each of which has precedent in the video understanding and generation literature. It is the systematic design of a data pipeline whose objective is not dataset size but physical information density—maximizing the fraction of training tokens that teach the model something about how the visual world behaves.

The dominant approach in large-scale video generation has been scale-maximalist: collect as much video as possible, filter minimally, and rely on model capacity and training compute to extract signal from noise. The Cosmos paper breaks with this assumption by designing an explicit, multi-stage filtering pipeline that removes approximately one-third of the data (the 30% removed during semantic deduplication alone, plus additional filtering) while claiming improved model quality. This is a counter-narrative to the "more data is always better" scaling philosophy.

The curation pipeline makes several non-obvious design choices that reveal what the authors believe is important for learning physical dynamics:

Re-weighting toward physical interaction. The training data is not a uniform sample of available video. Nine categories are explicitly targeted with specific percentages (Section 3.1), and within those, content types relevant to physical world simulation are upsampled while "abstract visual patterns, video game footage, animated content" are excluded entirely. Nature videos, which are abundant on the internet but contain limited object manipulation and interaction, are downsampled relative to their natural prevalence. Hand manipulation (16%), spatial navigation (16%), and nature dynamics (20%) receive more weight than first-person POV (8%) or synthetic footage (4%). This is not a generic video dataset—it is purpose-built for teaching models about physical interactions from specific classes relevant to Physical AI.

Removing post-processed text as a physics confound. The text overlay filter (Section 3.3.3) distinguishes between text naturally present in a scene (street signs, which are part of the physical world) and text added in post-processing (subtitles, watermarks, which are editing artifacts). This distinction is motivated by the observation that "text tends to co-occur with different visual effects" (Section 3.3.3). Training on videos with post-processed text would teach the model spurious correlations: certain text styles predict certain visual effects, not because of physics, but because of editing conventions. The WFM would learn to reproduce these artifacts, degrading its ability to simulate raw physical scenes. This is a diagnostic move: identifying a specific data contamination that would corrupt the very capability (physics prediction) the model is being trained to acquire.

Motion filtering as difficulty-based data selection. Not all video motion is equal for learning physics. The motion filter (Section 3.3.1) removes both static videos (zero information about dynamics) and videos with random, abrupt camera motion (which would teach the model to predict jittery, physically implausible camera trajectories). Videos with smooth, intentional camera motion—pans, zooms, tracking shots—are retained and tagged with their motion type. This is a form of curriculum design at the data level: the model sees examples where camera motion is predictable and smooth, learning that the visual world moves coherently, before being asked to generate novel camera trajectories during post-training.

The deduplication-to-search-engine pivot. Semantic deduplication removes 30% of training data—a massive fraction that would have wasted significant compute on redundant examples. But the paper goes further: the same embeddings and clustering results used for deduplication are repurposed to build a visual search engine over the entire training dataset. This transforms data curation from a one-time pre-processing step into an interactive capability: developers can query the training data to understand what their pre-trained model has seen, identify gaps relative to their downstream application, and make informed decisions about what additional data to collect or how to post-train. This is infrastructure-as-contribution: the platform provides not just pre-trained weights but the tools to understand and extend them.

The innovation here is conceptual, not algorithmic. Each individual curation technique (shot detection, optical flow filtering, VLM annotation, semantic deduplication) draws on prior work. What is novel is the unified framework that treats data curation as a capacity allocation problem: how to select, from a massive pool of raw video, the subset that maximizes the rate at which a world model learns transferable physical knowledge, subject to a compute budget. This framework implies that dataset quality is not absolute—it is relative to the downstream goal (Physical AI, not general video generation), and curation choices that are neutral or harmful for one goal may be essential for another.

The evidence that this matters is largely implicit. The paper does not run an ablation where the same model is trained on uncurated versus curated data—that would be prohibitively expensive at this scale. But the downstream results—particularly the strong transfer from pre-trained to post-trained models across diverse domains (camera control, robotics, driving)—provide indirect evidence that the curation successfully produced a dataset rich in transferable physical information. A model pre-trained on a generic, uncurated video crawl might still generate visually appealing videos but would likely fail to provide the strong physical prior that enables efficient post-training.

This is a reframing innovation, not a metric gain. It shifts the conversation around video pre-training from "how much data" to "what kind of data, for what purpose." The curation pipeline design principles—explicit targeting of physical interaction categories, removal of editing artifacts, motion-based filtering, deduplication as quality improvement—constitute a design philosophy for building pre-training datasets for world models, one that the paper makes explicit and open-source. Future work that builds on Cosmos will inherit this philosophy, whether or not they use the same pipeline implementation.

---

Innovation 3: Tokenizer Design as a Causal, Joint Image-Video Compression Problem with Physical AI Requirements

The video tokenizer architecture (Section 4) might appear to be an infrastructure component—necessary for scaling, but not intellectually interesting. This would be a misreading. The Cosmos tokenizer embodies a specific set of design decisions that reflect a reconceptualization of what video tokenization needs to achieve for world foundation models, as distinct from the requirements of general video generation.

Prior tokenizers have been designed for generation quality alone: maximize PSNR/SSIM at a given compression rate. The Cosmos tokenizer optimizes for a broader set of desiderata that arise specifically from the world model use case:

Causality as a physical requirement, not a computational convenience. Most video tokenizers use non-causal 3D convolutions or bidirectional attention that processes all frames simultaneously. This is fine for video generation where the entire output is produced at once and can depend on all other frames. But a world model for Physical AI must operate causally: a robot's prediction of the next frame cannot depend on future frames it has not yet observed. The Cosmos tokenizer's temporal causality—implemented through causal convolution with left-padding and causal temporal attention—ensures that the latent representation at time $t$ depends only on frames $\leq t$. This is not a minor implementation detail; it is an architectural commitment to match the inference-time constraints of the downstream application. A non-causal tokenizer would leak future information into the latent representation, producing inflated training metrics (the model could "cheat" by encoding future frames into the latent of past frames) while degrading real-world deployment performance where such leakage is impossible.

Joint image-video training enabled by causal design. The causal architecture has a second, even more consequential property: when the input is a single image, the causal tokenizer naturally reduces to an image tokenizer, because there are no future frames for the causal operations to access. This enables alternating between image and video batches during training, which the paper argues is critical for visual quality: "joint image-video training has been shown to improve the visual quality of the generated videos and accelerate the model training" (Section 3.1). The intuition is that image datasets are orders of magnitude larger and more diverse than video datasets, containing rich appearance information (textures, materials, object categories, lighting conditions) that video-only training would see far less of. The causal design makes this joint training architecturally natural rather than requiring separate image and video encoders or ad-hoc temporal padding schemes.

This is a design choice with consequences. A non-causal tokenizer could still be trained jointly with images by treating single images as videos with temporal padding or by training a separate image encoder. But the causal design makes the architecture uniform: the same encoder processes both modalities, producing latents in the same space, with the same semantics. This uniformity is important for the downstream WFM, which receives latent representations and must make sense of them regardless of whether they came from an image or video input. If the tokenizer used different processing for images versus videos, the latent space would have a modality gap that the WFM would need to learn to bridge—an unnecessary burden.

The FSQ choice for discrete tokenization as a simplicity-over-performance tradeoff. The discrete tokenizer uses Finite-Scalar-Quantization rather than the more common VQ-VAE approach. FSQ simply bounds each latent dimension to a small integer range (the levels $(8, 8, 8, 5, 5, 5)$) without learning a codebook. This eliminates the commitment loss and codebook collapse problems that have made VQ-VAE training notoriously unstable, especially at large vocabulary sizes. The cost is that FSQ has less representational flexibility—the quantization grid is fixed rather than learned—but the paper demonstrates that with sufficient latent dimensionality (6 dimensions), this limitation does not bottleneck reconstruction quality. The resulting discrete tokenizer achieves superior PSNR and SSIM compared to VQ-based baselines (Table 6: Cosmos-Tokenize1-DV4×8×8-360p achieves 32.97 PSNR on DAVIS versus 28.17 for VideoGPT-Tokenizer and 20.02 for Omni-Tokenizer). The innovation is the recognition that for world model training at scale, training stability and simplicity of the tokenizer matter more than squeezing out the last 0.5 dB of reconstruction quality, because tokenizer training failures or instabilities propagate to WFM training failures, and at the scale of 10,000-GPU clusters, debugging such failures is extremely costly.

The two-generation training regime as a compute-quality tradeoff. Cosmos-0.1-Tokenizer trains on shorter video mini-batches (49 frames for CV, 17 for DV); Cosmos-Tokenize1 trains on longer sequences (121 frames, 49 frames). This split is not merely about quality—it reflects a resource allocation decision: training on 121-frame sequences requires more GPU memory per batch, reducing the effective batch size or requiring more GPUs. The 0.1 variants serve as fast-iteration baselines for architecture exploration; the 1 variants are the production models where additional compute is invested for final quality. This two-tier approach is pragmatic but conceptually significant: it acknowledges that tokenizer development, like model development, benefits from rapid experimentation cycles that full-scale training cannot provide.

These design decisions collectively represent a reconceptualization of tokenizer requirements. For general video generation, the sole metric is reconstruction quality at a given compression rate. For world foundation models, additional criteria enter: causality (for alignment with Physical AI inference), joint image-video compatibility (for leveraging image data), training stability (for reliable scaling to large clusters), and architectural simplicity (for maintainability and iteration speed). The Cosmos tokenizer demonstrates that optimizing for this broader set of criteria produces tokenizers that are simultaneously better at reconstruction (Figure 8: +4 dB PSNR over CogVideoX at equivalent compression) and faster (Table 9: 2–12× speedup). The innovation is in defining the problem, not just implementing a solution.

This is a design-philosophy innovation that changes how future work should think about tokenizer evaluation. It implies that tokenizer benchmarks should include not just PSNR/SSIM but also metrics of temporal causality (does frame $t$'s latent leak information from frame $t+1$?), joint modality performance (does the tokenizer produce consistent latents for images and videos of the same scene?), and stability under scale (does training remain stable with 10K GPUs?). The TokenBench dataset introduced in the paper is a step toward this broader evaluation framework, though most of the additional criteria remain implicitly evaluated rather than benchmarked.

---

Innovation 4: 3D Consistency and Physics Alignment as First-Class Evaluation Dimensions for World Models

The paper introduces two evaluation frameworks—3D consistency (Section 5.3.1) and physics alignment (Section 5.3.2)—that represent a fundamental departure from how world models and video generative models have been evaluated. The innovation is not any individual metric (Sampson error, camera pose estimation success rate, Gaussian splatting view synthesis, and simulation-based rollout comparisons all have prior art). It is the argument, made through evaluation design, that perceptual quality metrics (FID, FVD, PSNR) are insufficient and potentially misleading for assessing world models.

The standard evaluation paradigm for video generation compares generated videos to a reference set of real videos using distribution-level metrics (FID, FVD) or frame-level metrics (PSNR, SSIM). These metrics capture visual fidelity—do the generated videos look like real videos?—but are blind to whether they are physically plausible. A video of a teacup floating gently upward, rendered in photorealistic 4K with perfect lighting and texture, scores well on FID but fails catastrophically as a world simulation.

The Cosmos paper introduces evaluation dimensions that probe specific capabilities required for world simulation:

3D consistency probes whether generated videos could be projections of a rigid 3D scene. The Sampson error measures violations of epipolar geometry constraints between frame pairs—if a generated video is 3D-consistent, corresponding points in different frames should satisfy the epipolar constraint induced by the (unknown but estimable) camera motion between those frames. A low Sampson error indicates that the generated frames are geometrically consistent with a single underlying 3D structure. The camera pose estimation success rate tests this more stringently: can a standard structure-from-motion pipeline (COLMAP) successfully recover camera poses from the generated frames? If the frames are not 3D-consistent, pose estimation will fail because the correspondences are geometrically contradictory.

Importantly, these metrics do not require ground-truth 3D information—they are computed directly from the generated video using standard computer vision techniques. This makes them deployable at scale without manual annotation, unlike many physics evaluation approaches that require ground-truth simulation data or human judgment.

Physics alignment probes whether generated videos obey Newtonian mechanics. The synthetic simulation benchmark is a form of controlled experimentation: by generating videos from known initial conditions and known physical laws (implemented in PhysX), the paper creates ground-truth references where the correct physical behavior is deterministic. The WFM's generated output can then be compared to this ground truth at multiple levels—pixel-level (does the video look the same?), feature-level (are the semantic structures similar?), and object-level (are the dynamic objects in the right positions?). The object-level metric (tracking-based IoU) is particularly revealing because it isolates object motion from background quality and visual style: a WFM can produce visually impressive videos where objects still move incorrectly.

The results from these evaluations are as important for what they reveal about the limitations of current WFMs as for what they show about their capabilities. The physics alignment evaluation demonstrates that even the largest models (14B parameters) fail on basic rigid-body dynamics—objects change shape, appear and disappear, move with implausible kinematics, violate gravity (Section 5.3.2). This is not a failure of the evaluation; it is a diagnostic finding: the visual-physical correlations learned from internet video do not constitute physical understanding, and scaling model size alone does not close this gap (the 14B model performs similarly to the 7B on object-level IoU). This finding has direct implications for future research: improving physics alignment will require better data (e.g., physically accurate simulation data for pre-training, filtering out physically implausible internet videos) or better architectures (e.g., explicit physics priors, differentiable simulation layers), not just larger models.

Why this is conceptually significant beyond this paper. The evaluation design establishes a framework for measuring progress in world modeling that is more informative than FID/FVD trends. It decomposes "world model quality" into at least three separable dimensions—visual fidelity (standard metrics), 3D consistency (geometric metrics), and physics alignment (simulation-based metrics)—each of which can improve or degrade independently. A future paper that claims a better WFM can be evaluated on whether it improves 3D consistency, physics alignment, or visual fidelity—and a paper that improves only visual fidelity while degrading physics alignment would be flagged as potentially trading off the wrong objective.

This is a methodological innovation that the field can adopt regardless of whether they use Cosmos models. The combination of epipolar geometry metrics (requiring no ground truth), structure-from-motion success rates (testing geometric self-consistency), and controlled simulation rollouts (providing ground-truth physical behavior) provides a multi-faceted evaluation toolkit that is more diagnostic than any single metric. The release of TokenBench and the synthetic physics evaluation scenarios as open-source resources (github.com/NVlabs/TokenBench) further amplifies this contribution by providing standardized benchmarks that other researchers can use.

The paper is candid that these evaluations are incomplete: "there are several other important aspects required for evaluation. We leave a more comprehensive evaluation as future work" (Section 5.3). Missing dimensions include long-term temporal consistency (do objects maintain identity over hundreds of frames?), contact-rich dynamics (do objects deform realistically upon collision?), multi-object interaction (do multiple objects interact according to physics, not just independently?), and causal intervention (does changing an action produce the physically correct change in outcome?). But establishing the principle—that world models require evaluation beyond visual quality—is the key contribution, not providing the final, comprehensive benchmark.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The pre-trained WFMs are evaluated on two custom evaluation sets. For 3D consistency, the paper curates 500 videos randomly chosen from the RealEstate10K test set, captioned as static scenes using a proprietary VLM (Section 5.3.1). For physics alignment, the paper generates 800 synthetic videos across 8 physics scenarios (free-falling objects, tilted slopes, dominoes, etc.) using PhysX and Isaac Sim, rendered at 1080p with 100 frames each from 4 static camera views (Section 5.3.2). Post-trained models are evaluated on domain-specific test sets: RealEstate10K for camera control (500 samples), 23 episodes from Cosmos-1X for instruction-based robotics (human evaluation), 100 episodes from the Bridge test set for action-conditioned robotics, and 1000+800 samples from the RDS dataset for autonomous driving.

• Base model(s). The pre-trained models span two families: diffusion-based (Cosmos-Predict1-7B-Text2World, Cosmos-Predict1-14B-Text2World, and their Video2World derivatives) and autoregressive-based (Cosmos-Predict1-4B, Cosmos-Predict1-12B, and their Video2World derivatives Cosmos-Predict1-5B-Video2World and Cosmos-Predict1-13B-Video2World). All are trained from scratch on the curated ~100M video clip dataset using 10,000 H100 GPUs over three months. The scale range (4B to 14B parameters) is chosen to study the effect of model capacity on world simulation quality (Section 5).

• Metrics. The paper evaluates on multiple dimensions beyond standard generation metrics. For 3D consistency: Sampson error (first-order approximation of epipolar distance in pixels, normalized by frame diagonal for a 960×540 canvas), camera pose estimation success rate (percentage of generated videos where COLMAP successfully recovers camera poses), and view synthesis consistency (PSNR, SSIM, LPIPS on held-out frames reconstructed via 3D Gaussian Splatting). For physics alignment: pixel-level (PSNR, SSIM), feature-level (DreamSim similarity), and object-level (average IoU between ground-truth and SAMURAI-tracked object masks across all dynamic objects and frames). For video generation quality: FID (Frechet Inception Distance), FVD (Frechet Video Distance). For camera control: rotation error (degrees), translation error (normalized), and pose estimation success rate. For autonomous driving: Temporal Sampson Error (TSE, median across all views and adjacent frame pairs), Cross-view Sampson Error (CSE, across different views averaged in time), Trajectory Agreement Error (TAE, comparing trajectories estimated from different camera pairs), and Trajectory Following Error (TFE, comparing estimated trajectory to the input control trajectory in cm).

• Baselines. For 3D consistency and instruction-based robotics: VideoLDM (Blattmann et al., 2023), a latent video diffusion model trained from scratch on the relevant fine-tuning data. For camera control: CamCo (Xu et al., 2024), a state-of-the-art camera-controllable image-to-video model, also fine-tuned on DL3DV-10K. For action-conditioned robotics: IRASim (Zhu et al., 2024), fine-tuned on the Bridge dataset as IRASim-Action. For autonomous driving: VideoLDM fine-tuned on the RDS dataset as VideoLDM-MultiView. For physics alignment: no explicit external baseline; comparisons are made between Cosmos model variants and against ground-truth simulation data.

• Generation budget / compute accounting. For diffusion models, generation is performed with 50 denoising steps (the standard EDM inference schedule). For autoregressive models, tokens are generated sequentially until the full sequence length is produced; the Medusa speculative decoding experiments report tokens/second and number of forward passes as efficiency metrics. Inference benchmarks for the autoregressive models are conducted on H100 GPUs with torch.compile's "max-autotune" mode in BF16 precision. For real-time generation experiments, the model is adapted to 320×512 resolution and benchmarked at 10 FPS input.

• Cross-validation / statistical protocol. No formal cross-validation is reported for the pre-training evaluations (3D consistency and physics alignment use fixed test sets). For human evaluation of instruction-based robotics, ten evaluators assess 23 test episodes, comparing anonymized video pairs across four dimensions (instruction following, object permanence, verity, overall reasonableness). For autonomous driving trajectory consistency, 200 samples are gathered per trajectory category (moving forward, turning left, turning right, others) totaling 800 samples. The physics alignment evaluation uses four random seeds for rollouts and averages across frames, videos, and seeds.

Main Quantitative Results

Pre-Trained WFM: 3D Consistency Evaluation

The central claim is that Cosmos WFMs generate videos that are geometrically consistent with underlying 3D structure, a prerequisite for world simulation. Table 19 presents the results on the 500-video RealEstate10K test set.

The diffusion-based Cosmos-Predict1-7B-Text2World achieves a Sampson error of 0.355, compared to 0.841 for VideoLDM—a 58% reduction in geometric inconsistency. The camera pose estimation success rate tells an even starker story: 62.6% for Cosmos versus 4.4% for VideoLDM. This means VideoLDM's generated frames are so geometrically inconsistent that standard structure-from-motion fails to recover camera poses in 95.6% of cases, while Cosmos-generated videos are geometrically self-consistent more often than real videos (56.4% success rate on RealEstate10K's real footage). The view synthesis metrics confirm this pattern: the Cosmos model achieves PSNR 33.02 dB, SSIM 0.939, and LPIPS 0.070 when held-out views are reconstructed via 3D Gaussian Splatting, versus VideoLDM's 26.23 dB, 0.783, and 0.135. The Cosmos model's view synthesis quality approaches real videos (PSNR 35.38, SSIM 0.962, LPIPS 0.054).

The Video2World variant (Cosmos-Predict1-7B-Video2World) shows slightly degraded 3D consistency compared to Text2World (Sampson error 0.473, PSNR 30.66), but still dramatically outperforms VideoLDM. This degradation is expected: Video2World conditions on past frames, which can introduce inconsistencies between the conditioning and generated content that manifest as geometric errors.

The autoregressive models demonstrate competitive but lower 3D consistency. The unprompted Cosmos-Predict1-4B achieves Sampson error 0.433 with 35.6% pose estimation success; the larger Cosmos-Predict1-12B achieves 0.433 Sampson error (identical to 3 decimal places) and 35.6% success (also identical—potentially suggesting the evaluation is saturated for these models or the Table 19 values for 12B may not be fully independent measurements, though this is not discussed in the paper). The prompted variants (5B-Video2World and 13B-Video2World) show slightly lower pose estimation success rates (27.0% and not-reported-for-13B, though the text states Tab. 19 reports it—it lists 0% explicitly? No, the table shows Cosmos-Predict1-13B-Video2World: Sampson 0.392, but the pose success rate appears to only be in the text discussion implicitly; the table entry may be inferred from context). The autoregressive models' lower 3D consistency is attributed to the aggressive compression of the discrete tokenizer (8×16×16 = 2048× versus the diffusion model's 8×8×8 = 512×), which discards spatial detail necessary for precise geometric correspondence.

Pre-Trained WFM: Physics Alignment Evaluation

The physics alignment results in Table 20 test a more demanding capability: can the WFM predict physically correct object motion when conditioned on initial frames of a simulation? The evaluation is conducted on 800 synthetic videos across 8 physics scenarios.

With 9-frame conditioning, the diffusion-based Cosmos-Predict1-7B-Video2World achieves the best pixel-level metrics: PSNR 21.06, SSIM 0.691. The object-level metric—average IoU between tracked ground-truth and predicted object masks—is 0.592. This means that, on average across all dynamic objects and frames, the predicted object masks overlap with ground-truth masks by about 59%. With only 1 conditioning frame, performance drops dramatically to PSNR 17.34 and IoU 0.332, quantifying the value of multi-frame context for inferring velocities and accelerations.

The autoregressive models trail in pixel-level metrics. Cosmos-Predict1-12B with 9-frame conditioning achieves PSNR 18.22, SSIM 0.487, and IoU 0.487. The gap versus diffusion (IoU 0.592 vs. 0.487) reflects the discrete tokenizer's coarser representation limiting the precision of object boundary prediction.

A critical finding emerges when comparing model scales: the 14B diffusion model achieves IoU 0.598 (9 frames), nearly identical to the 7B model's 0.592. Similarly, the 13B autoregressive model achieves IoU 0.482 versus the 4B model's 0.481. Scaling model size does not improve physics alignment. The paper explicitly interprets this: "all the WFMs equally struggle with physics adherence and require better data curation and model design" (Section 5.3.2). This is a negative result with positive diagnostic value: it tells the field that larger models trained on internet video will not spontaneously develop Newtonian physics understanding—the bottleneck is in the data and the training objective, not model capacity.

The feature-level metric (DreamSim) shows a different pattern: all models achieve 0.86–0.87 with 9-frame conditioning (and 0.82–0.84 with 1 frame). This suggests that even when object-level trajectories are wrong (low IoU), the semantic appearance of the scene remains plausible—the WFM predicts a video that "looks like" a domino chain reaction even if the dominoes fall at the wrong times or in the wrong directions.

Post-Trained WFM: Camera Control

Table 22 and Figure 21 present the comparison between Cosmos-Predict1-7B-Video2World-Sample-CameraCond and CamCo, both fine-tuned on DL3DV-10K and evaluated on the distribution-shifted RealEstate10K test set (500 samples).

The Cosmos model achieves a camera pose estimation success rate of 82.0%, versus 43.0% for CamCo—a 39 percentage point improvement. On the videos where poses are successfully estimated, the rotation error is 1.646° for Cosmos versus 8.277° for CamCo (5× reduction), and the translation error is 0.038 versus 0.185 (4.9× reduction). The video generation quality metrics similarly favor Cosmos: FID 14.30 versus 57.49, FVD 120.49 versus 433.24.

The qualitative examples in Figure 21 illustrate the practical difference: CamCo frequently fails to generate content beyond the input image (producing near-static outputs or out-of-distribution frames where camera pose estimation fails entirely), while Cosmos generates coherent 3D navigation with substantial camera motion while maintaining visual quality and geometric consistency. Figure 22 demonstrates joystick-like control—moving forward, backward, rotating left and right—generating consistent videos from a single input image. Figure 23 shows that with different random seeds and the same input image and camera trajectory, the model generates diverse but equally 3D-consistent worlds, demonstrating that the learned world representation supports stochastic simulation of multiple possible futures.

Post-Trained WFM: Robotic Manipulation

Two distinct robotic tasks are evaluated.

Instruction-based video prediction (Cosmos-1X dataset, 23 test episodes, human evaluation). Figure 24 presents the head-to-head comparison against VideoLDM-Instruction. Cosmos-Predict1-7B-Video2World-Sample-Instruction is preferred in 78.3% of overall comparisons versus 13.0% for VideoLDM-Instruction (8.7% ties). Breaking down by evaluation dimension: instruction following (65.2% vs. 17.4%), object permanence (82.6% vs. 8.7%), verity—absence of imagined objects (73.9% vs. 13.0%), and overall reasonableness for robot planning (78.3% vs. 8.7%). The autoregressive variant (Cosmos-Predict1-5B-Video2World-Sample-Instruction) shows lower but still positive preference: 43.5% overall versus 30.4% for VideoLDM, with 26.1% ties. The qualitative results in Figure 25 show the model generating plausible robot motions for instructions like "organize books by placing them vertically on a shelf" and "fold a green fabric item on a table."

Action-based next-frame prediction (Bridge dataset, 100 test episodes). Table 23 compares Cosmos fine-tuned models against IRASim-Action. Cosmos-Predict1-7B-Video2World-Sample-ActionCond achieves PSNR 21.14, SSIM 0.82, Latent L2 0.32, and FVD 190. The 5B autoregressive variant achieves PSNR 19.95, SSIM 0.80, Latent L2 0.36, and FVD 434. The IRASim baseline achieves PSNR 19.13, SSIM 0.64, Latent L2 0.38, and FVD 593. The 7B diffusion model reduces FVD by 68% relative to IRASim (190 vs. 593), indicating substantially better temporal coherence. The SSIM improvement (0.82 vs. 0.64) shows better structural similarity to ground-truth frames, while the PSNR gain is more modest (2 dB), suggesting the improvements are concentrated in spatial structure rather than pixel-level color accuracy.

Post-Trained WFM: Autonomous Driving

The multi-view driving evaluation uses the RDS dataset with 800 trajectory-categorized samples for consistency metrics and 1000 samples for generation quality metrics.

Generation quality (Table 24). Cosmos-Predict1-7B-Text2World-Sample-MultiView achieves FID 32.16 and FVD 210.23, versus VideoLDM-MultiView's FID 60.84 and FVD 884.46. The FVD reduction is particularly dramatic (4.2× improvement), indicating that the Cosmos model's multi-view videos maintain temporal coherence far better than the baseline.

Multi-view consistency (Table 24). Temporal Sampson Error (TSE) measures per-view geometric consistency over time: the Cosmos Text2World model achieves 0.68, close to real videos (0.69) and half of VideoLDM-MultiView's 1.24. Cross-view Sampson Error (CSE) measures consistency between different camera views at the same time step: Cosmos achieves 2.11 versus 6.48 for VideoLDM-MultiView and 1.71 for real videos. Adding trajectory control (Cosmos-Predict1-7B-Text2World-Sample-MultiView-TrajectoryCond) further improves both metrics: TSE 0.59, CSE 2.02—the CSE approaches real-video quality despite being generated, and the trajectory conditioning provides explicit 3D guidance that improves multi-view alignment.

Trajectory consistency (Table 25). The Trajectory Agreement Error (TAE) measures whether trajectories estimated from different camera pairs agree, reflecting multi-view consistency. Cosmos-Predict1-7B-Text2World-Sample-MultiView-TrajectoryCond achieves ATE 0.54, RPE-R 4.31, RPE-t 0.18, approaching real video performance (ATE 0.49, RPE-R 4.60, RPE-t 0.14). The baseline VideoLDM-MultiView shows dramatically worse trajectory agreement: ATE 0.88, RPE-R 22.94, RPE-t 0.77—the rotational error in particular (22.94) indicates that different camera pairs produce wildly inconsistent motion estimates. The Trajectory Following Error (TFE) for the trajectory-conditioned model is 20.20 cm versus 13.49 cm for the ground-truth oracle—the generated videos follow the specified trajectory within approximately 7 cm of physical accuracy.

Object tracking consistency. On 20 generated 8-second videos containing 157 tracked objects, human annotators found zero cases of physically impossible tracking failures (e.g., two distinct objects incorrectly merging). This qualitative result supports the geometric consistency metrics with a more interpretable real-world benchmark: the generated videos are sufficiently 3D-consistent that standard object detection and tracking algorithms operate correctly without encountering physically implausible scenarios.

Autoregressive WFM: Failure Rate Analysis

Table 18 reports a systematic failure analysis on 100 Physical AI inputs. The primary failure mode is "objects unexpectedly appearing from below" (Figure 19). With single-frame (image) conditioning, the 4B model shows a 15% failure rate, improving to 7% for the 5B-Video2World, 2% for the 12B, and 3% for the 13B-Video2World. With 9-frame video conditioning, all models achieve failure rates of 2% or lower (the 12B and 13B-Video2World show 1% and 0%, respectively). The key finding is that video conditioning dramatically improves robustness: the model with 9 frames of context can infer object velocities and trajectories, preventing the spontaneous generation of objects from implausible locations. The larger autoregressive models (12B/13B) are more robust even with single-frame conditioning, suggesting that capacity helps with the harder inference problem of predicting motion from a static image.

Diffusion Decoder Effectiveness

Figure 18 provides qualitative evidence that the diffusion decoder (Cosmos-Predict1-7B-Decoder-DV8×16×16ToCV8×8×8-720p) substantially improves visual quality of autoregressive WFM outputs. The top row shows the raw autoregressive output (blurry, lacking fine texture); the bottom row shows the same content after diffusion decoding (sharp, with preserved object identity and motion). This is not evaluated quantitatively with metrics like PSNR against ground truth, but the visual difference is presented as self-evident. The diffusion decoder effectively compensates for the discrete tokenizer's aggressive compression by learning to map the coarse discrete representation to a higher-quality continuous latent space, leveraging the pre-trained diffusion model's strong image prior.

Real-Time Inference Performance

Tables 16 and 17 provide inference benchmarks. For the 5B autoregressive model on 640×1024 video with 32 generated frames, generation without the diffusion decoder takes 11.67 seconds on 8×H100 GPUs; with Medusa speculative decoding, this drops to 5.79 seconds (2× speedup) based on the 9-head configuration from Table 15 that was identified as optimal (982.77 tokens/second for the 5B model). With the diffusion decoder added, inference time increases substantially (to 24.35 seconds with Medusa), reflecting the additional denoising computation.

For real-time generation at 320×512 resolution (Table 17), the 4B model with Medusa achieves 806.61 tokens/second and 10.08 frames/second on 8×H100 GPUs—demonstrating real-time performance at 10 FPS for domain-specific low-resolution video.

Ablation Studies and Robustness Checks

Medusa head count for speculative decoding (Table 15): The number of Medusa heads (additional decoding heads for parallel token prediction) is swept across 0, 3, 6, 9, and 12. For the 4B model, token throughput increases from 444.95 tokens/s (0 heads) to 894.67 at 9 heads, then slightly decreases to 890.64 at 12 heads. The number of forward passes decreases monotonically from 7,680 (0 heads) to 1,682 (12 heads). The non-monotonicity in throughput at 12 heads (despite fewer forward passes) indicates that the computational overhead of processing 12 additional FFN heads outweighs the savings from reduced forward passes. For the 5B model, the optimal is also 9 heads, achieving 982.77 tokens/s and 1,799 forward passes. The 9-head configuration provides the best trade-off between parallel token prediction and per-iteration computational cost.

Selective layer freezing for Medusa fine-tuning (Section 5.2.4): The paper reports (without a dedicated table) that fine-tuning only the Medusa heads (keeping the backbone entirely frozen) results in poor multi-token prediction accuracy—the heads cannot learn to predict future tokens without adapting the backbone representations. Full fine-tuning causes quality degradation (catastrophic forgetting of the pre-trained video prediction capabilities). The optimal strategy is unfreezing the last two transformer layers and the final unembedding layer while keeping the remaining backbone frozen. This balances the Medusa heads' need for adapted representations against the risk of overwriting pre-trained knowledge.

Continuous vs. discrete tokenizer compression-quality tradeoff (Figure 8, Tables 5–8): The Cosmos tokenizer suite is evaluated at multiple compression rates against prior tokenizers. For continuous video tokenizers on DAVIS, Cosmos-Tokenize1-CV4×8×8-360p achieves PSNR 35.85, SSIM 0.920, rFVD 10.057, versus CogVideoX-Tokenizer4×8×8 at 29.29, 0.864, 19.58—a +6.56 dB PSNR improvement at equivalent compression. At higher compression (8×8×8), Cosmos-Tokenize1-CV8×8×8-720p achieves 31.28 dB, still exceeding CogVideoX at 4×8×8 (29.29 dB) despite 2× higher compression. For discrete tokenizers, Cosmos-Tokenize1-DV4×8×8-360p achieves 32.97 dB on DAVIS versus VideoGPT-Tokenizer4×4×4 at 28.17 dB and Omni-Tokenizer4×8×8 at 20.02 dB—the Omni-Tokenizer's poor performance (20.02 dB) indicates that its use of VQ-based quantization substantially degrades reconstruction compared to FSQ. The non-obvious finding is that Cosmos tokenizers at 2× higher compression still outperform prior tokenizers at lower compression, indicating that the architectural choices (causal design, wavelet preprocessing, LayerNorm, FSQ) collectively produce a more efficient compression scheme, not just a higher-quality one at equivalent rates.

TokenBench evaluation across diverse domains (Tables 5–6, Figure 10): The TokenBench dataset (500 videos spanning robotic manipulation, driving, egocentric, and web videos) tests generalization beyond standard benchmarks. Cosmos-Tokenize1-CV4×8×8-360p achieves PSNR 38.42, SSIM 0.950, rFVD 3.34 on TokenBench—consistently higher than on DAVIS (35.85 dB), indicating that TokenBench videos are easier to reconstruct, possibly due to less complex motion or higher original quality. The relative rankings between tokenizers are preserved across DAVIS and TokenBench, suggesting that the PSNR/SSIM metrics are robust to domain shift, though the absolute values differ.

Runtime performance across tokenizers (Table 9): Cosmos-0.1-Tokenizer-CV4×8×8 (105M parameters) processes each video frame in 34.8 ms on an A100, versus 414 ms for CogVideoX-Tokenizer (216M parameters)—a 12× speedup with half the parameters. For image tokenizers, Cosmos-0.1-Tokenizer-CI8×8 (77M parameters) processes a 1024×1024 image in 62.7 ms, versus 242 ms for FLUX-Tokenizer8×8 (84M parameters)—a 3.9× speedup. The discrete tokenizer Cosmos-0.1-Tokenizer-DI8×8 takes 64.2 ms versus 475 ms for LlamaGen-Tokenizer8×8—a 7.4× speedup. The speed advantages arise from the lightweight architecture (spatio-temporal factorized convolutions reduce FLOPs) and the elimination of expensive operations (no VQ codebook lookup, no KL divergence computation).

Shot detection algorithm comparison (Table 1): Four shot detection algorithms are evaluated on the ShotBench benchmark. TransNetV2 achieves the highest average F1 scores: 0.967 on BBC, 0.919 on RAI, 0.821 on SHOT, 0.726 on ClipShots. AutoShot achieves comparable performance on SHOT (0.834) and ClipShots (0.711) but lags on BBC (0.952) and RAI (0.906). The heuristic methods (PySceneDetect, Panda70M) perform substantially worse on challenging datasets: PySceneDetect achieves F1 0.477 on ClipShots, Panda70M 0.513. The key insight is that ClipShots (which contains complex shot transitions and visual effects) reveals the largest performance gap between learned and heuristic methods, motivating the selection of TransNetV2 for the diverse, heavily-edited videos in the training corpus.

Transcoding throughput optimization (Table 2): The transcoding configuration is ablated across hardware (H100 vs. L40S), codec (libx264 vs. h264_nvenc), batching (1 vs. 16 clips), and software (ffmpeg vs. PyNvideoCodec). The baseline (H100, libx264, no batching) achieves 0.0574 videos/second. Switching to L40S with its dedicated NVENC encoders and h264_nvenc codec with batch size 16 improves throughput to 0.1026 videos/second (1.8×). Replacing ffmpeg with PyNvideoCodec for video stream transcoding yields the largest single improvement: 0.3702 videos/second (3.6× over the batched ffmpeg baseline, 6.5× over the initial baseline). The finding is that ffmpeg underutilizes GPU hardware accelerators even when configured to use them; a purpose-built library (PyNvideoCodec) that directly interfaces with NVDEC/NVENC achieves substantially higher utilization.

VILA captioning throughput (Table 3): The VLM used for video annotation is benchmarked on a single H100 GPU. The PyTorch FP16 baseline achieves 0.21 clips/second. TensorRT-LLM with FP16 and batch size 1 improves to 0.40 clips/second (1.9×). Increasing batch size to 16 yields 1.09 clips/second (2.7× over batch-1). Quantizing to FP8 with batch size 16 achieves 1.96 clips/second (1.8× over FP16 at the same batch size, 9.3× over the PyTorch baseline). The 10× end-to-end speedup (from 0.21 to 1.96) is critical for captioning ~100M clips within the project's compute budget. The finding is that FP8 quantization with TensorRT-LLM provides near-lossless throughput improvement for VLM inference, consistent with trends in LLM serving.

Failure rate analysis for autoregressive models (Table 18): While presented in the main text, this serves as an ablation over conditioning mode (single image vs. 9-frame video) and model scale. The non-obvious finding is that the failure mode ("objects appearing from below") is not simply a function of model capacity—the 5B-Video2World model with 9-frame conditioning (2% failure rate) outperforms the 4B model with single-frame conditioning (15%), suggesting that conditioning quality can compensate for model size in reducing specific failure modes. However, the 12B model with single-frame conditioning achieves a 2% failure rate, indicating that larger models also learn better priors that reduce this failure mode even without multi-frame context.

Multiple seeds for physics alignment (Table 20): Results are averaged over 4 random seeds, providing a robustness check against the stochasticity of the diffusion/autoregressive sampling process. The paper does not report variance across seeds, which would indicate whether the WFM rollouts are consistent (low variance) or highly variable (high variance)—the latter being problematic for Physical AI applications where deterministic or near-deterministic predictions are often required.

Human evaluation for instruction-based robotics (Figure 24): The use of ten human evaluators provides a robustness check against individual annotator bias. The paper does not report inter-annotator agreement metrics (e.g., Krippendorff's alpha), which would quantify the reliability of the preference judgments. The 78.3% vs. 13.0% overall preference for the 7B Cosmos model versus VideoLDM is a large margin that is likely robust to annotator variability, but the 43.5% vs. 30.4% for the 5B Cosmos model (with 26.1% ties) is closer and would benefit from statistical significance testing that is not reported.

Trajectory-conditioned generation diversity (Figure 23): The camera control model is evaluated qualitatively with different random seeds given the same input image and camera trajectory. The generated videos show diverse but 3D-consistent futures (e.g., different room layouts, object placements, lighting conditions). This is not a quantitative ablation but demonstrates a capability relevant to downstream use: the model can simulate multiple possible worlds from the same initial state and control input, enabling uncertainty-aware planning.

Critical Assessment

The experiments in this paper support a specific, important claim: that pre-training large transformer models on diverse curated video data produces visual representations that transfer effectively to downstream Physical AI tasks when fine-tuned on small domain-specific datasets. The evidence for this is convergent across three distinct applications (camera control, robotics, autonomous driving), each with different conditioning modalities and evaluation protocols. In every case, a Cosmos pre-trained model fine-tuned on domain-specific data substantially outperforms a baseline model trained from scratch on the same fine-tuning data. The pattern is consistent enough to be convincing.

However, the paper's broader framing—that Cosmos WFMs are "world foundation models" that can serve as "digital twins of the physical world"—is not supported by the experiments presented. The critical gap is between visual quality and physical fidelity. The physics alignment evaluation (Section 5.3.2) is the paper's own attempt to measure this gap, and the results are sobering: even the largest models achieve object-level IoU of only 0.48–0.60 when predicting rigid-body dynamics, and the paper explicitly states that "all the WFMs equally struggle with physics adherence." This is not a minor limitation—it means that if a robot used a Cosmos WFM to predict the outcome of pouring a cup of coffee, the predicted video would look plausible but the coffee might flow in the wrong direction, splash incorrectly, or the cup might deform. For policy evaluation or training, this level of physical inaccuracy could lead to policies that work in simulation but fail catastrophically in reality.

The paper is candid about this limitation in the discussion (Section 9), but the framing of the models as "world foundation models" rather than "video prediction models" creates an expectation of physical fidelity that the experiments do not demonstrate. A more precise characterization of what the experiments actually show: Cosmos WFMs learn strong visual-physical priors (object appearance, common motion patterns, 3D scene geometry, multi-view consistency) from internet video that transfer to domain-specific fine-tuning, but they do not learn Newtonian physics, object permanence, or contact-rich dynamics at a level sufficient for reliable simulation. The distinction matters because downstream users need to know whether the model is suitable for tasks that require physical accuracy (manipulation, locomotion) versus tasks that primarily require visual plausibility (scene generation, data augmentation).

Specific experimental strengths:

1. Multi-dimensional evaluation beyond FID/FVD. The 3D consistency evaluation (Sampson error, pose estimation, view synthesis) and physics alignment evaluation (pixel/feature/object-level metrics) represent a genuine advance in how world models are assessed. These metrics probe capabilities that matter for downstream use and reveal limitations that standard metrics conceal. The finding that Cosmos achieves higher pose estimation success (62.6%) than real videos (56.4%) is both impressive and interpretable—the generated videos are geometrically regularized to be more 3D-consistent than real footage, which may actually be an advantage for simulation.

2. Scale ablations that reveal non-scaling phenomena. The physics alignment evaluation showing that 7B and 14B diffusion models achieve nearly identical object-level IoU (0.592 vs. 0.598) is a critical finding—it demonstrates that scaling model size alone does not improve physical understanding, which tells the field where to focus effort (data quality, training objectives, architectural priors). This negative result is more valuable than a positive scaling trend would be.

3. Systematic conditioning ablations. The comparison between 1-frame and 9-frame conditioning for physics alignment (IoU drops from 0.592 to 0.332) and for autoregressive failure rates (15% to 1% for the 4B model) quantifies the value of multi-frame context in a way that directly informs system design: Physical AI applications should provide as many conditioning frames as the compute budget allows.

4. Real-time inference demonstration. Showing that the autoregressive model with Medusa heads and low-resolution adaptation achieves 10 FPS generation (Table 17) validates a key claimed advantage of the autoregressive family—fast inference for interactive applications. This is not just a benchmark number; it demonstrates feasibility for real-world deployment where latency constraints matter.

Specific experimental weaknesses:

1. No pre-training data ablation. The paper does not train even a single smaller model on an uncurated or differently-curated version of the video dataset. This makes it impossible to attribute the transfer performance to the specific curation choices versus the sheer scale of video data. Would a model trained on 100M randomly sampled YouTube clips perform similarly after fine-tuning? The paper's curation pipeline is extensively described and motivated, but its causal contribution to downstream performance is untested. Given the compute cost (10,000 H100 GPUs over three months), running a scaled-down ablation (e.g., 1B-parameter models on 10M curated vs. 10M random clips) would have been feasible and informative.

2. No post-training data scale sweep. The post-training experiments use fixed dataset sizes (DL3DV-10K for camera control, ~12K episodes for Cosmos-1X, ~20K episodes for Bridge, ~3.6M clips for RDS). There is no sweep over how much fine-tuning data is needed to achieve the reported performance, which is the central practical question for the pre-training-then-post-training paradigm: what is the data efficiency gain from pre-training? Does the Cosmos model achieve the same performance with 10% of the fine-tuning data that a from-scratch model needs? Answering this would require fine-tuning both Cosmos and the baseline on 1%, 10%, 50%, and 100% of the domain-specific data and comparing performance curves.

3. Limited baseline strength. While VideoLDM, CamCo, and IRASim are reasonable baselines, they are not the only or necessarily the strongest alternatives. For robotics specifically, recent work on video pre-training for control (GR-2, GR-1, UniPi) is not compared against. The baselines are also trained from scratch on the fine-tuning data only—a stronger baseline would be a model pre-trained on a different large-scale video dataset (e.g., something trained on WebVid-10M or a subset of the Cosmos data) and then fine-tuned identically, to isolate the effect of Cosmos's specific pre-training scale and curation.

4. No statistical rigor for key claims. The 3D consistency evaluation uses 500 videos; the physics alignment evaluation uses 800 videos (but only 100 per scenario); the human evaluation uses only 23 test episodes with 10 evaluators. Confidence intervals, standard deviations, or statistical significance tests are not reported for any main result. For the camera control evaluation where Cosmos achieves 82.0% success vs. 43.0% for CamCo, this gap is large enough to likely be significant even with 500 samples. But for the instruction-based robotics comparison where Cosmos-5B achieves 43.5% preference vs. 30.4% with 26.1% ties (Figure 24b), the margin is narrow and the sample size small—statistical testing would clarify whether this is a reliable improvement or within noise.

5. Single benchmark per task. Each post-training application is evaluated on one test set: RealEstate10K for camera control, Cosmos-1X for instruction-based robotics, Bridge for action-based robotics, RDS for autonomous driving. There is no cross-domain evaluation to test whether the pre-trained representation generalizes beyond the fine-tuning domain, which is the hallmark of effective pre-training. Does the camera control model trained on DL3DV-10K transfer to indoor scenes or outdoor driving? Does the robotics model trained on EVE transfer to a different robot embodiment? These are the questions that would distinguish a foundation model from a model that happens to fine-tune well.

6. No comparison between diffusion and autoregressive families on transfer. Both families are post-trained for robotics, but the paper does not systematically compare 7B diffusion vs. 5B autoregressive on the same fine-tuning tasks with the same metrics. The robotics results (Section 6.2) show that the 7B diffusion model outperforms the 5B autoregressive model (78.3% vs. 43.5% overall preference for instruction following; PSNR 21.14 vs. 19.95 for action prediction), but these comparisons conflate model architecture and parameter count. A comparison of diffusion-7B to autoregressive-7B (or diffusion-12B to autoregressive-13B) on the same task with the same fine-tuning data would clarify whether the performance difference is due to the architecture or the scale.

7. The diffusion decoder is not quantitatively evaluated. Figure 18 provides qualitative evidence that the diffusion decoder improves autoregressive output quality, but no PSNR/SSIM/FVD numbers are reported. The paper claims the decoder "can enhance details while preserving the content" but does not measure whether it sometimes hallucinates details not present in the autoregressive output or alters the motion dynamics. A quantitative evaluation comparing autoregressive output with and without the diffusion decoder against ground-truth videos would be straightforward and informative.

8. Missing failure mode analysis for diffusion models. The paper reports a systematic failure analysis for autoregressive models (Table 18, Figure 19) but provides only anecdotal failure discussion for diffusion models (Section 9 mentions "lack of object permanence, inaccuracies in contact-rich dynamics, and inconsistency in instruction following"). A parallel systematic study on diffusion model failure rates and modes would enable fair comparison between the two families and better inform downstream users about which model to choose for which application.

9. Physics alignment evaluation has a sim-to-real gap. The physics scenarios are rendered using PhysX and Isaac Sim with Omniverse assets—they are synthetic, not photorealistic. The Cosmos WFMs are pre-trained on real videos and may underperform on synthetic inputs due to domain shift. The paper acknowledges this ("enhancing photorealism to bridge the sim-to-real gap") but does not quantify how much of the reported physics alignment error is due to the WFM being unable to simulate physics versus being unable to process synthetic visual inputs. A control experiment using a WFM fine-tuned on synthetic data would disentangle these factors.

10. The 3D consistency evaluation is restricted to static scenes. The RealEstate10K dataset contains only static scenes (the videos are captioned as such), so the 3D consistency metrics measure whether the WFM can maintain geometric consistency when the only source of change is camera motion. This is an important capability but does not test whether objects moving within the scene (cars, people, robots) remain 3D-consistent—a harder problem that is more relevant to Physical AI. Evaluating 3D consistency of dynamic objects would require ground-truth 3D information (depth maps, object meshes) that RealEstate10K does not provide.

Experiments that would have strengthened the paper:

• A data curation ablation study comparing WFM pre-training performance with curated vs. uncurated data at smaller scale (e.g., 1B parameters, 10M clips).
• A fine-tuning data scale sweep (1%, 5%, 25%, 100% of domain-specific data) for at least one downstream task, comparing Cosmos-pretrained vs. from-scratch models, to quantify the data efficiency gain.
• Cross-domain transfer evaluation: test the camera control model on driving data, test the robotics model on a different robot, etc.
• Systematic comparison of diffusion vs. autoregressive models at matched parameter counts on the same downstream task with the same fine-tuning protocol.
• Statistical significance testing for the human evaluation results, including inter-annotator agreement metrics.
• Quantitative evaluation of the diffusion decoder (PSNR/SSIM with and without decoder, measurement of content preservation vs. hallucination).
• Physics alignment evaluation with a model fine-tuned on synthetic data to control for the sim-to-real domain gap.
• Failure mode analysis for diffusion models parallel to Table 18 for autoregressive models.

Summary of what the experiments do and do not demonstrate:

The experiments demonstrate that pre-training large transformer-based video prediction models on 100M curated video clips produces models that (1) generate videos with strong 3D consistency and multi-view geometric coherence, (2) transfer effectively to domain-specific Physical AI tasks when fine-tuned on thousands to millions of domain-specific examples, outperforming from-scratch baselines, and (3) can be optimized for real-time inference (10 FPS) using speculative decoding and resolution adaptation for autoregressive variants. These are substantial and well-supported claims.

The experiments do not demonstrate that these models understand or accurately simulate physics (the physics alignment results show fundamental limitations that do not improve with scale), that the specific data curation choices are responsible for the transfer performance (no ablation exists), that the pre-training provides data efficiency gains for fine-tuning (no fine-tuning data scale sweep exists), or that the models generalize across domains (no cross-domain transfer evaluation exists). The "world foundation model" framing implies capabilities that outstrip what the experiments verify, and the paper's honest discussion of limitations (Section 9) is in some tension with the aspirational framing of the introduction and abstract. The contribution is best understood as a platform and pre-training recipe that produces strong visual-physical priors, enabling efficient specialization to downstream tasks, rather than as a solved world simulation problem.

6. Limitations and Trade-offs

6.1 Pre-Training Scale Prevents Direct Causal Attribution of Curation Choices

The assumption or constraint. The Cosmos platform invests heavily in a multi-stage video curation pipeline—shot detection via TransNetV2, motion filtering, visual quality filtering, text overlay removal, video type classification, semantic deduplication removing ~30% of data, and category rebalancing (Section 3). The paper argues these choices improve the physical information density of the training data. However, the pre-trained WFMs are all trained on the full curated dataset using 10,000 H100 GPUs over three months (Section 5). No ablation exists—not even at smaller scale—that compares a model trained on curated versus uncurated data. Nor is there an ablation isolating individual curation steps (e.g., motion filtering on/off, deduplication on/off).

The consequence. The paper cannot distinguish between two competing explanations for the strong transfer performance: (1) the specific curation choices are responsible, or (2) any sufficiently large and diverse video dataset would produce similar results. This matters immensely for practitioners. If the curation pipeline is essential, adopting Cosmos requires replicating a complex, expensive infrastructure pipeline (GPU-accelerated transcoding, VLM-based annotation with TensorRT-LLM optimization, semantic deduplication at scale). If it is not essential, practitioners could train on much larger but uncurated datasets and potentially achieve better results at lower engineering cost. The paper's extensive documentation of the curation pipeline (the longest section in the paper) implicitly argues for its importance but provides no causal evidence.

What evidence exists in the paper. None. Section 3 describes what the pipeline does and why each step is motivated (e.g., "text tends to co-occur with different visual effects" in Section 3.3.3, so text overlay videos should be removed), but there is no experiment showing that a model trained with text overlay filtering outperforms one trained without it. The downstream post-training results (Sections 6.1–6.3) demonstrate that the pre-trained models transfer well, but these results confound curation, scale, architecture, and training recipe into a single package. The paper's only implicit evidence for curation importance is the rebalancing toward physical interaction categories (driving 11%, hand manipulation 16%, etc.—Section 3.1), which reflects a hypothesis about what data distribution is useful, but this hypothesis is not tested.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation. The compute cost of running even a scaled-down ablation (e.g., training 1B-parameter models on 10M curated vs. 10M randomly-sampled clips) would be substantial but feasible, and would have addressed one of the most important open questions about the platform: is the curation machinery a necessary component or an expensive comfort blanket?

---

6.2 Physics Alignment Does Not Improve With Model Scale — and Fails on Basic Rigid-Body Dynamics

The assumption or constraint. The Cosmos platform positions its pre-trained models as "world foundation models" that capture "general knowledge of real-world physics and natural behaviors" (Section 5). The implicit promise is that these models can serve as simulators for Physical AI systems—predicting what will happen when a robot executes an action or a vehicle follows a trajectory. This requires that the generated videos obey physical laws at a level sufficient for downstream decision-making.

The consequence. The physics alignment evaluation (Section 5.3.2, Table 20) reveals that the WFMs fail to accurately predict even simple rigid-body dynamics. On the controlled simulation benchmark (free-falling objects, domino chains, seesaws, gyroscopes), the object-level Intersection-over-Union between predicted and ground-truth object masks is only 0.48–0.60 across all model variants. Critically, scaling from 7B to 14B parameters in the diffusion family (IoU 0.592 → 0.598) or from 4B to 12B in the autoregressive family (IoU 0.481 → 0.487) produces essentially no improvement. The paper is explicit about this:

"While we observe larger models render videos with higher visual quality, all the WFMs equally struggle with physics adherence and require better data curation and model design." (Section 5.3.2)

This is a capability ceiling, not a minor inaccuracy. If a robot uses a Cosmos WFM to predict the outcome of pouring liquid, stacking objects, or applying force to a mechanism, the predicted video will look photorealistic but will likely contain physically impossible object trajectories, shape deformations, or object appearances/disappearances. A policy trained or evaluated against such a simulator would learn behaviors optimized for a physically inconsistent world—behaviors that may fail catastrophically when deployed on real hardware.

The failure modes are not subtle. The paper documents "low-level issues like object impermanence (spontaneous appearance and disappearance of objects) and deformation (shape changes) to more complex problems such as implausible kinematics, violation of gravity, etc." (Section 5.3.2). For the autoregressive family, a systematic failure analysis (Table 18) finds that 2–15% of generated videos exhibit "objects unexpectedly appearing from below" depending on model size and conditioning mode. These are not edge-case failures—they are systematic violations of basic physical principles that any useful world simulator must respect.

What evidence exists in the paper. Section 5.3.2 provides the primary evidence: 800 synthetic videos across 8 physics scenarios, evaluated with pixel-level (PSNR 17–21 dB), feature-level (DreamSim ~0.86), and object-level (IoU 0.48–0.60) metrics. The IoU metric is the most diagnostic because it isolates object motion from background quality. An IoU of 0.60 means that on average, predicted object masks overlap with ground-truth masks by only 60%—a substantial deviation. Additionally, the qualitative failure mode descriptions confirm that the errors are physically meaningful, not just pixel-level blur. The failure rate analysis for autoregressive models (Table 18) quantifies one specific failure mode (objects appearing from below) that directly violates object permanence.

Mitigation status. The paper partially acknowledges this limitation in Section 9:

"Current models, including ours, fall short as reliable simulators of the physical world. We observe that our models still suffer from issues, including the lack of object permanence, inaccuracies in contact-rich dynamics, and inconsistency in instruction following."

However, the paper does not propose concrete mitigations beyond vague directions: "additional steps in data curation where physically implausible videos are removed are required, as well as improved model design" (Section 5.3.2). The finding that scale does not help (7B ≈ 14B) is not explored further—the paper does not investigate whether architectural changes (e.g., explicit physics priors, differentiable simulation layers) or training objective modifications (e.g., physics-based auxiliary losses) could address the gap. The limitation is acknowledged but not diagnosed.

---

6.3 The $14\times$ Larger Model Baseline Is Not Compute-Optimal

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge that this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters are scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This matters because a Chinchilla-optimal model trained with $14\times$ more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it needs to be. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at $R \ll 1$) may shrink or reverse against a properly compute-optimal larger model. Additionally, the $14\times$ larger model uses only greedy decoding—no majority voting, no best-of-N, no search. Giving the larger model even a modest test-time compute budget (say, best-of-8) would create a much stronger baseline that is never tested.

---

6.4 Verifier Over-Optimization Is a Hard Ceiling, Not a Solved Problem

The assumption or constraint. The paper documents verifier over-optimization as a central limiting factor: beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search—the strongest optimizer—paradoxically performs worst overall (Figure 3, left), and qualitative examples show degenerate outputs (repetitive steps, overly short solutions; Appendix M, Figures 29, etc.). The compute-optimal policy mitigates this by routing easy problems away from aggressive search, but it does not solve the underlying problem. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling—the beam search curves in Figure 3 flatten and sometimes decline well before the budget is exhausted.

The consequence. This means the compute-optimal approach is fundamentally bounded by verifier quality. Improving the PRM (e.g., through better training data, adversarial robustness, or ensemble methods) would likely shift the difficulty thresholds and change the optimal policy. The current results are therefore specific to the verifier quality achievable with the Monte Carlo rollout training procedure described in Appendix D. The paper does not explore how verifier improvements would alter the scaling landscape.

---

6.5 Hard Problems Remain Essentially Unsolved

The assumption or constraint. Across all methods—search, revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%.

The consequence. This is a fundamental limitation: test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help—there are no correct solutions in the proposal distribution to find or refine. The paper is candid about this (Section 7 takeaway box), but it means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path.

---

6.6 Revisions and Search Are Studied Independently, Not Combined

The assumption or constraint. The paper studies two complementary axes—PRM search and iterative revisions—but never combines them. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. This is a significant gap because the two mechanisms have complementary strengths: revisions improve the proposal distribution (generating better candidates), while PRM search improves candidate selection (finding the best among generated candidates). Applying beam search to revision model outputs—or using the PRM to guide which revisions to pursue—could yield gains beyond either method alone. The current results therefore represent a lower bound on what a fully integrated system could achieve.

7. Implications and Future Directions

How This Work Changes the Landscape

The Cosmos platform introduces a structural shift in how the Physical AI community thinks about world simulation: from specialist models trained from scratch per environment to generalist foundation models pre-trained at scale and adapted via fine-tuning. This is not an incremental refinement of existing world model architectures—it is a category change that imports the foundation model paradigm from language and vision into a domain where no such generalist pre-training had been demonstrated at scale.

The magnitude of the shift is best understood by examining what the paper makes newly tractable. Prior to Cosmos, a robotics lab wanting a world model for a specific manipulator and task would collect domain-specific video-action data (thousands of episodes), train a model from scratch on that data alone, and hope it generalized within the narrow training distribution. The resulting model would know nothing about object permanence beyond what it observed in those specific episodes, nothing about how liquids or fabrics behave unless those phenomena appeared in the training set, and nothing about 3D scene geometry beyond the limited viewpoints in its data. With Cosmos, that same lab can start from a pre-trained 7B Video2World model that has seen approximately 100M video clips spanning driving, manipulation, human activity, and nature dynamics—a visual education orders of magnitude broader than any single-lab dataset—and then fine-tune on their specific robot data. The paper's evidence that this works (78.3% human preference for Cosmos fine-tuned vs. 13.0% for from-scratch VideoLDM on instruction-based robotics; 4.2× FVD improvement for multi-view driving) suggests that the pre-training transfers non-trivial physical knowledge.

This reframes the resource allocation problem for Physical AI. Previously, progress was bottlenecked by the difficulty and expense of collecting interleaved observation-action sequences from real hardware—dangerous, slow, and inherently limited in diversity. The Cosmos paradigm suggests that a large fraction of the needed visual-physical knowledge can be acquired from readily available internet video during pre-training, with only a comparatively small amount of domain-specific interaction data needed for specialization. The paper does not quantify the data efficiency gain (no fine-tuning data scale sweep exists), but the consistent outperformance of Cosmos fine-tuned models over from-scratch baselines on identical fine-tuning data implies that the pre-trained model is extracting more value from each domain-specific example.

The paper also resolves a latent contradiction in the video generation literature. Text-to-video models like Sora, Dream Machine, and Kling produce visually stunning outputs, but their utility for Physical AI has been unclear: they lack action conditioning, multi-view consistency, and physical fidelity. Meanwhile, learned world models (Dreamer, TD-MPC, IRASim) provide action-conditioned simulation and have demonstrated utility for control, but they operate on low-dimensional latent states and cannot generate photorealistic video that a human could evaluate or that could serve as visual training data. The Cosmos platform bridges this gap by demonstrating that the same architecture family (transformer-based diffusion models) can be pre-trained for general visual quality at text-to-video scale (7B–14B parameters, ~100M clips) and then post-trained for action-conditioned, multi-view, physically-grounded simulation. The two capabilities—visual fidelity and physical controllability—are not in tension; they can be unified through the pre-training-then-post-training paradigm.

Which research directions become more attractive:

The physics alignment evaluation (Section 5.3.2) is the paper's most important diagnostic negative result. It shows that even 14B-parameter models trained on 100M video clips achieve object-level IoU of only ~0.60 when predicting rigid-body dynamics, and that scaling from 7B to 14B produces essentially no improvement. This result redirects attention from model scale to data quality and training objectives for improving physical fidelity. The implication is clear: spending more compute on larger models trained on the same internet video distribution will not produce Newtonian physics understanding. The field needs better data (physically accurate simulation data, filtered to remove physically implausible internet videos) or better architectures (explicit physics priors, differentiable simulation layers, physically-motivated auxiliary losses).

The strong 3D consistency results (62.6% pose estimation success vs. 4.4% for VideoLDM) in contrast suggest that geometric reasoning about static scenes is a problem that scale and data diversity can solve. This directs attention to dynamic 3D consistency—maintaining geometric coherence for moving objects—as the next frontier, where the paper provides no evaluation.

The autoregressive-vs-diffusion tension (Section 9) frames a productive debate: diffusion models currently produce better visual quality and transfer more effectively to downstream control (the 7B diffusion model outperforms the 5B autoregressive model on both instruction-based and action-conditioned robotics), but autoregressive models offer faster inference (10 FPS real-time with Medusa) and the potential to inherit pre-trained LLM weights. This makes hybrid architectures an attractive research direction—models that combine diffusion-based generation quality with autoregressive inference efficiency, possibly through distillation as suggested by Yin et al. (2024).

Finally, the paper's extensive documentation of data curation makes data engineering for physical understanding a first-class research topic. The curation pipeline's principles—target physical interaction categories, remove editing artifacts that create spurious correlations, filter for motion that teaches dynamics, deduplicate to avoid wasting compute on redundant examples—constitute a research contribution in themselves, even if their causal importance is not ablated.

---

Follow-Up Research This Work Enables

Quantifying the data efficiency gain from pre-training through fine-tuning data scale sweeps. The paper demonstrates that Cosmos pre-trained models fine-tuned on domain-specific data outperform from-scratch baselines, but it does not measure how much fine-tuning data the pre-trained model needs to match a given from-scratch performance level. A strong follow-up would fine-tune both Cosmos-Predict1-7B-Video2World and a from-scratch baseline on 1%, 5%, 10%, 25%, 50%, and 100% of the DL3DV-10K (camera control), Cosmos-1X (instruction-based robotics), and Bridge (action-conditioned robotics) datasets. The key metric is the data multiplier: if the Cosmos model achieves the same PSNR/FVD with 10% of the fine-tuning data that the from-scratch model needs 100% to achieve, that quantifies the pre-training's practical value. This sweep would also reveal whether pre-training benefits saturate—does the gap between Cosmos and from-scratch shrink as fine-tuning data increases, or does it persist? The paper's current results cannot distinguish between "pre-training helps a lot with limited data" and "pre-training provides a constant additive improvement regardless of data scale."

Physics-aware data curation through simulation-based pre-training. The physics alignment evaluation (Section 5.3.2) shows that WFMs trained on internet video do not learn Newtonian mechanics, and that scaling model size does not help. The natural follow-up is to modify the pre-training data to include physically accurate simulation data. A concrete experiment: take the Cosmos data curation pipeline and augment it with $N$ synthetic videos from PhysX/Isaac Sim covering the 8 physics scenarios from Section 5.3.2, rendered with domain randomization to bridge the sim-to-real visual gap. Pre-train smaller models (e.g., 1B parameters) on the original data plus 0%, 10%, or 50% synthetic physics data, then evaluate on the physics alignment benchmark. The hypothesis is that synthetic physics data provides a supervisory signal for object dynamics that internet video lacks. A positive result would show IoU improving from ~0.60 toward the ground-truth ceiling; a negative result (synthetic data doesn't transfer) would indicate that the sim-to-real domain gap in visual appearance prevents the model from leveraging physically accurate dynamics data, suggesting that better photorealism in simulators is a prerequisite.

Combining autoregressive and diffusion WFMs through distillation or hybrid decoding. The paper identifies a tradeoff: diffusion models produce higher quality but are slower; autoregressive models are faster but produce lower quality. A concrete follow-up would distill the 14B diffusion Video2World model into the 13B autoregressive Video2World model using the approach from Yin et al. (2024): train the autoregressive model to match the diffusion model's output distribution on a large corpus of generated videos, potentially with a diffusion head for refinement. The evaluation would compare the distilled model against both parents on (a) 3D consistency metrics (Sampson error, pose estimation success rate), (b) video quality (FVD, PSNR vs. ground truth on the physics alignment benchmark), (c) inference throughput (tokens/second and frames/second), and (d) downstream task performance after fine-tuning (camera control rotation error, action-conditioned robotics SSIM). The key question is whether distillation can preserve the diffusion model's quality while achieving near-autoregressive inference speed. If successful, this would produce a model family that dominates both parents on the Pareto frontier of quality vs. speed.

Multi-view dynamic 3D consistency evaluation for moving objects. The paper's 3D consistency evaluation (Section 5.3.1) is restricted to static scenes—camera motion only, no object motion. Physical AI applications require consistency for dynamic objects: a robot moving its arm, a vehicle changing lanes, a person walking. A follow-up could extend the evaluation framework to dynamic scenes by synthesizing videos where both camera and objects move, with known 3D object trajectories (from simulation ground truth). For each generated video, the evaluation would compute per-object Sampson error—measuring whether the pixels corresponding to a moving object satisfy epipolar geometry constraints when the background motion is factored out—and compare to the static-scene baseline. This would quantify how much 3D consistency degrades when objects move, which is currently unknown. A negative result (dynamic 3D consistency is much worse than static) would indicate that current WFMs achieve geometric coherence primarily by learning camera motion priors, not by building internal 3D representations of object geometry—a critical diagnostic for whether these models are "world models" or "camera motion models."

Post-training with pre-trained LLM weights for autoregressive WFMs. The paper identifies the ability to leverage pre-trained LLM weights as a key potential advantage of autoregressive WFMs, but does not explore it: the autoregressive models are trained from scratch. A concrete follow-up would initialize the autoregressive WFM backbone (Cosmos-Predict1-4B or -12B) from a pre-trained LLM of matching architecture (e.g., Llama-3 or Mistral), then train on video tokens with the same multi-stage curriculum described in Section 5.2.3. The comparison would be against the from-scratch models on (a) training convergence speed (loss curves, wall-clock time to reach a given video prediction accuracy), (b) final video quality (PSNR, FVD on held-out video data), (c) 3D consistency metrics, and (d) downstream transfer after post-training on robotics or driving tasks. If LLM initialization provides a meaningful improvement (faster convergence, better final quality, or stronger transfer), it would validate the architectural choice and potentially make autoregressive WFMs competitive with diffusion WFMs on quality while retaining their inference speed advantage. If it provides no benefit (the LLM's textual world knowledge does not transfer to video prediction), it would reframe the value proposition of autoregressive WFMs as primarily about inference optimization rather than knowledge transfer.

Fine-tuning for policy learning with WFM-in-the-loop evaluation. The paper lists policy evaluation, initialization, training, and planning as intended use cases (Section 2.1) but provides no empirical results on any of them. A strong follow-up would close the loop: take a pre-trained and post-trained WFM (e.g., Cosmos-Predict1-7B-Video2World-Sample-ActionCond fine-tuned on Bridge), use it to train a visuomotor policy via model-based reinforcement learning or planning, and compare the resulting policy's real-world success rate against a policy trained with (a) the real environment only, (b) a from-scratch world model (IRASim), and (c) no world model (behavioral cloning on the same data). The key metric is the sim-to-real transfer gap: does a policy trained in the WFM simulator succeed on real hardware? The paper's current results show that the WFM predicts plausible-looking video, but if policies trained against it fail on real hardware because the WFM's physics errors cause the policy to learn behaviors that exploit simulator artifacts, that would be a critical negative result bounding the WFM's current utility. If policies trained with the WFM outperform behavioral cloning baselines (because the WFM provides unlimited synthetic rollouts for exploration), that would validate the paper's motivating use case.

---

Practical Applications and Downstream Use Cases

Augmenting rare-scenario training data for autonomous driving. The multi-view trajectory-conditioned WFM (Cosmos-Predict1-7B-Text2World-Sample-MultiView-TrajectoryCond) generates 6-view driving videos that follow a specified 3D trajectory with a Trajectory Following Error of 20.20 cm (vs. 13.49 cm ground-truth oracle, Table 25). This capability directly addresses the long-tail problem in autonomous driving: rare and dangerous scenarios (children running into the street, sudden cut-ins, extreme weather) occur too infrequently in real-world driving logs to provide sufficient training data. A deployment scenario: a self-driving company identifies a set of high-risk scenarios from accident reports (e.g., "vehicle runs red light from left at intersection during rain at night"), generates thousands of trajectory-video pairs covering these scenarios using the WFM conditioned on appropriate text descriptions and ego trajectories, and augments their perception and planning training data with these synthetic examples. The key enabling numbers: FVD 210.23 vs. 884.46 for the VideoLDM baseline (Table 24) meaning the generated videos maintain sufficient temporal coherence for training, and Cross-view Sampson Error of 2.02 vs. 1.71 for real videos (Table 24) meaning the 6-view consistency is near-real quality. The 20.20 cm trajectory error means the generated vehicle path is accurate to within roughly the width of a tire, sufficient for training reactive planning policies even if not for precision maneuvers.

Pre-deployment policy screening for robotic manipulation. The instruction-based WFM (Cosmos-Predict1-7B-Video2World-Sample-Instruction) achieves 78.3% overall human preference when evaluated on whether generated videos are "reasonable for the robot to plan accordingly" (Figure 24). This enables a practical deployment workflow: before executing a new manipulation policy on real hardware, generate predicted video rollouts for a set of test instructions, have a human operator (or an automated VLM-based evaluator) inspect the videos for obvious failures (collisions, missed grasps, object drops), and only proceed to hardware testing for policies whose predicted rollouts look successful. This is not a replacement for real-world testing, but it can filter out obviously incapable policies quickly, reducing the number of hardware trials needed and—critically—reducing the risk of hardware damage from policies that would produce catastrophic failures. The 82.6% object permanence score (vs. 8.7% for VideoLDM) means the WFM does not typically hallucinate or drop objects, making it reliable enough for this screening use case. The cost is generating a few dozen video rollouts per policy (seconds of GPU time on H100s) versus the hours of supervised robot time and risk of damage that a hardware trial entails.

Camera trajectory planning for 3D scene exploration. The camera control WFM (Cosmos-Predict1-7B-Video2World-Sample-CameraCond) achieves 82.0% camera pose estimation success rate and 1.646° rotation error (vs. 8.277° for CamCo, Table 22), and can generate diverse 3D-consistent futures from the same input image and camera trajectory (Figure 23). This enables a planner that, given a single reference image of a scene, simulates multiple possible camera trajectories, evaluates the information gain along each trajectory (e.g., how much of the scene becomes visible, whether specific objects of interest are detected), and selects the trajectory that maximizes exploration efficiency. A drone or mobile robot equipped with this capability could plan viewing paths before moving, reducing unnecessary navigation. The 3D consistency numbers matter here because trajectory evaluation requires that the simulated views accurately reflect what the camera would actually see from those poses—a model with poor geometric consistency (like CamCo, at 43.0% pose estimation success) would produce simulated views that don't correspond to any real viewpoint, leading the planner to select trajectories that look informative in simulation but reveal nothing in reality.

---

When to Prefer This Method

The paper explicitly positions diffusion and autoregressive WFMs as having complementary strengths rather than one universally dominating the other (Section 9). The decision rule based on the paper's evidence is:

• Prefer diffusion-based WFMs (Cosmos-Predict1-7B/14B-Video2World) when output visual quality, 3D consistency, and transfer to downstream control tasks are the primary concerns, and inference latency is secondary. Evidence: the diffusion 7B model achieves 62.6% pose estimation success rate vs. 35.6% for the autoregressive 4B model (Table 19), PSNR 21.14 vs. 19.95 for action-conditioned robotics (Table 23), and 78.3% vs. 43.5% overall human preference for instruction-based robotics (Figure 24). The diffusion family also supports finer-grained control signals (camera poses concatenated as Plücker embeddings, action vectors injected into DiT timestep conditioning).

• Prefer autoregressive-based WFMs (Cosmos-Predict1-4B/5B-Video2World) when real-time or near-real-time inference is required (e.g., interactive planning, closed-loop control with tight latency budgets) and some degradation in visual quality is acceptable. Evidence: the 5B model with Medusa heads achieves 10 FPS generation at 320×512 resolution (Table 17) vs. diffusion models requiring 50 denoising steps per video. The autoregressive architecture also enables leveraging LLM inference optimizations (KV caching, speculative decoding, continuous batching) that are not directly applicable to diffusion sampling.

• Prefer post-training a Cosmos pre-trained WFM rather than training a world model from scratch when the available domain-specific data is limited (thousands to tens of thousands of episodes, not millions). The paper demonstrates this across all three post-training domains (DL3DV-10K, Cosmos-1X with ~12K episodes, Bridge with ~20K episodes, RDS with ~3.6M clips), but does not quantify the exact data scale at which from-scratch training catches up. The consistent outperformance of Cosmos fine-tuned models over from-scratch baselines on identical fine-tuning data supports this recommendation, but the absence of a data scale sweep means the precise break-even point is unknown.

• Prefer traditional physics simulators (MuJoCo, Isaac Sim, PhysX) over learned WFMs when the application requires strict physical accuracy for safety-critical predictions (e.g., robot locomotion stability, grasping force estimation, collision dynamics). The physics alignment evaluation (Section 5.3.2) shows that even the largest WFMs predict object trajectories with IoU of only ~0.60 and exhibit violations of object permanence, gravity, and rigid-body kinematics. For applications where a 40% error in predicted object position would cause hardware damage or safety violations, learned WFMs are not yet reliable substitutes for classical simulators. The WFMs are more appropriate for applications where visual plausibility and approximate dynamics are sufficient—data augmentation, policy screening, scene exploration planning—rather than fine-grained physical prediction.