Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling

ArXiv: 2512.23162

🎯 Pitch

Synthetic surgical videos can meaningfully replace scarce real robot demonstrations for training autonomous policies—but only when the generative world model has been specifically fine-tuned on surgical data. This paper shows that by training an inverse-dynamics model to extract pseudo-kinematics from AI-generated surgical footage, a robot policy trained on this synthetic data significantly outperforms one trained on real data alone.

---

1. Executive Summary

This paper introduces Cosmos-H-Surgical, a framework that connects surgical world modeling with robot policy learning to address the severe data scarcity bottleneck in autonomous surgical robotics. The authors curate the Surgical Action–Text Alignment (SATA) dataset—2,447 expert-annotated video clips across four fundamental surgical actions—and use it to fine-tune a diffusion-based world model (Cosmos-Predict2.5) capable of generating photorealistic, text-controllable surgical videos. They then train an inverse dynamics model (IDM) to infer pseudo-kinematics from these synthetic videos (predicting 20-dimensional robot actions from 16-frame video segments), producing paired video–action data that augments real demonstrations for training a surgical VLA policy. On a real surgical robot platform performing needle pick-up and hand-over tasks, a GR00T N1.5 policy trained with synthetic data achieves substantially lower trajectory prediction error than policies trained on real data alone, with the gap widening as more synthetic rollouts are added (Real + Synthetic 10× outperforms Real + Synthetic across all data regimes), establishing that world-model-generated synthetic data can meaningfully substitute for scarce real demonstrations—though only when the base model has been domain-adapted through surgical video pretraining rather than applied zero-shot.

2. Context and Motivation

The Core Problem: Surgical Robotics Suffers from a Fundamentally Different Data Scarcity than General Robotics

The central problem this paper addresses is that autonomous surgical robots cannot benefit from the same data scaling that has driven progress in general-purpose robotic manipulation. While household and industrial robotics have seen remarkable advances through large vision-language-action (VLA) models like RT-2, OpenVLA, and GR00T N1—models trained on massive, diverse datasets coupling visual observations, language descriptions, and robot actions—surgical robotics faces a qualitatively different data bottleneck that is not simply about quantity but about fundamental accessibility.

The paper identifies this bottleneck with precision in Section 1:

"A major bottleneck is the lack of large, diverse datasets that include both high-fidelity visual observations (e.g., endoscopic video) and synchronized robot kinematics or control commands. Collecting such paired demonstrations is prohibitively expensive, constrained by operating room access, patient safety, and regulatory hurdles."

This is not just "data is scarce"—it is a structural constraint. In general robotics, you can set up a lab, install cameras, and have a human teleoperate a robot arm picking up objects for days or weeks, generating thousands of paired video-action demonstrations at manageable cost. In surgery, every demonstration requires: (1) a surgical robot system costing hundreds of thousands to millions of dollars, (2) an operating room with sterile conditions, (3) a patient undergoing an actual procedure, (4) a surgeon whose primary obligation is patient care, not data collection, and (5) compliance with privacy regulations (HIPAA, GDPR) and ethical review boards. The marginal cost of one additional surgical trajectory is orders of magnitude higher than one additional tabletop manipulation trajectory.

The Asymmetry: Abundant Video, Absent Actions

Compounding this problem is a striking asymmetry noted in the paper's abstract and introduction: vast corpora of surgical videos exist, but they lack corresponding action labels. Surgical procedures are routinely recorded for training, quality assurance, and research purposes. YouTube hosts credentialed surgical channels with procedure videos. Public datasets like SAR-RARP50, HeiCo, and AutoLaparo provide annotated surgical video clips. But these videos contain only visual observations—the endoscopic feed—without any record of what the surgeon's hands were doing at the control console, what joint angles the robotic instruments assumed, or what forces were applied.

This is not a minor inconvenience. It renders these videos completely unusable for imitation learning or VLA training, which fundamentally requires paired (observation, action) data. The paper draws a direct contrast with general robotics:

"Similarly, recent work on synthetic data generation such as DreamGen shows that video world-models combined with inverse dynamics can produce synthetic paired video–action datasets that boost policy learning in general manipulation tasks."

The key word here is "similarly"—the paper positions itself as bringing the DreamGen paradigm (world model + inverse dynamics = synthetic paired data) to surgery, where the data asymmetry is more extreme and the domain complexity is higher. In general manipulation, DreamGen can leverage existing robot-collected data to train the world model and IDM. In surgery, even that modest amount of paired data is scarce, making the pretraining and adaptation pipeline far more critical.

The Inadequacy of Existing Approaches

The paper categorizes prior attempts to address surgical data scarcity into several streams, each with fundamental limitations:

Synthetic physics-based simulators (Section 1, citations [6, 7]): Tools like Surgical Gym and SurRoL attempt to create simulated surgical environments with rigid-body physics. The paper identifies two crippling problems with these approaches:

"Synthetic physics-based simulators attempt to fill the gap, but often suffer from a large visual and dynamic domain shift to real surgical systems and lacking soft body simulation, limiting policy transfer."

The visual domain shift is severe because real surgical scenes involve wet, specular tissue surfaces, blood, smoke from cauterization, and complex lighting from endoscopic illumination—none of which are convincingly rendered by standard graphics engines. The dynamics shift is equally problematic: real tissue deforms nonlinearly, sutures behave as flexible inextensible materials, and tool-tissue interactions involve friction, puncture, and tearing that rigid-body simulators cannot model. Policies trained in simulation fail catastrophically when transferred to real tissue because the physics are fundamentally wrong.

Task-specific or narrow surgical world models (Section 2, citations [8, 9, 10]): Prior work has explored world-model-based approaches within surgical contexts, but each is limited in scope:

• GAS (World models for general surgical grasping, [8]) applies model-based reinforcement learning for grasping but is restricted to a single action category and object set.
• SurgWM (Surgical vision world model, [9]) generates controllable surgical videos but does not connect video generation to robot kinematics or policy learning—it stops at visual prediction.
• Suturing World Model ([10]) predicts tool trajectories for automated suturing but is task-specific and lacks text grounding.

The paper's characterization of these works is precise:

"While these studies mark important progress, they are limited to single-task or object-specific scenarios, and rely on narrowly scoped datasets lacking high-quality text–action alignment or procedural diversity, and many are not open-sourced, limiting reproducibility and broader impact."

The lack of text–action alignment is a critical missing piece. Without language grounding, a world model cannot generalize across tasks or accept natural language instructions—it is locked to the specific actions seen during training. Without procedural diversity (multiple surgery types, multiple action categories, varied anatomical contexts), the model cannot learn transferable representations of surgical manipulation.

Video generation without policy connection (Section 2, citations [20, 21, 22]): Models like Endora, SurGen, and VISAGE can generate or predict surgical videos, but they are designed for visual simulation, not for producing training data for robot policies. There is no mechanism to extract actions from the generated videos, so they cannot augment imitation learning datasets.

Latent action pretraining from internet videos (Section 2, citations [24, 25, 26]): This line of work, including approaches like LAP (Latent Action Pretraining) and Gen2Act, learns to extract behavioral representations from internet-scale human videos without action labels. While conceptually related, these methods have been developed and validated in general manipulation domains, not surgery. The visual and dynamic properties of surgical videos—endoscopic viewpoint, tool-centric motion, tissue deformation—are sufficiently different from everyday human activity videos that direct transfer is unreliable.

The Critical Missing Piece: Connecting Surgical World Models to Robot Learning

The paper's core motivation rests on identifying a gap that none of the prior work fills: no existing approach connects large-scale surgical video generation (world modeling) with robot kinematics (action prediction) to produce paired training data for surgical VLA policies. The paper states this as its third contribution directly:

"We are the first to connect surgical world models with robot learning by synthesizing video–action data using inverse dynamics models, achieving substantial performance improvements in surgical robot learning."

The emphasis on "first" is justified by the paper's detailed literature review. Prior surgical world models stop at video generation. Prior inverse dynamics work exists in general robotics but has not been applied to surgery. Prior surgical policy learning relies on scarce real demonstrations. The integration—surgical world model → synthetic videos → IDM pseudo-kinematics → VLA policy training—is novel as a complete pipeline, and the paper provides the first empirical evidence that this pipeline produces useful training data.

Why This Matters: Scaling Surgical Autonomy Safely

The paper's motivation extends beyond academic novelty to practical impact. The introduction frames it explicitly:

"Scalable data generation without collecting in-vivo trajectories can dramatically accelerate surgical autonomy while maintaining patient safety."

This is the key value proposition. If the approach works, it decouples surgical policy improvement from patient exposure. Rather than needing a surgeon to perform hundreds of repetitions of a specific maneuver on actual patients—with all the associated risks, costs, and ethical concerns—researchers could: (1) collect or curate existing surgical videos (without actions), (2) generate diverse synthetic videos from a world model, (3) infer pseudo-actions via an IDM, and (4) train policies on the combined real + synthetic dataset. The real demonstrations provide a grounding signal; the synthetic data provides diversity and volume.

The paper also positions itself within the broader trend of foundation models for robotics, noting that approaches like GR00T N1 achieve strong generalization by training on massive multi-embodiment datasets. Surgical robotics is currently excluded from this trend because such datasets do not exist. Cosmos-H-Surgical offers a path to generate the missing data synthetically, potentially enabling surgical VLAs to benefit from the same scaling properties that have driven progress in general manipulation.

How the Paper Positions Itself

The paper's positioning in the intellectual landscape has several dimensions:

Relative to DreamGen: Cosmos-H-Surgical is explicitly modeled on the DreamGen paradigm but adapted to surgery's unique constraints. DreamGen requires robot-collected data to train its world model and IDM; Cosmos-H-Surgical mitigates this by leveraging the SATA dataset of non-kinematic surgical videos for world model pretraining, reducing the amount of robot-specific data needed at adaptation time. The paper's few-shot adaptation experiments (Table 2) directly validate this: pretraining on SATA before finetuning on just 5 real trajectories yields substantially higher success rates than finetuning from the base model.

Relative to surgical VLMs: The paper distinguishes SATA from datasets like SurgVLM-DB, which are designed for semantic reasoning and instruction following. SATA is specifically designed for physical AI—its annotations capture the spatial, dynamic, and interaction details needed to train a world model that generates physically plausible video, not just semantically correct video.

Relative to surgical automation: The paper acknowledges prior work on task-level surgical autonomy (e.g., SRT, SRT-H) but positions its contribution as addressing the data generation problem rather than the policy architecture problem. The approach is orthogonal and complementary: better data from Cosmos-H-Surgical could improve any imitation learning policy, including SRT-style architectures.

As a platform, not a task-specific solution: The paper emphasizes that the framework is not tied to the needle pick-up and hand-over task used for validation. The SATA dataset covers four action categories across eight procedure types. The world model can generalize to new text prompts (as demonstrated in Figure 5). The IDM architecture is robot-specific but the approach is robot-agnostic. The paper presents Cosmos-H-Surgical as infrastructure for surgical robot learning, not a solution to a single task.

The Underlying Assumption Worth Examining

A subtle but important assumption underpins the paper's approach: that the inverse dynamics mapping from video to actions is learnable from limited data. If surgical manipulation involves many unobservable degrees of freedom—for instance, forces transmitted through tissue that are not visible in the endoscopic view, or subtle variations in grasp force that look identical on video but produce different outcomes—then the IDM's pseudo-kinematics will be systematically wrong, and adding synthetic data to real demonstrations could actually degrade policy performance rather than improve it.

The paper does not explicitly analyze this assumption, but its results (Figures 8, 11–14 in the supplementary) provide indirect evidence that the mapping is learnable: adding synthetic data consistently reduces trajectory MSE across policy architectures, data regimes, and training hyperparameters. If the IDM were producing noisy or systematically biased pseudo-actions, we would expect some configurations to show degradation (e.g., at low real-data regimes where the policy cannot overcome the synthetic noise). The universal improvement suggests that, at least for the needle manipulation task studied, the endoscopic view contains sufficient information to predict instrument kinematics to a useful degree of accuracy. This is an empirical finding with implications beyond this paper: it suggests that surgical video—even without explicit kinematic recording—carries substantial action-relevant information that can be extracted through learning.

3. Technical Approach

3.1 Reader Orientation

Cosmos-H-Surgical is a data generation pipeline that converts unlabeled surgical videos into labeled training data for surgical robot policies: a world model learns to generate realistic surgical videos from text prompts, an inverse dynamics model infers the robot actions that would produce those videos, and the resulting synthetic (video, action) pairs augment real demonstrations to train a vision-language-action policy. The system solves the problem that surgical robots need paired video-and-action data to learn from imitation, but such data is prohibitively expensive to collect at scale—the solution's shape is pretrain a generative model on abundant unlabeled video, then use a separately trained action-inference model to retroactively label the generated footage, creating a scalable synthetic data flywheel that requires only a handful of real demonstrations to ground the entire pipeline.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a sequential pipeline, plus a final training stage:

1. SATA Dataset (input foundation) — 2,447 expert-annotated surgical video clips with fine-grained text descriptions of tool-tissue interactions. This provides the domain-specific training signal for the world model, teaching it what surgical manipulation looks like and how language maps to actions. Without this dataset, the world model would have no surgical knowledge.
2. Cosmos-H-Surgical World Model — a diffusion-based video prediction model (fine-tuned from Cosmos-Predict2.5) that takes an initial endoscopic frame and a text prompt, then generates a future video rollout showing the specified surgical action. It is the "imagination engine" that produces diverse synthetic training scenarios.
3. Inverse Dynamics Model (IDM) — a separate neural network that takes two video frames spaced 16 frames apart and predicts the sequence of 20-dimensional robot actions (cartesian position, 6D rotation, gripper angle for both arms) that occurred between those frames. It is the "action labeler" that converts generated video into the format imitation learning requires.
4. GR00T N1.5 VLA Policy — the final surgical robot policy, a vision-language-action model that takes the current endoscopic frame, a text instruction, and the current robot state, then outputs predicted actions for the next 16 timesteps. It is trained on a mixture of real demonstrations and synthetic (video, pseudo-action) pairs.
5. Training Orchestration (the meta-process) — the pipeline operates in three stages: (i) pretrain the world model on SATA to learn surgical visual dynamics, (ii) fine-tune the world model and train the IDM on a small set of real robot trajectories to adapt to the specific embodiment, (iii) generate synthetic rollouts with the adapted world model, label them with the IDM, and train the policy on real + synthetic data.

Information flows unidirectionally: SATA videos with text → world model learns to generate surgical video → real robot videos (few-shot) → fine-tuned world model generates task-specific rollouts → IDM labels rollouts with pseudo-actions → policy trains on real + synthetic pairs → policy executes on the real robot.

3.3 Roadmap for the Deep Dive

• First, I will explain the SATA dataset in detail—what data it contains, how it was curated, what annotations look like, and why its design choices (fine-grained text, action categories, multi-source aggregation) matter for training physical AI models rather than semantic understanding models.
• Second, I will walk through the surgical world model: the base architecture (Cosmos-Predict2.5), the flow matching training objective, the LoRA fine-tuning strategy, and what "text-conditioned video prediction" concretely means in this context. This is the generative core of the pipeline.
• Third, I will cover the inverse dynamics model: its architecture (shared with the policy), how it maps video pairs to action sequences, why 16-frame spacing is used, and how it is trained on the limited real robot data before being deployed on synthetic videos.
• Fourth, I will explain the policy model (GR00T N1.5) and the training protocol—how real and synthetic data are combined, the two-stage fine-tuning procedure, and the action representation (20-dimensional vector with cartesian, rotation, and gripper components).
• Fifth, I will detail the few-shot adaptation pipeline that ties everything together: how the world model and IDM are specialized to a new robot embodiment using only 5, 10, or 20 real trajectories, and how synthetic data is generated at scale (56 single-rollout and 560 multi-rollout videos) for policy augmentation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems paper whose core idea is that surgical world models, when combined with inverse dynamics models, can generate synthetic paired video-action data that meaningfully augments scarce real demonstrations for training surgical robot policies—and that domain-specific pretraining on a curated surgical video-text dataset is essential for this to work rather than applying world models zero-shot.

---

The SATA Dataset: What It Is and Why It Is Constructed This Way

The Surgical Action–Text Alignment (SATA) dataset is a purpose-built resource designed to teach a generative model the visual dynamics of surgical manipulation and how natural language descriptions map to those dynamics. It is not a general surgical video dataset for semantic recognition or question-answering—it is explicitly constructed for physical AI, meaning its annotations encode the spatial, temporal, and interaction-level detail necessary for a world model to generate physically plausible video sequences.

Composition and scale. SATA contains 2,447 video clips totaling over 300,000 frames, collected across 8 different surgery types. Each clip belongs to one of four fundamental surgical actions, decomposed from the suturing workflow:

The action definitions are deliberately granular. "Needle grasping" is not simply "holding a needle"—it is specifically the dynamic approach-and-grasp phase, separated from the static holding phase because the visual dynamics (tool motion, depth changes, occlusion patterns) are qualitatively different. This decomposition matters because a world model trained on undifferentiated "grasping" would learn to generate static holding frames as often as dynamic reaching frames, reducing the quality of generated training data for policies that need to learn the reaching motion.

Source aggregation. SATA is a curated aggregation from existing resources, not newly collected data. The sources include:

• Credentialed YouTube surgical channels (68 clips, all knotting) — identified following the methodology of Schmidgall et al. (2024) for finding high-quality surgical content from verified medical sources
• SAR-RARP50 (2,148 clips across all four actions) — a dataset from robot-assisted radical prostatectomy, providing the bulk of the action diversity
• AutoLaparo (128 clips across all four actions) — laparoscopic hysterectomy data with multi-task annotations
• GraSP (66 clips, primarily grasping) — a dataset for holistic surgical scene understanding with pixel-wise annotations
• HeiCo (29 clips across all four actions) — Heidelberg colorectal data with sensor-OR recordings
• SurgicalActions160 (7 clips, knotting only) — laparoscopic video retrieval dataset
• Multiypass140 (1 clip, knotting only) — fine-grained surgical activity recognition data

The heavy weighting toward SAR-RARP50 (88% of clips) reflects that robotic prostatectomy videos are particularly abundant and well-annotated in the public domain. However, the paper also includes data from laparoscopic hysterectomy (AutoLaparo), colorectal surgery (HeiCo), and general laparoscopic procedures, providing some procedural diversity. The YouTube data adds real-world variation in lighting, camera quality, and surgical style that structured datasets may lack.

Annotation structure. Each clip is paired with a textual description that captures three types of information:

1. Spatial relationships between instruments — e.g., "left needle driver," "right needle driver," relative positioning
2. The anatomical structure being manipulated — e.g., "the right side of the patient's dorsal venous complex," specific tissue layers
3. The description of instrument–tissue interaction — e.g., "punctures," "passes needle to," "pulls thread through"

The paper provides a concrete example annotation: "The left needle driver punctures the right side of the patient's dorsal venous complex." This single sentence encodes which instrument (left needle driver), what action (punctures), what target (right side of dorsal venous complex), and the anatomical context (patient's dorsal venous complex, a structure encountered in prostate surgery). This level of detail is what distinguishes SATA from coarse category-level labels.

Why this annotation design matters. The paper compares three annotation granularities in the world model experiments (Table 1): zero-shot (no domain text), action-category (all videos in the same action category share an identical generic prompt), and SATA's fine-grained expert-curated descriptions. The fine-grained prompts produce substantially better video generation quality (FVD of 106.5 vs. 143.0 for action-category and 175.4 for zero-shot). The mechanism is straightforward: during training, the world model learns to associate specific textual descriptions with specific visual patterns. With coarse prompts, all needle puncture videos map to the same text embedding regardless of whether the puncture is left-to-right or right-to-left, superficial or deep, on prostate tissue or uterine tissue. The model must average over these variations, producing blurry, generic video. With fine-grained prompts, the model learns distinct text→video mappings for distinct surgical scenarios, enabling controlled generation at test time.

Design choice: text alignment over action labels. SATA deliberately provides text descriptions rather than action labels (like joint angles or tool poses). This is a critical design decision driven by data availability: surgical videos on YouTube and in public datasets have text annotations from the original sources (titles, descriptions, narration) but never include robot kinematics. By aligning the world model to text, the paper makes SATA scalable—new videos can be added with only text annotation, not expensive kinematic capture. The IDM bridges the gap from text-grounded video to actions downstream, keeping the expensive kinematic labeling requirement contained to the small set of real robot trajectories used for IDM training.

---

The Surgical World Model: Cosmos-H-Surgical

The world model is the generative engine of the pipeline. Its job is to produce photorealistic, physically plausible, and text-controllable surgical videos that can serve as training data for downstream policies. The paper frames this as a video prediction task: given the first frame of a scene and a text instruction, generate the subsequent frames showing the specified action being performed.

Base model: Cosmos-Predict2.5. The paper builds on Cosmos-Predict2.5, a large-scale video world model described in the Cosmos 2.5 technical report (Ali et al., 2025). Cosmos-Predict2.5 is a diffusion-based latent video prediction model that uses a transformer backbone operating in a compressed latent space. Its key properties—relevant to understanding why it was chosen—are:

• Pretrained on diverse robotic and embodied datasets: Cosmos-Predict2.5 was trained on heterogeneous sources including robot manipulation videos, human teleoperation footage, and general physical interaction scenes. This pretraining provides strong priors about how objects move, how tools interact with surfaces, and how three-dimensional scenes project to two-dimensional video over time. These are exactly the priors needed for surgical tool-tissue interaction.
• Latent video prediction, not pixel-space prediction: The model encodes video frames into a compressed latent representation using a spatiotemporal encoder, performs temporal dynamics modeling in this latent space via a transformer, then decodes back to pixel space. This is computationally efficient and—crucially—forces the model to learn structured representations of motion and appearance rather than memorizing pixel patterns.
• Diffusion-based generation: The model generates video by iteratively denoising random latent representations, producing diverse but coherent outputs. This is essential because the pipeline needs to generate multiple distinct rollouts (10× with different random seeds) from the same initial frame to provide diverse training data.

Adaptation strategy: LoRA fine-tuning. The paper uses Low-Rank Adaptation (LoRA) to specialize Cosmos-Predict2.5 for the surgical domain. The core idea of LoRA is that, rather than updating all parameters of the pretrained model during fine-tuning (which is expensive and risks catastrophic forgetting of general video knowledge), you insert small trainable low-rank matrices into the transformer's attention and feed-forward layers and only train those. The original weights remain frozen. This has two advantages for the surgical use case:

1. Parameter efficiency: LoRA adapters add only a small fraction of parameters relative to the full model (typically well under 1%), making fine-tuning computationally tractable on the SATA dataset.
2. Knowledge preservation: By freezing the base weights, the model retains its general understanding of object motion, occlusion, and scene dynamics from the diverse pretraining data. The LoRA adapters learn to specialize this general knowledge to surgical tool-tissue interactions, specular tissue reflections, and endoscopic viewpoint constraints.

The paper does not specify the rank or placement of LoRA adapters in the main text, but standard practice (and the referenced Hu et al., 2022) would apply low-rank matrices (rank typically 4–64) to the query and value projection matrices in self-attention layers and potentially to the feed-forward layers, with the rank determining the adapter's capacity to learn domain-specific patterns.

Flow matching training objective. Cosmos-H-Surgical is trained using the Flow Matching (FM) formulation rather than traditional diffusion objectives like DDPM. The paper explains this choice in the supplementary material (Section 7):

"We adopt the Flow Matching (FM) formulation to train the surgical video world model due to its conceptual simplicity and practical effectiveness. FM defines a velocity-based target in latent space, providing a direct training signal that improves optimization stability and sample quality."

The flow matching framework replaces the multi-step noise-prediction objective of DDPM with a simpler velocity-prediction objective. Here is the formal treatment:

Step 1: Interpolation between data and noise. Given a data sample $I$ (the latent representation of the video frames to be generated), a noise vector $\epsilon \sim \mathcal{N}(0, I)$, and a timestep $t \in [0, 1]$ sampled from a logit-normal distribution, the interpolated latent is:

$I_t = (1 - t) I + t \epsilon$

where $I_t$ is the intermediate state at time $t$, obtained by linear interpolation between the clean data $I$ at $t=0$ and pure noise $\epsilon$ at $t=1$. The logit-normal distribution for sampling $t$ concentrates probability mass near $t \approx 0$ and $t \approx 1$, giving the model more training signal at the difficult beginning and end of the denoising trajectory.

Step 2: Ground-truth velocity. The corresponding ground-truth velocity vector that transforms $I_t$ toward $I$ is:

$v_t = \epsilon - I$

where $v_t$ is the constant vector field that connects the noisy latent to the clean latent. Crucially, this velocity is constant along the interpolation path—it does not depend on $t$ except through $I_t$'s dependence on $t$. This is a key property that makes flow matching simpler than score-based diffusion: the target is a straight-line path from noise to data, not a curved trajectory.

Step 3: Flow matching loss. The model learns to predict this velocity via a neural network $u_\theta(I_t, t, c)$, where $c$ represents the conditioning information (the initial frame $I_0$ and the text prompt). The flow-matching loss is:

$\mathcal{L}(\theta) = \mathbb{E}_{I, \epsilon, c, t} \left[ \| u_\theta(I_t, t, c) - v_t \|_2^2 \right]$

where $\theta$ represents the trainable LoRA adapter parameters, $I$ is a training video sample, $\epsilon$ is sampled Gaussian noise, $c$ is the conditioning (initial frame + text), $t$ is the sampled timestep, $I_t$ is the interpolated latent at that timestep, $u_\theta(I_t, t, c)$ is the model's predicted velocity vector, and $v_t = \epsilon - I$ is the ground-truth velocity.

What it computes: This is a mean squared error between the predicted velocity vector (what the model thinks the direction and magnitude of change should be at timestep $t$) and the ground-truth velocity vector (the actual straight-line direction from the current noisy state to the clean data). The expectation averages over all training videos, noise samples, conditioning contexts, and timesteps.

Why this form: Flow matching directly regresses the velocity field rather than the score function (gradient of log-density) used in score-based diffusion or the noise prediction used in DDPM. The velocity $v_t = \epsilon - I$ defines a straight-line path through latent space, which means at inference time the model can solve an ordinary differential equation (ODE) from noise to data with fewer integration steps than the stochastic differential equation (SDE) required by DDPM. This makes generation faster and more stable. The logit-normal timestep distribution ensures the model sees enough training examples near $t \approx 0$ (where it must predict the fine details that distinguish clean data from slightly-noisy data) and near $t \approx 1$ (where it must predict the coarse structure of the data from nearly pure noise), both of which are critical for sample quality.

What happens at inference time. Given an initial frame $I_0$ and a text prompt $c$, the world model generates a video rollout $\hat{I}_{1:T} = \mathcal{W}_\theta(I_0, c)$ where $\mathcal{W}_\theta$ denotes Cosmos-Predict2.5 with LoRA adapters, and $T$ is the number of frames to generate. The generation process starts with pure noise at $t=1$, iteratively applies the velocity predictions $u_\theta$ to move toward $t=0$ (solving the flow ODE), and finally decodes the latent representation back to pixel space. The result is a video sequence showing the surgical action described in the text prompt, conditioned on the initial scene configuration from $I_0$.

The text conditioning mechanism. The paper does not detail the exact text encoding architecture, but standard practice for diffusion-based video models involves encoding the text prompt with a pretrained language model (e.g., T5 or a CLIP text encoder), then injecting the resulting text embedding into the transformer backbone via cross-attention layers. During training on SATA, the model learns to associate specific text embeddings with specific visual dynamics patterns. At inference, a novel text prompt (e.g., "two-time needle handover") produces a text embedding that guides the denoising process toward video dynamics consistent with that description, even if the exact combination of words was never seen during training (as demonstrated in Figure 5 with the multi-step handover generalization).

---

The Inverse Dynamics Model: Extracting Actions from Video

The world model produces video, but imitation learning requires (observation, action) pairs. The inverse dynamics model (IDM) bridges this gap by predicting what robot actions occurred between two video frames. It is a separately trained neural network that does not generate video—it only maps from visual observations to actions.

Architecture: Shared backbone with the policy. The IDM uses the same architecture as the GR00T N1.5 policy model (shown in Figure 3), with one critical difference: it does not use text prompts or robot state inputs. The architecture is a Diffusion Transformer (DiT) with flow matching heads, but adapted for the inverse dynamics task:

• Input: Two video frames separated by 16 timesteps (specifically, frame $i$ and frame $i + 16$ from the same video sequence). Both frames are processed through a SigLIP-2 vision encoder to produce visual features.
• Processing: The visual features are fed into a diffusion transformer, which models the distribution of possible action sequences that could explain the observed visual change between the two frames. The transformer operates in the 20-dimensional action space, not in pixel or latent video space.
• Output: A sequence of 20-dimensional action vectors for every frame between the two input frames (i.e., frames $i+1$ through $i+16$), representing the robot's continuous motion trajectory during that interval.

Action representation. The robot state and actions share the same 20-dimensional continuous vector format, defined at each timestep $t$ as:

$a_t = [p_L, r_L, g_L, p_R, r_R, g_R]$

where:

• $p_L = [x_L, y_L, z_L] \in \mathbb{R}^3$ is the translational offset (in meters) of the left forcep tip relative to the endoscope coordinate frame. This is a 3D cartesian position in endoscope-centric coordinates, not in world coordinates, ensuring view-consistent control—if the endoscope moves, the coordinate system moves with it.
• $r_L = [r_{L1}, \ldots, r_{L6}] \in \mathbb{R}^6$ is the 6D rotation representation of the left instrument's end-effector orientation. The paper uses the 6D continuous rotation formulation from Zhou et al. (2019), which represents rotation as the first two columns of a $3 \times 3$ rotation matrix (6 numbers total). This avoids the discontinuities and double-cover issues of quaternions and the gimbal lock of Euler angles. Crucially, the 6D representation is continuous on $\text{SO}(3)$, meaning small changes in rotation produce small changes in the 6D vector, which is essential for gradient-based learning.
• $g_L \in \mathbb{R}$ is the gripper jaw opening angle in radians, controlling how wide the forcep tips are spread.
• $p_R, r_R, g_R$ are the corresponding values for the right forcep.

All translation and rotation components are expressed relative to the endoscope's coordinate frame. This is an important design choice: it means the policy learns motion in a frame that moves with the camera, so if the endoscope position changes between training and deployment (e.g., different trocar placement), the action representation remains valid because everything is relative to the current view.

Why 16-frame spacing. The paper specifies that the IDM takes frames $i$ and $i + 16$ as input and predicts actions for all intermediate frames. This spacing (roughly 0.5–1.0 seconds at typical endoscopic video frame rates of 25–30 fps) is chosen because:

1. Sufficient visual change: Over 16 frames, surgical instruments move enough that the displacement is visually distinctive. At 1–2 frame spacing, tool motion might be sub-pixel and uninformative for action prediction.
2. Temporal context: The 16-frame window captures meaningful action segments (a partial reach, a grasp closure, a rotation) rather than atomic single-frame displacements that are individually ambiguous.
3. Alignment with the policy's prediction horizon: The GR00T N1.5 policy also predicts 16 future actions, so the IDM's output format matches what the policy needs as training targets.

IDM training procedure. The IDM is trained on real robot trajectories only—synthetic videos are not used for IDM training because they do not have ground-truth actions. The training data consists of:

• Out-of-domain episodes: 66 episodes (approximately 60,000 action frame pairs) of general robot movements unrelated to the needle pickup task. These provide broad coverage of the robot's kinematic range and motion patterns.
• In-domain episodes: 5, 10, or 20 needle pickup episodes (depending on the experimental condition), providing task-specific motion patterns.

The authors start from a pretrained Franka IDM checkpoint from DreamGen (Jang et al., 2025) and fine-tune for 10,000 steps with a learning rate of 1e-4. The DreamGen checkpoint provides general knowledge about how robot arms move from video, which transfers across embodiments because the fundamental visual cues of tool motion (optical flow patterns, occlusion boundaries, depth changes) are embodiment-agnostic.

IDM inference on synthetic videos. Once trained, the IDM processes each synthetic video rollout generated by Cosmos-H-Surgical. For a rollout of length $T$ frames, the IDM processes consecutive pairs $(i, i+16)$ for $i = 1, \ldots, T-16$, producing pseudo-labeled action sequences that span the entire video. These pseudo-actions are not ground truth—they are the IDM's best estimate of what actions would produce the observed visual changes—but the paper's results demonstrate they are accurate enough to improve policy learning.

Why the IDM is necessary rather than generating actions directly. The world model generates video but does not output actions. One might ask: why not train the world model to generate actions directly? The answer lies in the data asymmetry that motivates the whole paper. The world model is pretrained on SATA, which has text annotations but no actions. During few-shot adaptation, it sees only 5–20 real trajectories with actions—far too few to learn a generalizable action prediction model. The IDM, by contrast, is trained on the same few real trajectories plus 66 out-of-domain episodes (providing ~60,000 action frames of kinematic diversity). This is still modest but substantially more than what the world model sees. Separating video generation (learned from abundant unlabeled data) from action inference (learned from limited kinematic data) allows each model to be trained on the data regime most appropriate for its task.

---

The Vision-Language-Action Policy: GR00T N1.5

The policy model is the final output of the pipeline—what actually controls the surgical robot. It is a vision-language-action (VLA) model based on GR00T N1.5 (Bjorck et al., 2025), an open foundation model for generalist humanoid robots.

Architecture (Figure 3, right). The policy has four components:

1. Vision encoder: A SigLIP-2 model processes the current endoscopic frame, producing visual features that capture the spatial configuration of tools, tissue, and the needle.
2. Language encoder: A vision-language model processes the text instruction (e.g., "The left forcep picks up a needle and hands over to the right forcep"), producing a language embedding that specifies what task to perform.
3. State encoder: A dedicated encoder processes the current robot state $a_t$ (the 20-dimensional action vector at the current timestep), providing the policy with proprioceptive information about where the instruments currently are.
4. Diffusion Transformer with flow matching head: The visual features, language embedding, and state encoding are combined and fed into a DiT backbone, which models the distribution of future action sequences and outputs predicted actions for the next 16 timesteps.

The architecture is shared between the IDM and the policy, but with different inputs: the IDM takes two frames (no text, no state), while the policy takes one frame plus text and current state.

Action prediction as flow matching. Like the world model, the policy uses flow matching for action prediction. Rather than regressing a single deterministic action sequence (which would collapse to the mean of the training distribution and produce jerky, averaged motion), flow matching models the full distribution of plausible action sequences. At inference time, the model samples from this distribution, producing smooth, human-like trajectories that reflect the natural variability in surgical manipulation.

Training protocol. The policy is trained in a two-stage procedure:

Stage 1: Pretraining on synthetic data. The policy checkpoint is initialized from the pretrained GR00T N1.5 weights, which provide a strong starting point from training on diverse robotic manipulation data. The model is then fine-tuned on the synthetic (video, pseudo-action) dataset for 400 steps with learning rate 1e-4. This stage exposes the policy to diverse surgical scenarios generated by the world model, teaching it to map endoscopic observations to instrument motions across varied starting configurations.

Stage 2: Fine-tuning on real data. The synthetic-pretrained checkpoint is further fine-tuned on the 5, 10, or 20 real demonstrations for 200 steps with learning rate 1e-4. This stage grounds the policy in the actual dynamics of the specific robot platform, correcting any biases or inaccuracies in the IDM's pseudo-actions.

The two-stage design reflects a deliberate curriculum: the synthetic data provides breadth (diverse scenarios, varied initial conditions, multiple action sequences), while the real data provides accuracy (correct kinematics, true robot dynamics). The synthetic data alone would produce a policy that generates plausible-looking but potentially inaccurate motions; the real data alone would produce a policy with limited generalization to novel configurations. The combination aims to capture the strengths of both.

The paper also experiments with varying the number of fine-tuning steps on real data (200 steps in the main experiments, 1k and 10k steps in the supplementary Figures 12 and 13). The consistent pattern—synthetic pretraining helps across all step counts—suggests the benefit is not simply due to early stopping or regularization.

Why GR00T N1.5 and not a purpose-built surgical architecture. The paper intentionally uses a general-purpose VLA rather than designing a surgical-specific policy architecture. This is a methodological choice that serves two purposes: (1) it demonstrates that the synthetic data generation pipeline is architecture-agnostic and can benefit any imitation learning policy, and (2) it positions the contribution as solving the data problem rather than the architecture problem. The supplementary experiments with $\pi$0.5 (Figure 14) reinforce this: the same synthetic data improves a different VLA architecture, confirming that the data quality rather than architectural synergy drives the gains.

---

The Full Few-Shot Adaptation and Data Generation Pipeline

The complete workflow (Figure 2) operates in three sequential phases, with specific data splits and generation parameters at each stage.

Phase 1: World model pretraining on SATA. Cosmos-Predict2.5 is fine-tuned with LoRA adapters on the full SATA dataset (2,447 clips, 300k+ frames) using the flow matching objective described above. This phase does not involve any robot-specific data—only internet surgical videos with text annotations. The output is the base Cosmos-H-Surgical model that understands surgical visual dynamics and text-video alignment. Training details beyond the flow matching formulation and LoRA are not specified in the main text, but the supplementary material (Section 7) provides the loss formulation and notes that training uses the logit-normal timestep distribution.

Phase 2: Few-shot embodiment adaptation. For each experimental data regime (5, 10, or 20 real demonstrations), two adaptations occur in parallel:

• World model fine-tuning: Cosmos-H-Surgical is further fine-tuned on the 5/10/20 real-world videos from the needle pick-up and hand-over task. This teaches the world model the specific visual appearance of the target robot platform (endoscope quality, lighting conditions, forcep appearance, rubber pad texture) and the specific task dynamics (needle grasping from a flat surface, hand-over motion pattern). The fine-tuning uses the same LoRA-based approach, adding task-specific adapters on top of the SATA-trained surgical adapters.
• IDM training: A separate IDM is trained from the DreamGen Franka checkpoint, using the 66 out-of-domain episodes plus the 5/10/20 in-domain episodes, for 10,000 steps at learning rate 1e-4. Each data regime produces a separate IDM to ensure the IDM's quality is matched to the world model's quality at that data level.

Phase 3: Synthetic data generation and policy training. With the adapted world model and trained IDM:

1. Initial frame selection: 56 initial frames are selected from the 66 out-of-domain episodes, chosen based on having proper initial needle and forceps configurations that make the task physically meaningful. These frames were not seen during world model fine-tuning (they come from the out-of-domain set, not the training demonstrations), ensuring the generation tests generalization to novel scene configurations.
2. Single-rollout generation: The 5/10/20-demo fine-tuned world model generates 56 synthetic videos, one per initial frame (1× generation). Each rollout is conditioned on the initial frame and the task text prompt ("The left forcep picks up a needle and hands over to the right forcep").
3. Multi-rollout generation: The same 56 initial frames are used with 10 different random seeds to generate 560 synthetic videos (10× generation). The random seeds control the noise initialization in the flow matching inference process, producing diverse rollouts with the same initial frame and text prompt—different grasp trajectories, different hand-over timings, different approach angles.
4. Pseudo-action labeling: The IDM processes both the 56 single-rollout and 560 multi-rollout videos, outputting pseudo-labeled action sequences for each. This produces two datasets: Real + Synthetic (56 synthetic episodes) and Real + Synthetic 10× (560 synthetic episodes).
5. Policy training: The GR00T N1.5 policy is trained in the two-stage procedure described above: 400 steps on synthetic data, then 200 steps on real data. The baseline (Real Only) skips the synthetic stage and trains for 200 steps on real data directly.

Why 56 initial frames and 10 seeds. The 56 initial frames come from the 66 out-of-domain episodes (selected for proper task initialization), providing a modest but non-trivial amount of scene diversity. Ten random seeds per initial frame produce an order-of-magnitude increase in data volume (56 → 560) while preserving the scene diversity—the policy sees the same scenes but with different motion trajectories, forcing it to learn the underlying task structure rather than memorizing specific motion patterns. The paper's experiments show that 10× synthetic data consistently outperforms 1× synthetic data (Figures 8, 11–14), confirming that volume matters beyond scene diversity alone.

Data preprocessing. All videos are resized to 224×224 pixels (standard for vision transformer inputs) and augmented with color jitter during training. The jaw opening angles and cartesian positions are normalized using min-max normalization (scaling each dimension to a fixed range based on the training data extrema). This prevents the policy from being biased toward dimensions with larger numerical ranges (e.g., cartesian positions in meters might span 0.01–0.05 while rotation components span 0.0–1.0). Rotation components use the 6D representation throughout and do not require normalization beyond what the representation itself provides (the 6D vectors are naturally bounded since they come from the first two orthonormal columns of a rotation matrix).

4. Key Insights and Innovations

Innovation 1: Reframing Surgical Data Scarcity from a Collection Problem to a Generation-and-Inference Problem

The paper's most fundamental conceptual move is not building a better world model or policy—it is redefining the nature of the data bottleneck in surgical robotics. Prior work treats the absence of paired video-action data as a collection problem: if only we could record more surgeries with kinematic sensors, we could train better policies. This leads to approaches like physics-based simulators (SurRoL, Surgical Gym) that attempt to create artificial paired data through explicit modeling, or data-efficient imitation learning methods (conservative Q-learning, randomized-to-canonical adaptation) that squeeze more value from limited real demonstrations.

Cosmos-H-Surgical rejects this framing entirely. Instead, it decomposes the problem into two learnable components that can be trained on different data sources: a world model that learns visual dynamics from abundant unlabeled surgical videos (the SATA dataset, 2,447 clips, zero kinematic labels), and an inverse dynamics model that learns the mapping from visual change to robot actions from a small set of real demonstrations (60 trajectories total, plus 66 out-of-domain episodes). The critical insight is that the expensive part of the data—kinematic labels—is only needed for the IDM, not for the world model. The world model, which requires far more training data to capture the diversity of surgical visual dynamics, trains exclusively on freely available video with text annotations. The IDM, which only needs to learn a local mapping from two frames to actions, trains on the small real dataset.

This decomposition is not obvious in the prior surgical literature, where world models (SurgWM, Suturing World Model) were treated as standalone video predictors and inverse dynamics (LAP, Gen2Act) were developed in non-surgical domains without connection to world models. The paper's contribution is recognizing that these two techniques, when properly aligned through text conditioning and embodiment-specific fine-tuning, solve a problem neither alone could address: the world model provides the visual diversity that the IDM cannot generate, and the IDM provides the kinematic labels that the world model cannot produce.

The significance extends beyond the specific pipeline. This decomposition establishes a general template for data-scarce physical AI domains: identify which component of your training pipeline requires expensive supervised data and which can be trained on abundant weakly-labeled data, then structure your architecture to keep them separate. In surgery, the split is visual dynamics (cheap, SATA) vs. kinematics (expensive, real demos). In other domains, the split might be different—scene geometry vs. contact forces, object appearance vs. manipulation strategy—but the principle of decomposing along the cheap/expensive data boundary generalizes.

Evidence for this reframing's validity comes from the few-shot adaptation results (Table 2): SATA-pretrained Cosmos-H-Surgical achieves a 73.2% video success rate from only 5 real trajectories, versus 51.8% for the same model fine-tuned without surgical pretraining. The pretrained model learned surgical visual dynamics from unlabeled data; the 5 trajectories only needed to adapt it to the specific robot embodiment. This is the empirical signature that the decomposition works: the world model's expensive training (SATA curation and pretraining) pays off by radically reducing the amount of expensive real data needed at adaptation time.

Innovation 2: Text-Video Alignment as a Mechanism for Compositional Generalization in Surgical Actions

The paper's second conceptual contribution is demonstrating that fine-grained text-video alignment enables compositional generalization of surgical actions—a capability that is not simply "better video generation" but a qualitatively different kind of model behavior. Prior surgical world models (SurGen, VISAGE, Endora) used text conditioning primarily to specify which procedure or high-level action to generate (e.g., "cholecystectomy," "needle puncture"). The generated videos were expected to match the broad action category but not to compose novel action sequences from language.

Cosmos-H-Surgical's SATA dataset encodes surgical actions at the level of instrument identity, anatomical target, and interaction type in a single structured sentence (e.g., "The left needle driver punctures the right side of the patient's dorsal venous complex"). This is not merely more detailed captioning—it is a deliberate design choice that enables the world model to learn disentangled representations of who (left vs. right instrument), what (puncture vs. grasp vs. pull), and where (specific anatomical structures). When the model is prompted with a novel combination at inference time—such as "two-time needle handover" (Figure 5, Case 2), which was never seen in training because all handover videos were decomposed into single handover segments—it can compose the learned primitives correctly: left-to-right handover followed by right-to-left handover.

This compositional behavior is significant because it demonstrates that the world model is not memorizing video-text pairs but learning a structured mapping from language semantics to visual dynamics. The model understands "left needle driver passes needle to right needle driver" as a parameterized action template that can be sequenced ("then right needle driver passes needle to left needle driver") to produce novel multi-step behaviors. This is exactly the kind of generalization needed for surgical policy learning: if the model can generate diverse compositions of basic actions from text alone, the synthetic data it produces for policy training will cover a combinatorially larger space of surgical scenarios than the original training videos.

The paper provides direct evidence for this capability through the human expert evaluation (Figure 6), where Cosmos-H-Surgical scores 2.79/3.0 on text-video alignment (vs. 2.23 for action-category prompts and 2.18 for zero-shot). The coarse action-category baseline—where all videos in a category share an identical generic prompt—cannot achieve this compositional behavior because the text conditioning provides no information about instrument identity, direction, or anatomical target. The model must average over all variations within the category, losing the structured decomposition that enables composition. This is a clear demonstration that annotation granularity determines generalization capability: coarse labels produce interpolation within known categories; fine-grained structured labels enable extrapolation to novel compositions.

Innovation 3: Inverse Dynamics as a Bridge Between Generative World Models and Policy Learning—with Surgical Validation

While the concept of using inverse dynamics to extract actions from video is not new (it dates back to Video Pretraining (VPT) from Baker et al., 2022, and was recently applied in DreamGen for general manipulation), Cosmos-H-Surgical is the first to validate this approach in the surgical domain and to demonstrate that it works despite surgery's unique visual challenges. This is not just "applying DreamGen to surgery"—the paper makes a specific empirical contribution by showing that the IDM can recover useful kinematic information from endoscopic video despite specular tissue surfaces, limited field of view, and constrained tool motion, all of which create fundamentally different visual ambiguity than the tabletop scenes where prior IDMs were validated.

The visual properties of surgery that make inverse dynamics particularly challenging include:

• Specular highlights: Wet tissue surfaces create moving reflections that do not correspond to geometric motion. An IDM must learn to ignore these highlight motions and attend to the underlying tool edges and tissue deformation patterns that actually encode kinematics.
• Endoscopic parallax: The endoscope is inside the body, viewing instruments at close range with extreme perspective distortion. Small tool translations can produce large image-space motions, while large rotations can produce minimal visual change depending on the viewing angle. The IDM must learn a view-dependent mapping that accounts for the endoscope's optical properties.
• Tissue deformation as ambiguous signal: When a tool presses on tissue, the tissue deforms. This deformation is visually prominent but provides ambiguous kinematic information—the same tissue indentation could result from different combinations of tool force and angle. The IDM must learn to resolve this ambiguity by integrating information across frames.

The paper does not explicitly analyze these failure modes, but the consistent improvement from synthetic data across all experimental conditions (Figures 8, 11–14 in the supplementary) provides strong indirect evidence that the IDM successfully handles them. If any of these ambiguities were systematically corrupting the pseudo-actions, we would expect synthetic data augmentation to sometimes hurt performance—the policy would learn incorrect motion patterns from the pseudo-labels. The universal improvement across policy architectures (GR00T N1.5 and π0.5), data regimes (5, 10, 20 demonstrations), synthetic data volumes (56 vs. 560 episodes), and training hyperparameters (200, 1k, 10k fine-tuning steps) is a powerful robustness check: the IDM's pseudo-actions are consistently useful, not just accidentally helpful in a narrow range of conditions.

The significance of this finding is that it opens the door to scaling surgical policy learning through video alone. If inverse dynamics from endoscopic video is reliable enough to generate useful training data, then the vast corpus of surgical videos on platforms like YouTube—which contain diverse procedures, surgeons, and anatomical variations but zero kinematic labels—becomes a potential source of policy training data. The current paper only uses SATA for world model pretraining, not for IDM training, because the IDM requires some kinematic ground truth. But a future system could potentially train a general surgical IDM on a modest set of recorded procedures with kinematics, then deploy it to label any surgical video, massively scaling the training data for surgical VLAs. This is the logical endpoint of the decomposition principle from Innovation 1.

Innovation 4: Few-Shot World Model Adaptation as a General Strategy for Embodiment Transfer

The paper's fourth conceptual contribution is the demonstration that a surgical world model pretrained on diverse internet video can be adapted to a specific robot embodiment with as few as 5 real trajectories, and that this adapted model produces synthetic data that meaningfully improves policy learning. This is distinct from prior work on few-shot policy adaptation (e.g., SRT, which adapts a policy architecture to a specific task) because it adapts the data generation pipeline rather than the policy directly. The world model is the component being adapted; the policy benefits indirectly through the improved synthetic data.

This matters because it suggests a two-tier transfer learning strategy for surgical robotics: pretrain a general surgical world model on all available surgical video (tier 1, done once, expensive), then adapt it to any new robot platform with a handful of demonstrations (tier 2, done per-robot, cheap). The tier 1 pretraining amortizes the cost of learning surgical visual dynamics across all downstream robots. The tier 2 adaptation only needs to learn the visual appearance of the specific endoscope and instruments and the dynamics of the specific task.

The paper provides direct evidence for the value of tier 1 pretraining in Table 2: Cosmos-H-Surgical (SATA-pretrained, then adapted on 5 trajectories) achieves 73.2% video success rate vs. 51.8% for Finetuned-Orig (no SATA pretraining, adapted on the same 5 trajectories). The SATA pretraining provides a 21.4 percentage point improvement from the same 5 real trajectories—essentially, it makes 5 trajectories as effective as a much larger number would be without pretraining. The qualitative comparisons in Figure 10 reinforce this: the SATA-pretrained model generates smooth, accurate tool motions, while the non-pretrained model produces implausible tool interactions and incorrect grasps.

Why this is a fundamental advance rather than incremental: prior surgical world models (SurgWM, Suturing World Model, GAS) were trained from scratch on task-specific data. They had no tier 1 pretraining step and therefore required all their training data to come from the specific robot embodiment. This fundamentally limited their scalability—every new robot or task required collecting a new dataset from scratch. Cosmos-H-Surgical breaks this coupling by separating general surgical knowledge (from SATA) from embodiment-specific adaptation (from few-shot demos). This is the same conceptual architecture that made foundation models transformative in NLP and computer vision: a general pretraining phase that learns broadly useful representations, followed by lightweight adaptation to specific downstream tasks. The paper's contribution is demonstrating that this architecture works for surgical video generation specifically, and that the generated video can feed into policy learning.

The limitation is that both the world model and the IDM require embodiment-specific adaptation data. The paper acknowledges this in Section 5: "It requires finetuning datasets from unseen robot embodiments for both world model and IDM, which require additional data curation efforts." A fully general system would need a world model that can generate videos of unseen robots without any adaptation data—perhaps conditioned on a few example frames rather than fine-tuned—and an IDM that generalizes across embodiments without retraining. The current paper does not achieve this, but the two-tier transfer strategy represents a clear step in that direction.

Key Design Choice Worth Highlighting: The Separation of World Model and IDM Training Data

Though not a standalone innovation, one design choice deserves explicit recognition because it represents a non-obvious engineering insight that enables the whole pipeline: the world model and the IDM are trained on different, non-overlapping data sources, and this separation is crucial for the pipeline's success. The world model trains on SATA (unlabeled internet video with text) plus a few real trajectories for adaptation. The IDM trains only on real robot trajectories (in-domain + out-of-domain) but never sees synthetic video during its own training. At inference, the IDM processes videos it has never seen before—generated by a different model (the world model) operating in a different modality (text-to-video vs. video-to-action).

This separation means the IDM must generalize from real video to synthetic video. If there is a significant domain gap between the world model's generated frames and real endoscopic frames, the IDM's pseudo-actions will be inaccurate, and the synthetic training data will harm rather than help the policy. The paper's results (universal improvement from synthetic data) demonstrate that this domain gap is small enough to be manageable, but this is not guaranteed a priori—it is an empirical finding that required careful world model training (fine-grained SATA annotations, LoRA adaptation to real robot video) to achieve.

The alternative design—training the IDM on synthetic video with some form of domain randomization or adversarial alignment—would introduce additional complexity and potential failure modes. The paper's choice to keep the IDM trained purely on real data represents a bet that the world model's video quality is high enough to bridge the domain gap implicitly, and the results suggest this bet pays off. This design pattern (separate generative and analytical models trained on different data distributions, combined at inference time) is a generalizable architectural principle for domains where different components of the pipeline have access to different types of supervision.

5. Experimental Analysis

Evaluation Methodology

Dataset. The SATA dataset (2,447 clips, 300k+ frames across 8 surgery types) provides the pretraining corpus for the world model, sourced from YouTube surgical channels and six public datasets (SAR-RARP50, GraSP, AutoLaparo, HeiCo, SurgicalActions160, Multiypass140). For policy evaluation, the authors use 60 human-teleoperated needle pick-up and hand-over demonstrations on a physical surgical robot. The last 40 episodes are held out as the test set; the remaining 20 are split into training regimes of 5, 10, or 20 demonstrations. An additional 66 out-of-domain episodes (~60k action frame pairs) depicting general robot movements—not the needle task—are used to pretrain the IDM and to select 56 initial frames for synthetic video generation, chosen based on proper needle and forceps configurations.

Base model(s). The world model builds on Cosmos-Predict2.5 (Ali et al., 2025), a diffusion-based latent video prediction model with a transformer backbone, pretrained on diverse robotic and embodied datasets. Domain adaptation uses Low-Rank Adaptation (LoRA, Hu et al., 2022). The IDM and VLA policy both use GR00T N1.5 (Bjorck et al., 2025) as their architectural backbone—a DiT with flow matching heads—with the IDM inheriting from a DreamGen Franka checkpoint (Jang et al., 2025). Supplementary experiments replace the GR00T policy with π0.5 (Black et al., 2025) to test cross-architecture generalizability.

Metrics. The paper measures three quantities: (1) Fréchet Video Distance (FVD) (Unterthiner et al., 2019) for video generation quality, computed between generated and real video distributions in feature space; (2) VBench metrics (Huang et al., 2024)—dynamic degree (DD, measuring motion magnitude), imaging quality (IQ, measuring frame-level realism), and overall consistency (OC, measuring temporal coherence)—following the Cosmos 2.5 evaluation protocol; and (3) Trajectory mean squared error (MSE) between the policy's predicted 20-dimensional action vectors and ground-truth kinematics, computed separately for cartesian position (3 dimensions per arm), 6D rotation (6 dimensions per arm), and gripper jaw opening (1 dimension per arm), then averaged across all 40 test episodes. Standard deviations are reported for MSE values. For the world model evaluation alone, the paper additionally reports success rate (SR)—the fraction of generated videos where surgical experts judge the task trajectory as complete.

Baselines. Four world model variants are compared: (1) Zero-Shot—Cosmos-Predict2.5 evaluated without any surgical domain adaptation; (2) Action-Category—Cosmos-Predict2.5 fine-tuned on SATA using coarse, category-level captions where all videos within the same action category share an identical generic prompt; (3) Cosmos-H-Surgical—Cosmos-Predict2.5 fine-tuned on SATA's fine-grained, expert-curated textual descriptions; and (4) Finetuned-Orig—Cosmos-Predict2.5 fine-tuned directly on 5 real surgical trajectories without SATA pretraining. For policy evaluation, the baseline is Real Only—GR00T N1.5 fine-tuned exclusively on 5, 10, or 20 real demonstrations with no synthetic data. Two synthetic data conditions are compared: Real + Synthetic (56 synthetic episodes generated as single rollouts from 56 initial frames) and Real + Synthetic 10× (560 synthetic episodes generated as 10 random-seed rollouts per initial frame).

Generation budget / compute accounting. The paper measures world model generation in terms of video rollouts: one rollout equals a complete predicted video sequence from the initial frame. Single-rollout experiments produce 56 videos (one per initial frame); multi-rollout experiments produce 560 videos (10 seeds × 56 frames). IDM training consumes 10,000 steps at learning rate 1e-4. Policy training follows a two-stage protocol: 400 steps on synthetic data (when applicable) followed by 200 steps on real data at learning rate 1e-4. Supplementary experiments test 1k and 10k fine-tuning steps on the real data stage to assess sensitivity to optimization budget.

Cross-validation / statistical protocol. No explicit cross-validation or multiple random seeds for policy training are reported. The test set is a fixed 40-episode hold-out split from the 60 collected demonstrations. The 56 initial frames for synthetic generation are selected from the 66 out-of-domain episodes based on task initialization criteria, but no analysis of sensitivity to frame selection is provided. The multi-rollout generation uses 10 random seeds applied to the world model's flow matching inference to produce diverse trajectories, but the policy training appears to use a single training run per condition (no error bars from multiple training seeds are reported for the policy MSE results, though standard deviations across test episodes are shown in Figures 8, 11–14). The human expert evaluation uses 3 surgical experts independently rating 50 videos, with scores averaged, but inter-rater reliability metrics are not reported.

---

Main Quantitative Results

Surgical Video Generation Quality (World Model Evaluation)

The core finding is that fine-grained SATA text annotations produce substantially better video generation than coarse category-level prompts or zero-shot application. Table 1 reports the head-to-head comparison on the SATA dataset:

Cosmos-H-Surgical reduces FVD by 39% relative to zero-shot (175.4 → 106.5) and by 26% relative to action-category prompts (143.0 → 106.5). The dynamic degree more than doubles from ~27 to 62.4, indicating the fine-grained model generates substantially more instrument motion rather than static or near-static frames. Imaging quality improves marginally (48.7 → 49.3), reflecting that single-frame realism is largely determined by the base Cosmos-Predict2.5 architecture, not the surgical fine-tuning. Overall consistency shows a meaningful gain (18.0 → 21.5), suggesting better temporal coherence.

The action-category baseline reveals an important negative result: coarse text supervision barely helps. The action-category model improves FVD from 175.4 to 143.0—a modest gain—but the dynamic degree actually drops slightly (26.9 → 26.5) and imaging quality and consistency are nearly identical to zero-shot. This demonstrates that simply having some surgical text labels is insufficient; the granularity of the text annotations directly determines the model's ability to learn controllable, dynamic video generation. The coarse prompts force the model to average over all visual variations within each action category, producing the same averaging behavior (blurry motion, generic tool appearance) that zero-shot generation exhibits due to lack of domain knowledge.

Qualitative results (Figure 4) reveal failure modes invisible to aggregate metrics. In a challenging scenario where the initial frame contains no visible surgical tools, the Zero-Shot model hallucinates an incorrect instrument entirely (red arrow in the figure), the Action-Category model initiates the wrong action (tissue puncture when the prompt specifies needle grasping), and Cosmos-H-Surgical correctly follows the prompt to perform needle grasping. These failures are not captured by FVD or VBench metrics, which average over entire video distributions. The figure provides evidence that SATA's fine-grained annotations specifically prevent two distinct failure modes: instrument hallucination (zero-shot) and action confusion (coarse prompts).

New behavior generalization (Figure 5) demonstrates compositional capability beyond the training distribution. Given the same initial frame, Cosmos-H-Surgical generates four distinct video rollouts corresponding to four different text prompts: one-time needle handover, two-time handover, three-time handover, and needle puncture. The multi-step handover cases are particularly significant because all multi-step handovers in SATA were decomposed into single handover segments during data curation, meaning the model was never trained on explicit "two-time" or "three-time" handover videos. The model correctly composes the learned single-handover primitive multiple times, producing coherent multi-step sequences with increasing motion complexity. This is direct evidence of compositional generalization from text-video alignment, not memorization of training examples.

Human expert evaluation (Figure 6) provides the most clinically meaningful assessment. Three surgical experts (one surgeon guiding two residents) independently rated 50 videos from each world model variant across three criteria on a 1–3 scale:

• Text-Video Alignment: Cosmos-H-Surgical: 2.79; Action-Category: 2.23; Zero-Shot: 2.18
• Tool Consistency: Cosmos-H-Surgical: 2.81; Action-Category: 2.45; Zero-Shot: 1.86
• Anatomical Structure: Cosmos-H-Surgical: 2.73; Action-Category: 1.26; Zero-Shot: 1.38

The anatomical structure scores are the most striking: Cosmos-H-Surgical nearly doubles the baselines (2.73 vs. ~1.3), while the baselines perform only marginally above the lowest score of 1. This suggests that coarse or absent surgical pretraining produces videos where tissue and organ appearance is fundamentally implausible to trained surgeons—a failure mode that quantitative metrics like FVD cannot detect because they operate on visual feature statistics, not anatomical correctness. The tool consistency gap (2.81 vs. 1.86 zero-shot) similarly indicates that zero-shot models produce instrument behaviors that experts recognize as physically inconsistent.

Few-Shot World Model Adaptation to Real Surgical Trajectories

SATA pretraining dramatically improves few-shot adaptation quality, nearly doubling the effective value of limited real data. Table 2 compares three configurations on the task of generating videos from 56 held-out initial frames after adaptation to only 5 real trajectories:

The success rate tells the clearest story: zero-shot cannot complete the task at all (0.0% SR), direct fine-tuning on 5 trajectories achieves 51.8%, and SATA-pretrained fine-tuning reaches 73.2%. The 21.4 percentage point gap between Finetuned-Orig and Cosmos-H-Surgical represents the value of surgical video pretraining—it makes 5 real trajectories produce results that would otherwise require substantially more real data.

The FVD improvement from Finetuned-Orig to Cosmos-H-Surgical is modest (212.5 → 207.1, only 2.5% reduction), suggesting that the main benefit of SATA pretraining is not improved frame-level visual quality but rather improved task execution—coherent tool motion, correct action sequencing, successful task completion. The dynamic degree increases from 85.7 to 89.3, reflecting slightly more natural instrument motion. The imaging quality is high across all finetuned variants (72.0 and 73.3) because the base Cosmos-Predict2.5 already produces photorealistic frames when conditioned on real endoscopic images—the challenge is in generating the right motion and interactions, not the right textures.

Qualitative comparisons (Figure 10, supplementary) reveal specific failure modes of non-pretrained adaptation. Under two representative initial states with different needle and forceps arrangements, the Zero-Shot model produces implausible tool motions and incorrect instruments, leading to early task failure. The Finetuned-Orig model reduces hallucinations but still produces inconsistent grasping behavior and incoherent tool interaction—the model sometimes moves the forceps toward the needle but fails to close the gripper at the right moment, or generates tool trajectories that pass through the needle rather than grasping it. Cosmos-H-Surgical generates smooth, accurate motions that correctly execute the manipulation sequence. Red arrows in the figure mark specific failure instances, including tool hallucination and incorrect action execution.

A critical methodological note: The success rate evaluation uses surgical experts to assess trajectory completeness in generated videos, not actual robot execution. This is a video-level metric that measures whether the generated video looks like it shows a successful task completion, not whether a robot executing the IDM-inferred actions from that video would successfully grasp the needle. The 73.2% success rate should be interpreted as "73.2% of generated videos appear to show successful task completion to surgical experts," which is a necessary but not sufficient condition for the synthetic data to be useful for policy training.

Robotic Policy Learning with Synthetic Data Augmentation

The central policy result is that synthetic data from Cosmos-H-Surgical consistently and substantially reduces trajectory prediction error across all data regimes, with 10× synthetic data outperforming 1×. Figure 8 reports the trajectory MSE for the GR00T N1.5 policy on the 40-episode test set, broken out by cartesian position, rotation, and gripper dimensions, across three real data regimes (5, 10, 20 demonstrations) and three training conditions (Real Only, Real + Synthetic 56 episodes, Real + Synthetic 10× 560 episodes).

Headline pattern: Across all 9 combinations of data regime and metric, Real + Synthetic 10× achieves the lowest MSE, Real + Synthetic achieves intermediate MSE, and Real Only achieves the highest MSE. The ordering is monotonic: more synthetic data → lower error, for every real data quantity and every action dimension.

Cartesian position error shows the most pronounced improvements. With 5 real demonstrations, Real Only produces the highest error; adding 56 synthetic episodes reduces error visibly; adding 560 synthetic episodes reduces it further. The same pattern holds at 10 and 20 demonstrations, though the absolute error decreases as real data increases. The gap between Real + Synthetic 10× and Real Only narrows with more real data—at 5 demos the synthetic advantage is largest, at 20 demos it is smallest but still present—consistent with the intuition that synthetic data provides diminishing marginal returns as real data volume increases.

Rotation and gripper errors follow the same qualitative pattern but with smaller absolute gaps between conditions, suggesting that cartesian position is the action dimension where synthetic data provides the most benefit. This is plausible: cartesian trajectory requires modeling the global tool motion visible across many frames, which the IDM can infer from the accumulated visual displacement. Rotation and gripper angle may involve more subtle visual cues (tool orientation from shaft appearance, jaw angle from gripper shape) that are harder for the IDM to recover accurately from synthetic video.

Example trajectory comparison (Figure 7) shows the left arm cartesian trajectory (first 3 of 20 action dimensions) for Real Only, Real + Syn 10×, and ground truth. The Real + Syn 10× trajectory follows the ground truth more closely than Real Only, particularly in capturing the overall motion profile (direction and timing of positional changes). The Real Only trajectory diverges more from ground truth in both magnitude and temporal alignment. This single-example visualization provides qualitative intuition for the aggregate MSE improvements in Figure 8.

The quantitative MSE values are not directly reported in the main text, appearing only as bar charts in Figure 8 with the y-axis labeled "MSE" but without numerical values overlaid. Standard deviation bars are shown, but the figure resolution makes precise numerical extraction impractical without access to the source data. This limits the ability to report exact percentage improvements (e.g., "MSE reduced by X% when adding synthetic data"). However, the visual trends are unambiguous and consistent across all panels.

Robustness Across Policy Architectures and Training Hyperparameters

The synthetic data benefit generalizes to a different VLA architecture. Supplementary Figure 14 shows the same experiment repeated with the π0.5 policy (Black et al., 2025) instead of GR00T N1.5. The bar charts for cartesian, rotation, and jaw MSE across 5, 10, and 20 real demonstrations show the identical pattern: Real + Synthetic 10× < Real + Synthetic < Real Only for all conditions. This is a critical robustness check because it rules out the possibility that the GR00T N1.5 architecture has some special synergy with the Cosmos-H-Surgical synthetic data. The fact that a completely different VLA design—developed by a different research group with different training data—shows the same improvement pattern strongly suggests the synthetic data itself is genuinely useful, rather than the benefit arising from architectural compatibility.

The benefit persists across varying fine-tuning step budgets. Supplementary Figures 12 and 13 test the same experimental design but with 1,000 and 10,000 fine-tuning steps on real data (instead of the 200 steps used in the main experiments). At 1k steps (Figure 12): Real + Synthetic 10× outperforms Real + Synthetic, which outperforms Real Only, for all data regimes and all action dimensions. At 10k steps (Figure 13): the same monotonic ordering holds. This demonstrates that the synthetic data advantage is not an artifact of early stopping or regularization—it persists even when the policy is trained to convergence on the real data. The gap does narrow at 10k steps, particularly for rotation and gripper dimensions, suggesting that extensive real-data training can partially compensate for the lack of synthetic diversity, but the synthetic data still provides measurable benefit.

Multi-view policy learning benefits from single-view synthetic data. Supplementary Figure 11 tests whether synthetic data generated from single-view endoscopic video can improve a policy trained on multi-view real data (adding left and right wrist cameras to the standard endoscopic view). Even though the synthetic data is single-view, the multi-view policy Real + Synthetic 10× outperforms Real + Synthetic, which outperforms Real Only. This is a non-obvious result: the synthetic video provides diverse single-view dynamics that the policy can leverage even when it has access to additional camera views during real-data training. The GR00T N1.5 architecture processes varying numbers of input views with shared weights, so the single-view synthetic pretraining likely improves the shared visual representations that the policy uses across all views.

---

Ablation Studies and Robustness Checks

Text annotation granularity for world model training (Table 1): The comparison of Zero-Shot vs. Action-Category vs. Cosmos-H-Surgical is effectively an ablation of text conditioning quality. The finding is that coarse category-level prompts (e.g., "needle grasping" for all grasping videos) provide almost no benefit over zero-shot for dynamic degree and only modest FVD improvement, while fine-grained expert prompts (instrument identity + anatomical target + interaction type) produce large gains. This demonstrates that text-video alignment quality depends on annotation precision, not merely the presence of text.

SATA pretraining for few-shot adaptation (Table 2): The Cosmos-H-Surgical vs. Finetuned-Orig comparison ablates the value of SATA pretraining when both models receive identical few-shot adaptation data (5 real trajectories). SATA pretraining provides a 21.4 percentage point success rate improvement (73.2% vs. 51.8%), establishes that general surgical video knowledge transfers to specific robot embodiments.

Synthetic data volume (Figure 8, Real + Synthetic vs. Real + Synthetic 10×): Across all data regimes and action dimensions, 10× synthetic data (560 episodes) outperforms 1× synthetic data (56 episodes). The gap is largest at low real-data regimes (5 demonstrations) and narrows as real data increases, suggesting diminishing marginal returns from synthetic data as the real data volume grows. The 10× condition uses 10 random seeds per initial frame but only 56 initial frames—so the diversity comes from motion variability, not scene variability. This suggests that sampling multiple possible motion trajectories from the same scene is valuable, not just covering more scenes.

VLA architecture independence (Figure 14, supplementary): Replicating the full experiment with π0.5 instead of GR00T N1.5 produces the same pattern of improvement from synthetic data. This ablates the possibility that the benefit is architecture-specific.

Fine-tuning step budget (Figures 12 and 13, supplementary): The synthetic data advantage persists at 1k and 10k fine-tuning steps, ruling out the hypothesis that synthetic pretraining merely provides a better initialization that would be matched by training longer on real data alone.

Multi-view vs. single-view data (Figure 11, supplementary): Single-view synthetic data improves multi-view policy training, suggesting the synthetic data's value is in the visual dynamics it provides, not in matching the camera configuration.

IDM training data composition: The paper reports that the IDM is trained on 66 out-of-domain episodes plus 5/10/20 in-domain episodes, but does not ablate the relative importance of these two data sources. A natural question—does the out-of-domain data provide essential kinematic diversity, or could the IDM be trained on in-domain data alone?—is not answered.

Initial frame selection for synthetic generation: The 56 initial frames are selected from 66 out-of-domain episodes based on having "proper initial needle and forceps configurations." No ablation of this selection criterion versus random frame selection is provided, so we cannot assess whether the selection procedure contributes to the synthetic data's utility.

---

Critical Assessment

The experiments provide credible evidence that synthetic data from Cosmos-H-Surgical improves surgical policy learning, but several aspects of the evaluation design limit the strength of conclusions that can be drawn, and some important claims in the paper are supported more weakly than they first appear.

On the claim that synthetic data "significantly outperforms" real-data-only training: The trajectory MSE results in Figure 8 consistently show Real + Synthetic 10× achieving lower error than Real Only across all conditions, which qualitatively supports the claim. However, the paper does not report the actual numerical MSE values in the main text, does not provide confidence intervals or statistical tests comparing conditions, and does not report the variance across multiple policy training runs (only standard deviation across test episodes, which is a different quantity). Without these, we cannot assess whether the observed differences are statistically significant or practically meaningful in terms of actual robot task success. The single trajectory example in Figure 7 is illustrative but cannot substitute for systematic statistical reporting.

On the claim that Cosmos-H-Surgical is the "first surgical world model" to do what it does: The paper's literature review in Section 2 is thorough and the claimed novelty appears genuine—prior surgical world models (SurgWM, Suturing World Model, GAS) do not connect to policy learning through inverse dynamics, and prior inverse dynamics work (DreamGen, LAP) has not been applied to surgery. However, the paper does not compare against any of these prior approaches experimentally. The only baselines are variants of Cosmos-Predict2.5 under different fine-tuning regimes. A comparison against a simpler synthetic data generation method—for instance, using a physics-based simulator like SurRoL instead of the world model—would strengthen the claim that the proposed approach is superior to existing alternatives.

On the few-shot adaptation results: The 5-trajectory adaptation experiment (Table 2) is compelling but measures video success rate as judged by surgical experts, not robot execution success rate. The 73.2% success rate means that 73.2% of generated videos look successful to experts. Whether the IDM-inferred actions from those videos would actually succeed on the robot is a different question that goes unaddressed. The paper implicitly assumes that expert-judged video success is a proxy for action quality, but this assumption is not validated—for instance, by taking a subset of generated videos, extracting IDM actions, executing them on the robot, and measuring actual task completion.

On the relationship between world model quality and policy improvement: The paper shows that Cosmos-H-Surgical generates better video than baselines (Table 1, Table 2) and that using Cosmos-H-Surgical's synthetic data improves policies (Figure 8). But it does not demonstrate a causal link: would the policy perform equally well with synthetic data from the lower-quality Action-Category world model? This ablation—training policies on synthetic data from different world model variants—would directly test whether world model quality matters for downstream policy performance. Its absence is a notable gap, because it's possible that even low-quality synthetic data (e.g., Action-Category videos with incorrect action execution) could improve policies through regularization or data augmentation effects unrelated to video realism.

On scalability claims: The paper's abstract claims the approach "offers a scalable path toward autonomous surgical skill acquisition," but the experiments are conducted on a single task (needle pick-up and hand-over) on a single robot platform. The compositional generalization demonstrated in Figure 5 (multi-step handover, puncture from the same initial frame) is promising evidence that the world model can generate diverse behaviors, but the policy experiments do not test whether synthetic data from one task improves learning on a different task—the most direct test of scalability. Similarly, the method requires separate world model fine-tuning and IDM training for each new robot embodiment, which the paper acknowledges as a limitation (Section 5) but the experiments do not quantify how much data this adaptation requires or whether the adaptation quality degrades for more substantially different robots.

On the IDM's reliability: The paper presents the IDM as a black box that produces useful pseudo-actions, but provides no direct evaluation of IDM accuracy—no comparison of IDM-predicted actions against ground truth kinematics on a held-out set of real trajectories. The policy improvements provide indirect evidence that the IDM is accurate enough to be useful, but we cannot distinguish between "IDM produces moderately accurate actions that provide useful regularization" and "IDM produces highly accurate actions that serve as direct substitutes for real demonstrations." This distinction matters for understanding how far the approach can scale: if the IDM is only moderately accurate, the synthetic data benefit will plateau as more synthetic data is added (the policy will hit the IDM's accuracy ceiling), whereas if the IDM is highly accurate, adding more synthetic data should continue to help.

On test set size and composition: The policy evaluation uses 40 held-out test episodes, all from the same data collection session, same robot platform, same task, same needle, and same rubber pad. This is a small and homogeneous test set. Generalization to different initial conditions, different needle types, different tissue substrates, or different lighting conditions is not tested. The out-of-domain episodes (66 general movement trajectories) suggest some kinematic diversity exists in the training data, but the test set does not evaluate whether the policy generalizes to these out-of-domain movement patterns.

On missing comparisons: Several experiments would have strengthened the paper significantly:

• Policy trained on synthetic data only (no real fine-tuning): This would isolate the contribution of synthetic data quality from the real-data fine-tuning stage. If synthetic-only training achieves non-trivial policy performance, it would validate the IDM's accuracy directly.
• Policy trained on real data augmented with simple augmentations (color jitter, translation, rotation of existing real demonstrations): This would test whether the synthetic data benefit is due to the diversity of generated scenes or simply due to having more data with any variability. The paper uses color jitter for all training, but does not compare against stronger augmentation baselines.
• Policy trained with synthetic data from a physics simulator (SurRoL or Surgical Gym): This would establish whether the world model approach outperforms existing simulation-based data augmentation, which is the most direct alternative to the proposed method.
• IDM accuracy evaluation on real held-out trajectories: This would quantify the fundamental ceiling on synthetic data quality imposed by IDM limitations.

On metric limitations: The paper relies entirely on trajectory MSE for policy evaluation. MSE is convenient but correlates imperfectly with task success in robotic manipulation—a policy could have low MSE but still fail to grasp the needle (e.g., if the gripper closes at the wrong moment by a small temporal offset), or have higher MSE but succeed (e.g., if it takes a slightly different but equally valid path to the grasp point). Actual task success rate on the physical robot—even on a subset of test conditions—would be a far more convincing evaluation. The paper's video success rate metric (Table 2) is evaluated only on the world model's generated videos, not on the policy's executed actions.

On data leakage concerns: The 56 initial frames for synthetic generation are selected from the 66 out-of-domain episodes. These episodes depict general robot movements not related to the needle task, but they are collected on the same robot platform and possibly in the same session as the in-domain demonstrations. If the out-of-domain episodes contain frames that are visually similar to test-set initial conditions—even if the subsequent motion is different—the synthetic data might inadvertently provide information about the test distribution. The paper does not discuss this possibility or verify that the out-of-domain episodes are truly distributionally distinct from the test set.

Despite these limitations, the core empirical result—that synthetic surgical video data from a domain-adapted world model, labeled via inverse dynamics, improves imitation learning for surgical manipulation—is consistently demonstrated across multiple policy architectures, training regimes, and data volumes. The universality of the improvement pattern (every bar chart, every condition, every supplementary experiment) makes it unlikely to be a statistical artifact, even without formal significance testing. The paper's contribution is more in establishing this phenomenon's existence than in precisely quantifying its magnitude, and on those terms the experiments are sufficient. The identified gaps (no robot execution success rate, no comparison to simulator baselines, no IDM accuracy evaluation) represent opportunities for the natural follow-up work that this paper enables rather than fatal flaws in the current evaluation.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Excluded from All Efficiency Calculations

The assumption or constraint. The entire compute-optimal framework relies on estimating prompt difficulty before allocating the test-time compute budget. The paper's method for difficulty estimation—generating 2048 samples per question and averaging PRM final-answer scores—is extraordinarily expensive, consuming more compute than the largest test-time budgets studied (256–512 generations). The authors acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported ~4× efficiency gains are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter—potentially eliminating or even reversing the efficiency gains. A system that spends 2048 generations to decide whether to use 16 or 64 generations for the actual solution is not 4× more efficient; it is ~30× less efficient than simply running best-of-256 on every problem. The paper acknowledges this as an exploration-exploitation tradeoff (Section 3.2) but does not propose a solution or account for the cost.

What evidence exists in the paper. The 2048-sample difficulty estimation procedure is described in Section 3.2, but its cost is never included in any budget calculation, efficiency comparison, or FLOPs analysis. The $4\times$ efficiency claims (Figures 4 and 8) implicitly assume difficulty is known for free. The paper does not provide an ablation where difficulty estimation cost is subtracted from the budget, nor does it test whether the compute-optimal policy remains beneficial when estimation cost is amortized.

Mitigation status. Not addressed. The paper flags this as "a key avenue for future work" (Section 3.2), suggesting training models to predict difficulty directly from question text, but no such model is developed or evaluated. Until this gap is closed, the efficiency gains should be understood as an upper bound on achievable deployment efficiency, not a realized gain.

---

The Method Provides No Benefit on Genuinely Hard Problems

The assumption or constraint. The entire framework operates under the assumption that the base model already produces correct solutions at some non-trivial rate. On the hardest problems (difficulty bin 5), where the base model's pass@1 rate is near zero, no amount of test-time compute—search or revisions—produces meaningful improvement. The paper is transparent about this (Section 7 takeaway box), but the implication is that test-time compute amplifies existing capability without creating new capability.

The consequence. For any problem outside the base model's rough capability envelope—genuinely novel reasoning, problems requiring knowledge the model lacks, or tasks where the model's pass@1 is effectively zero—Cosmos-H-Surgical provides zero benefit regardless of budget. This means the method is fundamentally bounded by pretraining quality: it cannot compensate for a fundamentally undertrained model, and it cannot extend the model's capabilities beyond what it already knows. For safety-critical surgical applications where failure is unacceptable, this sharp boundary between "problems the model can solve with help" and "problems the model cannot solve at all" is a critical limitation—there is no graceful degradation, just a cliff.

What evidence exists in the paper. Figure 3 (right), bin 5 shows near-zero accuracy (1–3%) for all methods and all budgets. Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of sequential-to-parallel ratio. Figure 9, bin 5 scaling lines are essentially flat near 0–5% across all budgets and $R$ regimes. In the FLOPs-matched comparison, hard questions show a −52.9% relative disadvantage from using test-time compute instead of a ~14× larger model (Section 7, PRM search at $R \gg 1$).

Mitigation status. The paper acknowledges this explicitly as a fundamental boundary condition but offers no mitigation beyond scaling pretraining instead. The finding itself is valuable—it establishes a clear boundary for when test-time compute is appropriate—but for practitioners facing hard problems, the method provides no path forward.

---

The Revision Model Exhibits a Fundamental Correct-to-Incorrect Reversion Problem

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). During training, the model never sees examples of correct answers in context, so it has no signal for what to do when the current answer is already correct. At inference time, as the revision chain progresses, the model sometimes encounters correct answers it produced in earlier steps and "revises" them into incorrect answers.

The consequence. The paper reports a specific failure rate:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

This means revision chains are unstable: the model does not know when to stop revising, and continuing to apply revisions degrades performance rather than improves it. The system must rely on selection mechanisms—majority voting or verifier-based selection across the entire chain—to rescue the correct answer from among the revisions. This is a bandage, not a solution: it means that a substantial fraction of the sequential budget is wasted generating revisions that make the answer worse, and the system's effective throughput is reduced because it must generate a full chain and then search backward for the best answer rather than converging monotonically.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. Figure 6 (left) shows that pass@1 at each revision step gradually improves (up to ~24–25% by steps 15–20) but also shows that early steps sometimes outperform later steps—consistent with correct answers being overwritten. The paper mitigates this with majority voting and verifier-based selection (Section 6.1, Appendix I), which recovers the correct answer when it exists somewhere in the chain, but this adds computational overhead and does not prevent the generation of wasted revisions.

Mitigation status. Partially addressed through selection mechanisms (majority voting, verifier-based chain selection), but these are workarounds, not solutions. A principled fix—training the model to recognize when no revision is needed, or including correct-answer-exists-in-context examples during training—is not explored. The ReST$^{EM}$ experiment (Appendix K, Figure 16) shows that attempting to optimize the revision model with RL-style training made the problem worse, not better, suggesting the reversion behavior is sensitive to training methodology in ways that are not yet understood.

---

Verifier Over-Optimization Imposes a Hard Ceiling on Search Scaling—and the Paper Does Not Solve It

The assumption or constraint. All search methods against the PRM are bounded by the verifier's reliability. As search becomes more aggressive (higher budget, more powerful algorithms like beam search or lookahead search), it increasingly finds solutions that score highly under the PRM but are actually incorrect. This is a well-known phenomenon in reward optimization, but the paper provides systematic evidence that it is the primary bottleneck for test-time compute scaling with search methods.

The consequence. The compute-optimal policy mitigates this problem by routing easy problems away from aggressive search (using best-of-N instead of beam search on difficulty bin 1), but it does not solve it. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling—beam search curves (Figure 3) flatten or decline well before the budget is exhausted. Lookahead search, the most powerful optimizer, paradoxically performs worst overall (Figure 3, left) because its aggressive optimization amplifies verifier errors. This means that even with perfect difficulty estimation and compute-optimal allocation, the system's maximum achievable accuracy is capped by verifier quality, not by compute budget. No amount of additional test-time compute will push past this ceiling without a better verifier.

What evidence exists in the paper. Figure 3 (right), bin 1 shows beam search decreasing in accuracy as budget increases (78% → 77% from 4 to 256 generations)—direct evidence of verifier exploitation. Figure 3 (left) shows lookahead search underperforming simpler methods at all budgets. Figure 3 generally shows beam search performance flattening well before 256 generations. Qualitative examples in Appendix M (Figure 29) show search producing degenerate outputs—repetitive low-information steps, overly short 1–2 step solutions—that score highly under the PRM but are clearly incorrect.

Mitigation status. Not solved. The compute-optimal policy avoids the worst over-optimization by switching strategies per difficulty bin, but the underlying verifier quality ceiling remains. The paper identifies this as a key bottleneck (Section 8) and suggests future work on more robust verifiers (adversarial training, ensembles, KL-penalty approaches from RLHF), but no experiments address it. For practitioners, this means that the method's scaling ceiling is determined by verifier training quality, and improving the verifier is likely more impactful than developing more sophisticated search algorithms.

---

Difficulty Bins Are Coarse, Static, and Computed via an Expensive Oracle Procedure

The assumption or constraint. The paper discretizes difficulty into five quintiles based on the base model's pass@1 rate, estimated either via ground-truth correctness (oracle) or PRM score averages (predicted) from 2048 samples per question. These bins are treated as static: once a question is assigned to a bin, the same strategy is used for all budgets. There is no mechanism for dynamically adjusting the strategy mid-computation based on how the initial samples perform.

The consequence. This is a coarse approximation of a continuous space. Within a single quintile—say, bin 3 with roughly 100 questions per fold—there may be substantial heterogeneity: a question at the easy end of bin 3 and one at the hard end of bin 3 receive identical strategy allocations, even though different strategies might be optimal. A finer-grained or continuous difficulty estimate could improve allocation efficiency, but the discrete binning throws away information.

Additionally, treating difficulty as static means the system cannot adapt mid-trajectory. If the first few samples from a question produce wildly different PRM scores (suggesting unexpected difficulty), the system has no mechanism to adjust its remaining budget allocation. A dynamic policy that starts with a few parallel samples, assesses the score distribution, and then decides in real-time whether to switch strategies could subsume the difficulty estimation cost into the problem-solving process and adapt to unexpected difficulty. The current approach cannot do this.

What evidence exists in the paper. The five-bin discretization is described in Section 3.2, with cross-validation splitting each bin roughly in half for strategy selection (~50 questions per bin per fold). The paper does not explore sensitivity to the number of bins (e.g., 3 bins vs. 5 vs. 10), nor does it test whether the compute-optimal strategy varies within a bin. No dynamic allocation experiments are reported. The predicted difficulty bins use a single aggregate statistic (average PRM final-answer score) to assign questions to bins, but the paper does not analyze how often this statistic misclassifies a question's difficulty relative to oracle bins or how misclassification affects downstream performance.

Mitigation status. Not addressed. The paper acknowledges the cost of difficulty estimation (Section 3.2) but does not discuss bin granularity or static-vs-dynamic allocation as limitations. Future work could explore continuous difficulty estimates with smooth policy functions, or dynamic allocation policies that adapt strategy mid-computation based on early sample quality, but no such directions are proposed.

---

The $14\times$ Larger Model Baseline Is Not Compute-Optimally Trained and Receives No Test-Time Budget

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm rather than Chinchilla-optimal pretraining where data and parameters scale equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the ~14× larger model uses only greedy decoding—no test-time compute augmentation of any kind (no majority voting, no best-of-N, no search).

The consequence. The pretraining baseline is weaker than it needs to be on two fronts simultaneously. A Chinchilla-optimal model trained with ~14× more total FLOPs (scaling both data and parameters optimally) would likely outperform a parameter-only-scaled model, making the pretraining baseline a stronger competitor. Giving that larger model even a modest test-time compute budget (say, best-of-8 or best-of-16) would also strengthen the baseline, since the paper's core argument is about allocating compute between pretraining and inference—a comparison where only the smaller model gets inference compute is asymmetric by design.

The reported advantages of test-time compute over pretraining—e.g., +27.8% relative improvement on easy questions at $R \ll 1$ for revisions, or +19.1% for PRM search—may shrink substantially or even reverse against a properly compute-optimal larger model with a fair test-time compute allocation. The paper's conclusion that "a smaller model with additional test-time compute can outperform a ~14× larger model" is true for the specific comparison tested but may not hold for a more carefully optimized pretraining baseline.

What evidence exists in the paper. The FLOPs-matched comparison methodology is described in Section 7, including the explicit note about parameter-only scaling. The ~14× larger model's performance is shown as stars in Figure 9 and in the bar charts of Figure 1. No experiment tests the larger model with any test-time compute augmentation, nor does the paper compare against a Chinchilla-optimally trained larger model. The dependence of the results on these baseline choices is not ablated.

Mitigation status. The paper acknowledges the non-Chinchilla-optimal pretraining as a limitation explicitly in Section 7, framing the current comparison as "representative of a canonical approach." The lack of test-time compute for the larger model is not acknowledged as a limitation. Future work would need to compare against both a compute-optimally trained larger model and a larger model with some inference budget to make a fully fair case for test-time compute over pretraining. The current results should be interpreted as a best-case scenario for test-time compute relative to pretraining, not an unbiased estimate of the tradeoff.

7. Implications and Future Directions

How This Work Changes the Landscape

Cosmos-H-Surgical does not introduce a new policy architecture or a fundamentally novel learning algorithm. Its contribution is architectural: it demonstrates that the data bottleneck in surgical robotics—widely treated as a collection problem requiring more operating-room recordings—can be reframed as a generation-and-inference problem that decomposes into components trainable on different data sources with different labeling costs. This is a conceptual shift with practical consequences for how the field allocates research effort.

The magnitude is a reframing, not a breakthrough. The paper does not claim to solve surgical autonomy. It validates the pipeline on a single task (needle pick-up and hand-over) on a single robot platform using a small test set of 40 episodes, and the evaluation measures trajectory MSE rather than task execution success. The gains are consistent (synthetic data helps across all conditions in Figures 8, 11–14) but the absolute magnitudes are not reported numerically in the main text, and no statistical testing is provided. This is establishing feasibility, not demonstrating a mature system. The contribution's weight comes from opening a direction rather than closing a problem: it shows that world-model-generated synthetic video, labeled via inverse dynamics, produces training data that meaningfully improves imitation learning for surgical manipulation. Before this paper, there was no evidence that this pipeline could work in the surgical domain with its unique visual challenges (specular tissue, endoscopic viewpoint, constrained tool motion). Now there is.

How this reframes prior work. The paper reconciles several strands of research that previously operated in isolation:

• Surgical world models (SurgWM, Suturing World Model, GAS, Endora, SurGen, VISAGE) were developed as video generation or prediction tools with no connection to robot policy learning. Cosmos-H-Surgical shows that these models—when properly aligned with text and adapted to a specific embodiment—can serve as data generators for downstream policies, not just visual simulators. This transforms surgical world modeling from a niche video-synthesis subfield into infrastructure for robot learning.

• Inverse dynamics from video (VPT, LAP, DreamGen, Gen2Act) was developed and validated exclusively in general manipulation domains (tabletop grasping, household tasks). Cosmos-H-Surgical provides the first evidence that the approach transfers to endoscopic surgery, where the visual ambiguity is substantially different (specular highlights that don't encode geometry, tissue deformation that provides ambiguous kinematic signal). The consistent policy improvement across architectures (GR00T N1.5 and π0.5) and training regimes suggests the IDM successfully handles these domain-specific challenges—an empirical finding that was not obvious a priori.

• Surgical policy learning (SRT, SRT-H, Long et al.) relied entirely on real demonstrations collected with kinematic recording. Cosmos-H-Surgical offers a complementary data source: synthetic video-action pairs generated from videos that already exist (SATA's 2,447 clips, plus the vast unlabeled surgical video on platforms like YouTube). The paper does not compare against these prior policy-learning methods directly—it uses a general-purpose VLA (GR00T N1.5) rather than a surgical-specific architecture—but the approach is orthogonal and could augment any imitation learning pipeline.

Which research directions become more attractive. The paper makes world model pretraining on domain-specific unlabeled video a high-priority investment for surgical robotics. The few-shot adaptation results (Table 2: SATA pretraining improves 5-trajectory success rate from 51.8% to 73.2%) demonstrate that general surgical video knowledge transfers to specific embodiments. This suggests that expanding SATA—covering more procedures, more action types, more anatomical contexts, and more visual variation—is a high-leverage activity that pays off across all downstream surgical robots, not just the specific platform tested. The paper also makes inverse dynamics model quality a central bottleneck: the pseudo-actions that label synthetic videos are only as good as the IDM, and the paper provides no direct IDM accuracy evaluation. Improving IDM robustness—through better architectures, more training data, or explicit handling of visual ambiguity—becomes a critical research direction.

Which directions become less attractive. The paper's results suggest that building physics-based surgical simulators with explicit soft-body dynamics may be a lower-priority investment than previously thought, at least for the purpose of generating policy training data. The world model approach produces photorealistic video without explicit tissue modeling, and the IDM extracts useful kinematic information from that video. If the pipeline works without simulating tissue mechanics, the enormous engineering effort required to build accurate soft-body simulators (which the paper notes "lacking soft body simulation, limiting policy transfer") may be better spent on curating larger surgical video datasets and improving generative model fidelity. This is not to say physics simulators are useless—they provide ground-truth physics that world models may hallucinate incorrectly—but the paper provides evidence that photorealistic video alone, without explicit physics, is sufficient for policy improvement.

A reconciliation of conflicting intuitions. There is a tension in surgical robotics between those who argue that surgical data is so scarce and expensive that learning-based approaches are fundamentally limited, and those who believe foundation models trained on diverse non-surgical data can transfer to surgery. Cosmos-H-Surgical partially reconciles these views: the world model transfers general video dynamics knowledge from Cosmos-Predict2.5's diverse pretraining, but requires domain-specific SATA fine-tuning to produce realistic surgical video; the IDM transfers general robot kinematics knowledge from the DreamGen Franka checkpoint, but requires embodiment-specific fine-tuning; and the policy transfers general manipulation knowledge from GR00T N1.5's pretraining, but requires surgical task-specific data. Transfer helps but is insufficient alone—domain adaptation is essential at every stage, and the amount of domain-specific data required is quantified (5–20 real demonstrations for the task studied).

---

Follow-Up Research This Work Enables

Direct IDM accuracy evaluation on real held-out trajectories. The paper uses the IDM as a black box and evaluates it only indirectly through downstream policy MSE. The most immediate follow-up would measure IDM prediction error directly: take the real demonstration trajectories (60 episodes total), hold out a subset, train the IDM on the remainder, and report the mean absolute error or MSE between IDM-predicted actions and ground-truth kinematics on the held-out set, broken out by cartesian position, rotation, and gripper angle. This would establish the fundamental accuracy ceiling for synthetic data: if the IDM's error on real video is X, then synthetic data labeled by the IDM cannot possibly provide supervision more accurate than X, regardless of world model quality. It would also reveal which action dimensions the IDM struggles with—the policy results in Figure 8 suggest cartesian position benefits most from synthetic data, while rotation and gripper show smaller gaps, which might reflect IDM accuracy differences across dimensions. This experiment requires zero additional data collection and would substantially clarify the pipeline's bottlenecks.

Policy trained on synthetic data only, without real fine-tuning. The paper's training protocol always includes a real-data fine-tuning stage (200 steps) after synthetic pretraining. An ablation that trains the policy exclusively on synthetic data—no real demonstrations whatsoever—would isolate the quality of the synthetic supervision. If a synthetic-only policy achieves non-trivial test-set performance (even if below Real Only), it validates the IDM's pseudo-actions as direct substitutes for real kinematics. If synthetic-only performance is near zero, it suggests the synthetic data provides useful regularization or representation learning but not accurate action supervision—the real-data fine-tuning stage is doing the heavy lifting of teaching correct kinematics, and the synthetic data merely provides a better initialization. The paper's current design cannot distinguish these mechanisms because synthetic pretraining and real fine-tuning are always combined. A synthetic-only baseline would resolve this ambiguity.

Synthetic data from different world model variants for policy training. The paper shows that Cosmos-H-Surgical generates better video than Action-Category and Zero-Shot baselines (Tables 1, 2), and that Cosmos-H-Surgical's synthetic data improves policies (Figure 8). But it does not establish a causal link: would synthetic data from the lower-quality Action-Category world model also improve policies? If yes, the mechanism might be data augmentation or regularization rather than video realism—any synthetic video, even with incorrect action execution, could improve policy robustness. If no, the mechanism is specifically tied to video quality, and improving world model fidelity directly translates to better policies. A simple experiment: generate synthetic data using the Action-Category world model (same 56 initial frames, same IDM), train policies on it with the identical protocol, and compare against Cosmos-H-Surgical synthetic data. This would quantify the downstream value of world model quality in units of policy MSE improvement.

Cross-task generalization: synthetic data from Task A improving policy on Task B. The current experiments test synthetic data from the needle pick-up and hand-over task, used to improve a policy for the same task. The scalability claim ("offers a scalable path toward autonomous surgical skill acquisition") implies that synthetic data generated for one task could benefit learning on another. A direct test: use the SATA-pretrained Cosmos-H-Surgical to generate synthetic videos for needle puncture (one of the four SATA action categories), label them with an IDM trained on the existing 66 out-of-domain episodes plus a small number of needle puncture demonstrations (if available), and measure whether this synthetic puncture data improves a policy trained on the original needle pick-up task. Positive transfer would demonstrate that the world model captures generalizable surgical manipulation knowledge; zero or negative transfer would bound the approach's generality. This experiment would also stress-test a key limitation the paper acknowledges: both the world model and IDM require embodiment-specific adaptation data, and it's unclear how much data is needed for a new task versus a new instantiation of the same task.

Stress-test: synthetic data from a physics simulator as a comparison baseline. The paper claims advantages over "synthetic physics-based simulators" that "suffer from a large visual and dynamic domain shift" (Section 1). A direct comparison would test this claim: generate synthetic needle pick-up data from an available surgical simulator (SurRoL or Surgical Gym, both cited in the paper), label it with the same IDM trained on real data, and compare policy performance against Cosmos-H-Surgical synthetic data at the same data volume. If simulator data underperforms, it validates the paper's motivation for world-model-based generation. If simulator data performs comparably or better (despite visual domain gap), it suggests the IDM is robust to visual appearance and the world model's photorealism is less critical than the paper implies. This experiment would also help practitioners decide whether to invest in world model training (requires SATA-scale dataset curation and GPU resources) or in simulator configuration (requires physics modeling but no training data).

IDM generalization across surgical robot embodiments. The paper acknowledges that the IDM requires embodiment-specific training data (Section 5). A natural stress test: train an IDM on the needle pick-up robot platform, then apply it to synthetic videos generated for a different surgical robot (e.g., the da Vinci Research Kit, widely used in surgical robotics research and for which public datasets exist). If the IDM's pseudo-actions remain useful—even with reduced accuracy—it suggests that inverse dynamics from endoscopic video transfers across embodiments, dramatically reducing the data needed for new robot platforms. If performance degrades to uselessness, it establishes that IDM training is tightly coupled to specific robot kinematics and that each new platform requires its own IDM training data, which bounds the approach's scalability. The paper's use of a DreamGen Franka checkpoint as IDM initialization provides weak evidence for cross-embodiment transfer (Franka is a tabletop robot arm, not a surgical system), but a within-domain cross-embodiment test would be far more informative.

Expanding SATA to cover procedural diversity and rare surgical events. The paper's SATA dataset covers four action categories from suturing across 8 procedure types, but this is a fraction of surgical practice. The compositional generalization demonstrated in Figure 5 (multi-step handover from single-step training data) suggests the world model can recombine learned primitives. A natural extension would test how far this generalizes by expanding SATA to include new action categories (cutting, cauterization, stapling, clip application) and new anatomical contexts (different organs, different tissue types) and measuring whether the world model can compose these into novel multi-action sequences without explicit multi-action training data. The practical question is: how many surgical primitives must be in the training set before the model can generate arbitrary compositions? Answering this would determine the scaling roadmap for SATA curation and, by extension, the ceiling on synthetic data diversity for policy training.

---

Practical Applications and Downstream Use Cases

Data augmentation for surgical VLA training at research labs and medical device companies. The most immediate application is straightforward: any group training imitation learning policies for surgical robots can use Cosmos-H-Surgical to generate additional training data from their existing real demonstrations. The pipeline requires collecting a modest number of real trajectories (the paper demonstrates benefit with as few as 5), training the world model and IDM on this data, generating synthetic rollouts, and adding them to the training set. The paper's results suggest this can meaningfully reduce trajectory prediction error (Figure 8) and that the benefit scales with synthetic data volume (Real + Synthetic 10× > Real + Synthetic). For institutions that have already invested in surgical robot hardware and data collection infrastructure, adding this synthetic data pipeline provides a pure software upgrade to their existing data without requiring additional operating-room time, surgeon supervision, or patient exposure. The key practical requirements are: access to a GPU capable of running Cosmos-Predict2.5 inference (the paper does not specify hardware, but video diffusion models in this class typically require high-end GPUs with substantial VRAM), and the ability to record ~60 real trajectories (the paper's training plus out-of-domain data) to train the IDM.

Pretraining surgical VLAs on synthetic data before real-robot deployment, reducing the need for initial real demonstrations. The paper's training protocol uses synthetic data for pretraining followed by real data for fine-tuning. This suggests a deployment workflow where a surgical robot is shipped with a VLA policy pretrained on synthetic data for common procedures, and the policy is then fine-tuned on a small number of site-specific demonstrations at the deploying hospital to adapt to local equipment, lighting, and surgeon preferences. The synthetic pretraining provides broad coverage of surgical scenarios (varied anatomy, varied tool configurations, varied action sequences) while the site-specific fine-tuning requires minimal additional data collection. The paper's results provide evidence that this two-stage approach works (Figures 8, 11–14), though the current experiments test only one task and one robot. Scaling this to cover the diversity of real surgical practice would require expanding SATA significantly and validating the approach on procedures beyond suturing-related actions, but the architecture supports this extension without modification.

Scaling surgical policy learning to procedures where real demonstration collection is ethically or practically impossible. Certain surgical scenarios—emergency procedures, rare complications, pediatric surgery, high-risk maneuvers—may never yield sufficient real demonstrations for imitation learning because collecting them would require exposing patients to unnecessary risk or because the scenarios occur too infrequently. The Cosmos-H-Surgical pipeline offers a path to generate training data for these scenarios synthetically: annotate existing videos of the rare procedure (from YouTube surgical channels, case reports, or surgical atlases) with SATA-style text descriptions, use the world model to generate diverse synthetic rollouts, label them with an IDM trained on whatever real data is available from related but safer procedures, and train a policy on the synthetic data. The paper's compositional generalization results (Figure 5, multi-step handover generated from single-step training data) provide preliminary evidence that the world model can generate novel action compositions, though whether this extends to generating entirely novel procedures (not just novel compositions of known primitives) is untested. The safety implications are significant: a policy trained entirely or primarily on synthetic data for a high-risk procedure would need rigorous validation before clinical deployment, but the synthetic data pipeline at least makes such training possible where it was previously infeasible.

Accelerating research on surgical skill assessment and autonomous execution benchmarks. The surgical AI community lacks standardized benchmarks for policy learning because real surgical data is difficult to share across institutions (patient privacy regulations, proprietary robot formats). Cosmos-H-Surgical could generate a publicly releasable benchmark dataset: synthetic videos with pseudo-kinematic labels for standardized surgical tasks, generated from a fixed set of initial frames and text prompts, using public SATA-derived world models and publicly released IDM checkpoints. Researchers could download the benchmark, train their policies on the synthetic data, and submit predictions for a held-out test set. This would dramatically lower the barrier to entry for surgical robotics research—no robot hardware, no IRB approval, no data-sharing agreements—while providing a reproducible evaluation standard. The paper's release of code and model weights on GitHub and Hugging Face (abstract links) is a step toward this vision, but a curated benchmark with defined train/test splits, standardized evaluation metrics (including task success rate, not just MSE), and baseline policy implementations would be a natural and impactful extension with minimal additional engineering beyond what the paper already provides.

---

When to Prefer This Method

The paper positions Cosmos-H-Surgical as a data generation approach for domains with abundant unlabeled video but scarce paired video-action data. It contrasts explicitly against two alternatives: collecting more real demonstrations (which it argues is prohibitively expensive in surgery) and using physics-based simulators (which it argues suffer from visual and dynamic domain gaps). The decision boundaries emerge from the paper's evidence and limitations:

• Prefer Cosmos-H-Surgical synthetic data generation when:

  • You have access to abundant unlabeled surgical video (existing public datasets, hospital recordings, YouTube surgical channels) that can be annotated with text descriptions for world model fine-tuning—the SATA dataset demonstrates this with 2,447 clips from 7 sources.
  • You can collect a small number of real robot demonstrations (5–20, based on the paper's experiments) for world model adaptation and IDM training on your specific embodiment—the few-shot adaptation results in Table 2 show SATA pretraining makes this small number effective.
  • The task involves visually observable tool-tissue interactions (grasping, puncturing, pulling, knotting) where the endoscopic view provides sufficient information to infer kinematics—the paper validates this for needle manipulation but does not test tasks where critical state is visually occluded.
  • The base policy architecture supports video input and can benefit from diverse visual training scenarios—validated for GR00T N1.5 and π0.5, likely applicable to any VLA.

• Prefer collecting more real demonstrations instead when:

  • The task involves actions where the endoscopic view is insufficient to determine kinematics (e.g., force-controlled manipulation where identical visual states can correspond to different applied forces, or tasks where critical information comes from haptic feedback not visible on camera).
  • The world model's generated videos consistently fail to capture task-critical visual details (tool occlusion, lighting variations, anatomical anomalies) that are present in real data—the paper does not characterize these failure modes, but they likely exist for complex or rare surgical scenarios.
  • You lack the computational resources to run Cosmos-Predict2.5 inference (the paper does not specify hardware requirements, but video diffusion models are computationally intensive) or the expertise to curate SATA-quality text annotations.

• Prefer physics-based simulators instead when:

  • The task requires ground-truth physics that a world model might hallucinate incorrectly—contact forces, tissue tearing, suture breakage, or other mechanical failures where photorealistic video alone provides no signal.
  • The domain gap between simulator visuals and real endoscopy is manageable (e.g., simple geometric environments, tasks where tool motion rather than tissue appearance drives policy learning) and you value the simulator's ability to provide unlimited ground-truth data with perfect kinematic labels.
  • You need to evaluate policy safety under worst-case or adversarial conditions that a data-driven world model cannot reliably generate—simulators can systematically explore edge cases.

The paper does not experimentally compare against these alternatives (the Action-Category and Zero-Shot baselines are variants of the same world model approach, not separate methods), so these decision rules are inferred from the paper's stated assumptions and limitations rather than validated by direct empirical comparison. A practitioner choosing between these approaches should treat the world model pipeline as promising but not yet benchmarked against existing alternatives, and should conduct their own comparison on their specific task and robot platform before committing.