RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning

ArXiv: 2602.18742

🎯 Pitch

Fake training videos for robots often come with wrong action labels that poison policy learning. RoboCurate solves this by replaying predicted actions inside a physics simulator and rejecting any video whose motion doesn’t match the replay, boosting real-world dexterous manipulation success by a staggering 179.9% on a humanoid robot. This simple simulation-consistency filter unlocks action-quality verification that vision-language models cannot provide, making synthetic data genuinely safe and effective for training.

---

1. Executive Summary

This paper introduces RoboCurate, a synthetic robot data generation framework that evaluates and filters the quality of IDM-predicted actions by replaying them in a simulator and measuring motion consistency between the simulator rollout and the generated video—a process termed simulator-replay consistency (matching a generated video against a physically grounded proxy rollout to catch physically implausible motion or incorrect action labels). The method also expands observation diversity beyond the available dataset through image-to-image editing and action-preserving video-to-video transfer, tested on the GR-1 Tabletop, DexMimicGen, and real-world ALLEX humanoid benchmarks using GR00T N1.5 as the base policy. RoboCurate yields relative improvements of +70.1% on GR-1 Tabletop (300 demos), +16.1% on DexMimicGen in the pre-training setup, and +179.9% on the real-world ALLEX humanoid dexterous manipulation setting, while also enabling emergent success on novel behavior tasks (from 0.0% to 25.0% on pour can), establishing that action-level filtering through simulator replay substantially outperforms VLM-based video plausibility filtering as a data curation strategy for policy learning.

2. Context and Motivation

The Core Problem: Synthetic Robot Data Is Plentiful but Unreliable

The fundamental tension this paper tackles is that robot learning is starved for data, but the most scalable source of that data—video generative models—produces outputs of inconsistent quality that can actively harm policy performance. This is a distinctly different problem from the more commonly discussed sim-to-real gap. The sim-to-real gap concerns visual and physical discrepancies between simulation and reality. Here, the problem is internal to the synthetic data pipeline itself: even when videos look plausible, the actions inferred from them may be physically wrong, and there has been no reliable way to detect this without ground-truth action labels.

The scale of the data scarcity problem is worth appreciating. The paper notes that robotics data is "far more limited than vision and language data due to costly and labor-intensive collection processes" (Section 1). Collecting even modest demonstration datasets—say, 48 trajectories for a single pick-and-place task on the ALLEX humanoid—requires substantial human teleoperation effort. Scaling this to the diversity of tasks, objects, and environments needed for general-purpose robot manipulation is economically and logistically prohibitive. This is the same bottleneck that drove computer vision and natural language processing toward large-scale web-scraped datasets, but robotics lacks an analogous internet-scale source of labeled action data.

This gap matters for two reasons, one practical and one scientific:

Practical impact. The paper demonstrates that synthetic data can teach a policy to perform entirely novel behaviors—tasks for which zero real demonstrations exist. On the ALLEX humanoid, the pour-can task goes from 0.0% success with real-data-only training to 25.0% when synthetic data is added (Table 3). This is not an incremental improvement on a known task; it is the emergence of a qualitatively new capability. If synthetic data pipelines could be made reliable, the cost structure of robot learning would fundamentally change: collecting 50 real demonstrations per task would no longer be the bottleneck, because novel behaviors could be bootstrapped from generated videos.

Scientific significance. The paper exposes a subtle failure mode in the dominant paradigm of neural trajectory generation. The standard pipeline—generate a video with an image-to-video model, then label it with an inverse dynamics model (IDM)—treats the IDM as a black-box annotator. But IDMs are themselves learned models, trained on limited real data. When they encounter generated videos that deviate from the training distribution (unusual object configurations, implausible hand trajectories, motion blur artifacts), their action predictions become unreliable. The paper's core insight is that the video and the actions can be mutually inconsistent in ways that neither a video-quality VLM nor the IDM itself can detect, because neither has access to a ground-truth physical reference. This is a principled limitation of any pipeline that relies solely on visual or learned signals for quality assessment.

Where Prior Approaches Fall Short

The paper identifies three categories of prior work, each with specific limitations that RoboCurate is designed to address.

Simulation-based synthetic data is visually impoverished and engineering-intensive. The established approach for scaling robot data is to generate it in simulation (e.g., MimicGen, RoboCasa). Simulation provides ground-truth actions and physically plausible dynamics, but at a steep cost: creating diverse, realistic simulated environments requires substantial manual effort (modeling objects, authoring scenes, tuning physics parameters), and the resulting data inevitably looks different from real-world camera feeds. This visual discrepancy—the sim-to-real gap—forces practitioners to either invest in domain randomization (randomizing textures, lighting, camera angles during training) or accept degraded real-world performance. The paper acknowledges simulation's value but positions neural trajectory as a complementary approach that trades perfect physics for photorealism: generated videos "are visually similar to real-world data than simulation" (Section 1) and do not require scene authoring.

Neural trajectory without curation produces marginal gains. The baseline neural trajectory pipeline—exemplified by DreamGen (the method of jang2025dreamgen)—generates videos from initial frames using an I2V model and labels them with an IDM. On GR-1 Tabletop with 300 demonstrations, this approach yields only a +26.6% relative improvement over real data alone (Table 1: 15.4% → 19.5%). On DexMimicGen, the gain is a mere +4.0% (Table 2: 44.6% → 46.4%). These are surprisingly weak results given the substantial investment in generating 10K synthetic trajectories. The paper's diagnosis is that uncurated neural trajectory contains two intertwined failure modes:

1. Video generation failures. The I2V model can produce physically implausible videos—objects overlapping unnaturally, hands teleporting, arms passing through tables. These are not subtle artifacts; they fundamentally violate the physics that the policy is trying to learn. Yet the video model has no internal mechanism to detect or correct these failures, because it is trained purely on pixel-level reconstruction objectives, not on physical consistency.

2. IDM prediction errors. Even when the video is physically plausible, the IDM may predict incorrect actions. This can happen for subtle reasons: the IDM may misperceive the robot's end-effector position due to motion blur, or it may fail to generalize to object configurations not seen in its (limited) real training data. Crucially, these errors are not detectable from the video alone—a video where the hand appears to reach the object correctly might have IDM actions that would cause the real robot to miss entirely.

VLM-based video quality filtering is too coarse. The most recent prior work attempts to address the quality problem by using vision-language models to judge whether generated videos follow instructions (DreamGenBench) or obey physical laws (VideoCon-Physics). The paper's ablation in Table 4 directly compares these approaches against RoboCurate's action-level filtering on GR-1 Tabletop:

The VLM-based methods provide a modest boost over uncurated data (+3.3 and +3.1 percentage points, respectively), but substantially underperform RoboCurate's action-level filtering (+6.2 points). The paper's explanation is that VLM judgments are "typically too coarse to capture task-critical motion for policy learning (e.g., whether the arm moves far enough to reach the object) and often stop at superficial compliance with basic physics without directly evaluating the actions themselves" (Section 1). A video can look physically plausible—no objects clipping through tables, no obvious teleportation—while still containing IDM actions that fail to execute the intended task. VLMs, which process frames as visual tokens and reason about them through language, lack the fine-grained motion understanding and the physical reference frame needed to catch these action-level errors.

The IDM bottleneck is underappreciated. A subtle but important point: the IDM is trained on real data, so its errors on synthetic videos are fundamentally a distribution-shift problem. The generated videos come from a different distribution than the real training videos (different lighting, textures, object configurations), and the IDM's predictions degrade under this shift. Prior work treats the IDM as a fixed, reliable component of the pipeline. RoboCurate is the first method to explicitly verify the IDM's outputs against a physically grounded reference, essentially asking: "If I executed these actions in a simulator, would the resulting motion match what the video shows?" This reframes the IDM from an unquestioned annotator to a hypothesis generator whose outputs must be validated.

How This Paper Positions Itself

RoboCurate is not a new video generation model, a new IDM architecture, or a new policy learning algorithm. It is a data curation framework that sits between generation and training, filtering and improving synthetic trajectories before they are fed to the policy. The paper's positioning has three key aspects:

First, it introduces simulator-replay consistency as a verification primitive. The core idea—replay predicted actions in a simulator and compare the resulting video to the generated video—is conceptually simple but was not previously used for synthetic data curation in robotics. This bridges two worlds that are typically separate: the neural world of video generation and IDM prediction, and the physical world of simulation. The simulator provides ground-truth physics for the actions, even though the generated video does not. By comparing the two videos, the system gains a reference signal that neither video alone contains.

Second, it treats diversity and quality as complementary, not competing, objectives. Prior work on neural trajectory focused primarily on generating diverse task scenarios via prompt engineering. RoboCurate explicitly expands this diversity along visual axes (scene appearance, object textures, lighting, backgrounds) through I2I editing and V2V transfer, while simultaneously filtering for quality. The diversity ablation (Table 5) shows that both task diversity (varying the combinatorial space of skills, objects, and placements) and visual diversity (varying the appearance of the same task) independently improve policy performance. This is a practically important finding: it means the community should not be satisfied with simply generating more videos of existing scenes, but should actively inject controlled visual variation.

Third, it demonstrates that action-level filtering enables emergent capabilities, not just incremental improvements. The most striking result is not the +70.1% relative improvement on GR-1 Tabletop—impressive as that is—but the fact that RoboCurate's data enables a policy to perform a novel behavior (pouring a can) with zero real demonstrations (Table 3: 0.0% → 25.0%). This suggests that curated synthetic data can cross a capability threshold that uncurated data cannot: the signal from correct actions on diverse visual scenes is strong enough to teach the policy a new motor skill, while uncured data's noise drowns out this signal. The paper's Best-of-N sampling variant (using the attentive probe's score to select the best among N generated videos for each task) is particularly important in this regime, since it avoids wasting the limited synthetic budget on low-quality samples in data-scarce fine-tuning settings.

A note on what the paper does not claim. RoboCurate does not eliminate the need for real data. The real demonstrations are essential for training the video generation model (which requires real initial frames and video dynamics), the IDM (which requires real action-labeled trajectories), and the attentive probe (which requires real simulator-replay pairs for training). The framework amplifies the value of existing real data by using it to validate synthetic data, but it is not a zero-shot or sim-only approach. This is a realistic and important constraint: the paper shows how to get more out of limited real data, not how to eliminate it entirely.

3. Technical Approach

3.1 Reader Orientation

RoboCurate is a data pipeline that sits between a video-generating AI model and a robot's policy-learning algorithm, generating synthetic robot videos, inferring what actions the robot should take from those videos, and then validating those inferred actions by physically simulating them and checking whether the simulated motion matches the original video. It solves the problem that AI-generated robot videos—while visually convincing—often contain physically impossible motion or mislabeled actions that would teach a real robot the wrong thing, yet prior methods had no way to catch these errors because they lacked a ground-truth physical reference. The shape of the solution is a generate-then-verify-then-curate loop: generate diverse videos and preliminary action labels, replay the actions in a physics simulator to produce a ground-truthed reference video, train a neural network to compare the original and simulated videos and detect mismatches, then discard or re-select synthetic trajectories based on this comparison before the policy ever sees them.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a pipeline, with two auxiliary diversity-expansion stages feeding into the generation stage:

1. Video Generation Model (fine-tuned Cosmos-Predict2-14B). Takes an initial frame (real or edited) and a language instruction as input, produces a synthetic video of a robot executing that instruction. This is an image-to-video diffusion model operating in a compressed latent space.

2. Diversity Engines (I2I Editing and V2V Transfer). Before video generation, an image-to-image diffusion model (FLUX.2-dev) edits the initial frame along controlled visual axes (table, object, lighting, background) while preserving scene structure via Canny edge conditioning. After generation, a video-to-video model (Cosmos-Transfer2.5-2B) transfers the visual appearance of successful videos while preserving the underlying motion, again using Canny edge video conditioning. Both expand the visual coverage of synthetic data without requiring new real initial frames.

3. Inverse Dynamics Model (IDM). A diffusion transformer that takes two frames from the generated video (current and future) and predicts the sequence of robot joint actions that would transform the scene from the first frame to the second. This is the pseudo-labeler: it converts action-free videos into action-labeled trajectories. Trained on real robot data, it serves as a bridge from vision to action.

4. Simulator Replay Engine. Executes the IDM-predicted actions in a physics simulator (using the same robot embodiment) and renders the resulting motion as a video. This is the ground-truth reference: the simulator guarantees that the robot motion in the rendered video is exactly the physical consequence of the actions. Any discrepancy between this simulator video and the original generated video signals either (a) the generated video has physically implausible motion, or (b) the IDM predicted incorrect actions.

5. Attentive Probe (Action Verifier). A lightweight trainable classifier built atop a frozen pre-trained video encoder (V-JEPA2, 0.3B parameters). It takes a pair of videos—the generated video and its corresponding simulator-replay video—and outputs a probability that the two videos are motion-consistent. Trained on automatically constructed positive and negative pairs from real robot data, it learns to detect subtle action-level mismatches that VLMs miss. The probe's output score determines whether a synthetic trajectory is kept or discarded (or, in Best-of-N mode, which of several generated trajectories is selected).

The information flow is: real initial frame → (optional) I2I editing → video generation conditioned on initial frame and VLM-generated task instruction → (optional) V2V transfer → IDM pseudo-labeling → simulator replay of IDM actions → attentive probe comparison of generated video and simulator video → accept/discard decision → accepted trajectories join the policy training dataset.

3.3 Roadmap for the Deep Dive

• First, the video generation model and its flow-matching training objective, since all synthetic data originates here and the choice of diffusion framework affects downstream signal quality.
• Second, the diversity expansion mechanisms (I2I editing and V2V transfer), because they are the key architectural additions that differentiate RoboCurate's data from simpler I2V-only generation, and their design choices (Canny edge conditioning, prompt axes) determine what visual variation the policy can learn.
• Third, the inverse dynamics model, since it bridges the generated video and the actions that must be verified—understanding what the IDM computes and why it can fail is essential to understanding what the attentive probe must detect.
• Fourth, the simulator-replay verification mechanism and the attentive probe, which is the paper's core technical contribution: converting action verification into video-comparison classification, constructing training pairs without human labels, and the cross-attention architecture that enables fine-grained motion comparison.
• Fifth, the Best-of-N sampling variant, which applies the probe during generation rather than post-hoc filtering, critical for the data-scarce co-finetuning regime.
• Sixth, the post-processing pipeline (instruction following and trajectory plausibility checks using VLMs), which acts as a coarse pre-filter before the attentive probe's fine-grained action-level verification.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and data curation paper whose core idea is that the quality of synthetic robot trajectories can be verified by replaying predicted actions in a physics simulator and checking for motion consistency between the generated video and the simulator rendering, and that this verification—combined with controlled visual diversification—produces synthetic training data that substantially outperforms both real-data-only baselines and uncurated synthetic data.

---

The Video Generation Model: Cosmos-Predict2-14B with Flow Matching

The neural trajectory pipeline begins with video generation. The paper fine-tunes Cosmos-Predict2-14B, a pre-trained video diffusion model, on robot manipulation data (ActionNet and GR00T-GR1-100 for pre-training experiments, manually collected ALLEX data for co-finetuning). This model operates in a compressed latent space rather than pixel space, following the standard latent diffusion architecture where an encoder $\mathcal{E}$ maps pixel-space video $\mathbf{w}$ to a latent representation $\mathbf{x} = \mathcal{E}(\mathbf{w})$. All generation and training occur in this lower-dimensional latent space, with a decoder reconstructing pixels at the end.

The model uses the Flow Matching framework rather than the more common denoising diffusion probabilistic model (DDPM) formulation. The key difference: instead of learning to predict the noise that was added to a clean signal (the DDPM approach), flow matching defines a deterministic linear path between the data and Gaussian noise, and trains the model to predict the velocity field along this path—essentially, the direction and magnitude of the "flow" needed to transport samples from noise to data.

Formally, given a clean latent video $\mathbf{x}$ and Gaussian noise $\boldsymbol{\epsilon}$, the intermediate noisy latent at timestep $t \in [0, 1]$ is defined as a straight-line interpolation:

$\mathbf{x}_t = (1 - t)\mathbf{x} + t\boldsymbol{\epsilon}$

where $\mathbf{x}_t \in \mathbb{R}^{d}$ is the noisy latent at interpolation fraction $t$, $\mathbf{x} \in \mathbb{R}^{d}$ is the clean latent (the encoded video), and $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ is isotropic Gaussian noise of the same dimensionality.

What it computes: a weighted blend where at $t = 0$ the sample is the clean data, at $t = 1$ it is pure noise, and at intermediate $t$ it is a mixture. The path is linear in the latent space, meaning the corruption schedule adds noise uniformly rather than through a non-linear variance schedule as in standard diffusion.

Why this form: the linear interpolation is the defining characteristic of flow matching (specifically, the conditional flow matching variant of Lipman et al., 2022). It defines a straight probability path between the data distribution and the noise distribution, which simplifies the training objective compared to standard diffusion's curved paths. This straight-line property means the model can use fewer sampling steps at inference while maintaining quality, though the paper does not explicitly ablate this choice against alternatives.

The model itself is parameterized as a velocity predictor $v_\theta$ trained to regress the target velocity field $\mathbf{u}_t = \boldsymbol{\epsilon} - \mathbf{x}$. The training objective is mean squared error between predicted and target velocity:

$\mathcal{L}(\theta) = \mathbb{E}_{t,\mathbf{x},\boldsymbol{\epsilon},\mathbf{c}}\left[\|v_\theta(\mathbf{x}_t, t, \mathbf{c}) - (\boldsymbol{\epsilon} - \mathbf{x})\|^2_2\right]$

where $v_\theta$ is the velocity-predicting neural network, $\mathbf{x}_t$ is the noisy latent at timestep $t$, $\mathbf{c}$ denotes conditioning information (the initial frame image and task instruction text), and $\boldsymbol{\epsilon} - \mathbf{x}$ is the target velocity—literally the vector pointing from clean data to pure noise. The expectation is taken over the distribution of timesteps, clean data samples, noise samples, and conditioning variables.

What it computes: for each training sample, a noisy latent is created at a random timestep by blending clean data and noise. The model sees this noisy latent along with the conditioning (initial frame + instruction) and predicts the velocity $\hat{v} = v_\theta(\mathbf{x}_t, t, \mathbf{c})$. The loss penalizes the squared Euclidean distance between this prediction and the true velocity $\boldsymbol{\epsilon} - \mathbf{x}$, which is simply the direction from data to noise. Minimizing this loss trains the model to estimate, at any point along the noise-adding trajectory, the instantaneous direction and speed needed to move toward the data distribution.

Why this form: predicting velocity rather than noise or clean data directly has specific advantages in the flow matching framework. At inference time, sampling proceeds by solving an ordinary differential equation (ODE) from $t = 1$ (noise) to $t = 0$ (data), integrating the predicted velocity field. MSE on velocity is the natural maximum-likelihood objective for this ODE-based generative model because it encourages the model to learn a smooth, well-behaved vector field that correctly transports samples. If the model instead predicted the noise (as in DDPM), the relationship between the prediction and the sampling ODE would be indirect, requiring a score-function transformation. The velocity prediction directly feeds into the ODE solver.

The paper does not provide specific hyperparameters for the video generation fine-tuning (learning rate, batch size, number of steps), but specifies that the model is fine-tuned from a pre-trained checkpoint on the ActionNet and GR00T-GR1-100 datasets. Generation is conditioned on an initial frame (from the relevant dataset) and a VLM-generated task instruction (prompt details in Appendix C.1). Videos are generated at 256×256 resolution.

---

Visual Diversity Expansion: Image-to-Image Editing and Video-to-Video Transfer

The visual diversity pipeline is a two-stage augmentation that RoboCurate adds to the base neural trajectory generation pipeline. The key design principle is controllable variation: rather than generating random visual changes that might break task semantics, the system injects variation along explicitly specified axes while using edge-map conditioning to preserve scene structure and motion.

Image-to-Image (I2I) Editing with FLUX.2-dev. Before video generation, the initial frame—the single image that seeds the image-to-video model—can be edited to introduce visual diversity. The paper uses FLUX.2-dev, a diffusion-based image editor, conditioned on a Canny edge map of the original image. The edge map is computed by applying the Canny edge detection algorithm to the original frame; it captures the structural outlines (edges of the table, robot arm, objects) while discarding texture, color, and lighting information.

The editing process works as follows. First, a VLM is queried to produce a detailed text description of the initial image. This description and the original task instruction are combined to identify which elements in the scene are the target object and the manipulation context. Using this context, the system generates multiple edited variants of the image along four axes: (1) table appearance (changing the table's texture, color, or material), (2) target object identity and appearance (e.g., swapping a red cube for a blue cylinder or a wooden block), (3) lighting conditions (changing the direction, intensity, or color temperature of illumination), and (4) background environment (changing wall colors, adding or removing background objects). The specific prompts for each axis are detailed in Appendix C.3, but the general structure is: the VLM receives the original scene description, the target object identity, and the axis to vary, and produces a modified description that differs only along that axis. FLUX.2-dev takes the original image and the modified description as input, conditioned on the original Canny edge map to preserve structural layout, and outputs the edited image.

Why Canny edge conditioning matters. Without edge conditioning, the diffusion model might change the spatial layout of the scene—the table could shift position, the robot arm could move, objects could relocate. Any such spatial change would make the original task instruction invalid for the edited frame (a "pick up the cube" instruction is useless if the cube is no longer in the scene). The Canny edge map acts as a structural constraint: it tells the diffusion model "keep the edges where they are, but feel free to change what's inside them." This ensures the edited frame remains a physically valid starting state for the same manipulation task. The paper's prompt design further enforces this by querying the VLM to "identify the target object and relevant manipulation context," so the editing instructions are anchored to task-relevant elements.

After editing, each edited initial frame is fed into the image-to-video diffusion model with task instructions, producing a video that is visually distinct from the original but structurally identical.

Video-to-Video (V2V) Transfer with Cosmos-Transfer2.5-2B. While I2I editing modifies the starting frame to create diversity, V2V transfer modifies the entire video to augment appearance while preserving motion. This is a crucial distinction: V2V transfer does not change what the robot does—it changes what the scene looks like while the robot does it.

The V2V model, Cosmos-Transfer2.5-2B, takes a previously generated video (one that passed quality checks or was selected) and transfers its visual appearance. Like I2I editing, it is conditioned on Canny edge maps—but now on edge video rather than edge images, meaning the structural conditioning spans the entire temporal sequence. This ensures the robot's motion, object trajectories, and spatial relationships are preserved frame by frame. The appearance transfer operates along the same four axes as I2I editing (table, object, lighting, background) using prompts analogous to the image editing prompts (Appendix C.4), but with an additional constraint: the robot's color is explicitly preserved, since the embodiment identity should remain unchanged across visual variations.

An important design choice: object identity is held constant. The paper specifies that for V2V transfer, the system "keep object identity and shape unchanged and only modify texture and color." This means a red cube remains a cube—it just becomes a differently textured or shaded cube. This constraint enables action reuse: because the object's physical properties (shape, size) are unchanged, and the motion is edge-preserved, the IDM-predicted actions from the original video remain valid for the transferred video. If the V2V transfer changed the object's shape (e.g., from a cube to a sphere), the grasping action that worked for the original object would no longer be geometrically valid for the transferred one, and the action labels would be incorrect. By restricting variation to appearance (texture, color) rather than geometry, the system can safely reuse actions without re-running the IDM.

Generation ratio. The paper generates approximately 10K augmented tabletop manipulation videos with an I2I:V2V ratio of 2:1, chosen "for efficiency given the generation cost" (Appendix B). V2V transfer processes an already-generated video, so it has higher computational cost per output than I2I editing (which only processes a single frame). The 2:1 ratio balances diversity from new initial scenes (I2I) against diversity from appearance transfer (V2V).

Task instruction diversity as a complementary dimension. Alongside visual diversity, the paper expands the set of task instructions used to condition video generation. A proprietary VLM (Gemini 3 Pro) generates plausible task instructions from initial frames using few-shot prompting with examples from the original dataset—this ensures the VLM produces instructions in the same format and level of detail as the real data. Instructions vary along four axes: (1) behavior type (pick, place, push, open, close), (2) target object identity, (3) placement location, and (4) robot hand type. For bimanual manipulation tasks, the system explicitly maintains a 1:1 ratio of left-hand to right-hand active instructions. The "hand type" axis is important because the policy must learn to coordinate both hands from data that specifies which hand performs which action.

The task diversity is measured combinatorially: each unique combination of skill, target object, placement, and hand type constitutes a distinct task. The paper's ablation (Table 5) varies the percentage of this combinatorial space covered—25%, 50%, and 100%—and shows monotonic improvement in policy performance with coverage.

---

The Inverse Dynamics Model (IDM): Pseudo-Action Labeling

The IDM is the component that converts action-free generated videos into action-labeled trajectories. Without it, synthetic videos would be visually useful but would lack the action supervision needed for imitation learning.

Architecture. The IDM follows the Diffusion Transformer (DiT) architecture. DiT adapts the standard transformer architecture—which processes sequences of tokens through self-attention and feed-forward layers—to operate on noised action sequences rather than text or image tokens. The model takes two frames as input: a current observation $x_t$ and a future observation $x_{t+H}$, where $H$ is the action horizon (the number of future timesteps to predict actions for). It outputs a sequence of intermediate actions $a_{t:t+H-1}$, representing the joint-space commands (positions, velocities, or torques for each robot joint) that would transform the scene from $x_t$ to $x_{t+H}$.

The DiT is trained with flow matching, using the same framework as the video generation model but applied to action sequences rather than video latents. Given a clean action chunk $A_t = a_{t:t+H-1}$ from real data, noise is added via $A_t^\tau = \tau A_t + (1 - \tau)\boldsymbol{\epsilon}$ for $\tau \sim \mathcal{U}(0, 1)$, and the model predicts the velocity $\boldsymbol{\epsilon} - A_t$. The conditioning is the pair of frames $(x_t, x_{t+H})$.

Why DiT + flow matching for actions. Actions are high-dimensional continuous sequences (44-dim joint space for GR-1, 48-dim for ALLEX), making them naturally suited to diffusion-based generation—the model can express multi-modal action distributions (e.g., two different valid ways to grasp an object) that a deterministic regressor would collapse to the mean. The transformer backbone processes the entire action sequence jointly with attention, enabling the model to learn temporal coherence (actions should be smooth, not jerky) and coordination between joints. Flow matching provides a simpler sampling procedure than DDPM, which matters during inference when the IDM must run on every frame pair of every generated video.

Training data and the distribution-shift problem. The IDM is trained exclusively on real robot data—for pre-training experiments, the open-source checkpoint is trained on ActionNet (GR-1 embodiment); for co-finetuning, the IDM is trained from scratch on the manually collected ALLEX dataset (48 demonstrations, batch size 64, 20K gradient steps). This creates the core fragility of the neural trajectory pipeline: the IDM is trained on the real data distribution, but at inference time it encounters generated videos that are visually different (due to I2I editing and V2V transfer, but also simply due to the video model's inherent generation artifacts). The IDM's predictions on these out-of-distribution frames become unreliable.

Why the IDM can fail silently. Unlike a video classifier that outputs a discrete label (and thus has a clear confidence signal if the softmax is diffuse), the IDM outputs a deterministic (after denoising) action sequence. There is no built-in uncertainty estimate. A drastic failure—the IDM predicting a hand trajectory that passes through the table—produces actions that look numerically plausible (they're in the right format and value range) but are physically impossible. A subtle failure—the IDM predicting a grasp position that is 2 cm off from the object's true location—produces actions that would cause the real robot to miss the grasp entirely, but would look nearly identical to the correct actions when plotted. Neither failure mode can be detected from the video alone, because the video doesn't contain ground-truth actions to compare against. This is why the simulator replay and attentive probe are necessary: they provide an external physical reference that reveals these failures.

---

Simulator-Replay Verification and the Attentive Probe

This is the paper's central technical contribution: converting the action-verification problem into a video-comparison classification problem that can be trained without human labels.

Step 1: Simulator replay of IDM actions. Given a synthetic trajectory sample consisting of a generated video $\mathbf{w}_{\text{gen}}$ and its IDM-predicted actions $a_{\texttt{IDM}}$, the system replays $a_{\texttt{IDM}}$ in a physics simulator and renders the resulting motion as a video $\mathbf{w}_{\text{sim}}(a_{\texttt{IDM}})$. The simulator uses the same robot embodiment (GR-1 humanoid or ALLEX humanoid) and the same camera viewpoint as the generated video. Critically, the simulator guarantees that the robot motion in $\mathbf{w}_{\text{sim}}(a_{\texttt{IDM}})$ is the exact physical consequence of executing $a_{\texttt{IDM}}$—there is no sim-to-real gap in the verification step, because the comparison is between two synthetic videos (one generated, one simulated), not between simulation and reality.

If the generated video is physically plausible and the IDM actions are correct, then $\mathbf{w}_{\text{gen}}$ and $\mathbf{w}_{\text{sim}}(a_{\texttt{IDM}})$ should show the same robot motion: the hand should move along the same trajectory, the object should be grasped at the same frame, the placement should occur at the same location. If the generated video shows implausible motion (hand teleporting, objects deforming) or the IDM predicted incorrect actions (hand reaching the wrong position), the two videos will diverge.

Why this converts the problem. Action verification originally required knowing the ground-truth actions for the generated video, which by definition don't exist for synthetic data. The simulator replay replaces ground-truth actions with ground-truth physics: the simulator's rendering is guaranteed to be consistent with $a_{\texttt{IDM}}$, so any mismatch with $\mathbf{w}_{\text{gen}}$ implicates either the video generation (the video is physically wrong) or the IDM (the actions are wrong). The system doesn't need to know which is the culprit; it only needs to detect that the pair is inconsistent and discard it.

Step 2: Training the attentive probe. Comparing two videos for motion consistency is itself a non-trivial perceptual task. Simple similarity metrics—pixel-wise MSE, structural similarity (SSIM), or cosine similarity between embeddings from a pre-trained video encoder—capture appearance similarity but are easily fooled: two videos can have identical motion but very different appearances (due to V2V transfer or different lighting), producing low similarity scores and false rejections. Conversely, two videos can have similar appearance but subtly different motion (the hand reaching 3 cm too far), producing high similarity scores and false acceptances.

The paper addresses this by training a lightweight classifier—the attentive probe—on top of a frozen pre-trained video encoder. The probe learns to specifically attend to motion-consistency cues while ignoring appearance differences.

Video encoder. The backbone is V-JEPA2, a pre-trained video encoder with 0.3B parameters. V-JEPA2 (Video Joint Embedding Predictive Architecture) is trained with a self-supervised objective: given a masked video, predict the latent representations of the masked regions from the visible regions. This pre-training encourages the model to learn structured video representations that capture motion and temporal dynamics, not just static appearance. The paper freezes the V-JEPA2 encoder during probe training, meaning its weights are never updated—the probe only learns the classifier head on top of the frozen features.

For a video pair $(\mathbf{w}_1^{t_1:t_1+H}, \mathbf{w}_2^{t_2:t_2+H})$ where each video is a clip of length $H$ frames, the encoder computes embeddings:

$\mathbf{z}_1 = f_\phi(\mathbf{w}_1^{t_1:t_1+H}), \quad \mathbf{z}_2 = f_\phi(\mathbf{w}_2^{t_2:t_2+H})$

where $f_\phi$ is the frozen V-JEPA2 encoder with parameters $\phi$, $\mathbf{w}_1^{t_1:t_1+H}$ is the first video clip starting at time $t_1$ and spanning $H$ frames, $\mathbf{w}_2^{t_2:t_2+H}$ is the second clip (potentially from a different time or episode), $\mathbf{z}_1 \in \mathbb{R}^{d}$ and $\mathbf{z}_2 \in \mathbb{R}^{d}$ are the resulting fixed-dimensional embeddings.

What it computes: the encoder processes each video clip independently, producing a single vector that summarizes the video's content (appearance, motion, spatial layout) into a compact representation. The dimensionality $d$ is determined by the V-JEPA2 architecture (not specified in the paper, but typical for a 0.3B video model it would be in the range of 512–1024).

Why frozen encoder: freezing the backbone serves two purposes. Practically, it makes probe training fast and data-efficient—only the small classifier head needs optimization. More importantly, it prevents the encoder from adapting to the specific appearance of simulator-rendered videos, which would defeat the purpose. If the encoder could learn to recognize simulator videos by their rendering artifacts (e.g., lack of motion blur, uniform lighting), it would learn a shortcut that doesn't generalize to motion comparison. By freezing V-JEPA2's pre-trained features, the probe is forced to work with representations that were learned from natural videos and are therefore appearance-agnostic enough to compare across rendering domains.

Cross-attention architecture. The embeddings $\mathbf{z}_1$ and $\mathbf{z}_2$ are concatenated—not simply added or subtracted—and fed to an attention-based probe $g_\theta$. The probe consists of a single cross-attention layer with a learnable query token, followed by a linear head:

$\ell = g_\theta([\mathbf{z}_1, \mathbf{z}_2])$

where $g_\theta$ is the probe with trainable parameters $\theta$, $[\mathbf{z}_1, \mathbf{z}_2]$ denotes concatenation of the two embeddings into a single sequence (the two embeddings serve as key-value tokens), and $\ell \in \mathbb{R}$ is a scalar logit indicating alignment.

The cross-attention mechanism works as follows: the learnable query token (a vector initialized randomly and updated during training) attends to both video embeddings simultaneously. By computing attention weights between the query and each embedding, the probe can learn to focus on the features that are most relevant for comparing the two videos—presumably, features related to robot motion, object displacement, and temporal correspondence, while ignoring features related to lighting, texture, or background differences. The cross-attention output is passed through a linear layer that maps to a single scalar logit.

The probability of the pair being aligned (motion-consistent) is computed by passing the logit through a sigmoid:

$p = \sigma(\ell) = \frac{1}{1 + e^{-\ell}}$

where $p \in [0, 1]$ is the predicted alignment probability, and $\sigma(\cdot)$ is the sigmoid function that squashes the unbounded logit to a probability.

What it computes: the probe takes two video embeddings produced independently by the frozen encoder, processes them jointly through cross-attention to find correspondences between the two representations, and outputs a single number between 0 and 1 representing its confidence that the two videos depict the same physical motion.

Why cross-attention over alternatives. A simpler alternative would be to compute cosine similarity between $\mathbf{z}_1$ and $\mathbf{z}_2$ and threshold it. The paper's ablation (Table 6) shows this approach ("V-JEPA2 without attentive probe") yields 23.8% success rate vs. 26.2% for the attentive probe. The problem is that cosine similarity weights all embedding dimensions equally, so appearance differences between the generated video and the simulator-rendered video (which looks different in lighting, texture, and rendering style) dominate the similarity score, drowning out the motion signal. The cross-attention mechanism allows the probe to learn a weighted combination of features: it can learn to attend strongly to dimensions that encode motion and weakly to dimensions that encode appearance. A more sophisticated alternative—computing similarity after projecting both embeddings through a learned MLP—would also help, but cross-attention with a learnable query is particularly expressive because the query can dynamically focus on different aspects of the two videos depending on their content (e.g., attending to the hand region if the hand is moving, attending to the object region if the object is being manipulated).

The probe has 8 cross-attention heads ($n_{\text{heads}} = 8$), a single cross-attention layer ($L = 1$), and is trained with binary cross-entropy loss.

Training data construction without human labels. The critical practical challenge is: how do you get training data for a motion-consistency classifier without manually labeling thousands of video pairs? Human labeling of subtle motion mismatches (e.g., "did the hand reach 3 cm too far?") is extremely difficult and unreliable—the paper's ablation (Table 6, "with human labels") shows that probes trained on 2K manually binary-labeled video pairs underperform even the simple cosine-similarity baseline, achieving 23.5% vs. 23.8%. The paper instead constructs positive and negative pairs automatically from real-world demonstrations, where the ground-truth actions are known.

The construction uses real videos $\mathbf{w}_{\text{real}}$ paired with their known action sequences $a_{\text{real}}$. For each real action $a_{\text{real}}$, the system renders a simulator rollout video $\mathbf{w}_{\text{sim}}(a_{\text{real}})$. By definition, this pair is perfectly aligned: the simulator video was generated from the exact same actions as the real video, so the motion should be identical.

Positive pairs. For each episode in the real dataset, at each starting time $t$, the system extracts a clip of length $H$ (16 frames) from both the real video and the corresponding simulator video:

$\mathcal{P}^+ = \left\{ \left( \mathbf{w}_{\text{real}}^{t:t+H}, \ \mathbf{w}_{\text{sim}}(a_{\text{real}})^{t:t+H} \right) \right\}_t$

where $\mathcal{P}^+$ is the set of positive pairs, $\mathbf{w}_{\text{real}}^{t:t+H}$ is the real video clip from time $t$ to $t+H$, and $\mathbf{w}_{\text{sim}}(a_{\text{real}})^{t:t+H}$ is the corresponding simulator clip covering the same temporal window. The temporal alignment is exact—both clips start and end at the same relative times within the episode.

Negative pairs (Type 1): Temporally shifted. For each episode, the system pairs a real clip at time $t$ with a simulator clip at a different time $t' \neq t$ from the same episode:

$\mathcal{P}^{-}_{\text{shift}} = \left\{ \left( \mathbf{w}_{\text{real}}^{t:t+H}, \ \mathbf{w}_{\text{sim}}(a_{\text{real}})^{t':t'+H} \right) \mid t' \neq t \right\}$

where $\mathcal{P}^{-}_{\text{shift}}$ is the set of temporally shifted negative pairs. Because the robot is in different poses at times $t$ and $t'$, the motion in the two clips will be different—the hand will be at different positions, the object at different stages of manipulation, etc. This teaches the probe to detect temporal misalignment: the two videos should show the same motion at the same relative time.

Negative pairs (Type 2): Cross-episode. The system pairs a real clip from one episode with a simulator clip from a different episode, at the same temporal index $t$:

$\mathcal{P}^{-}_{\text{cross}} = \left\{ \left( \mathbf{w}_{\text{real}}^{t:t+H}, \ \mathbf{w}_{\text{sim}}({a}'_{\text{real}})^{t:t+H} \right) \mid a'_{\text{real}} \neq a_{\text{real}} \right\}$

where $\mathcal{P}^{-}_{\text{cross}}$ is the set of cross-episode negatives. The two episodes involve different actions—perhaps different tasks, different objects, or different motion trajectories—at the same temporal offset. This teaches the probe to detect task-level motion differences: even if the timing is aligned, the movement should be the same action, not just any action.

The complete training set is $\mathcal{P} = \mathcal{P}^+ \cup \mathcal{P}^{-}_{\text{shift}} \cup \mathcal{P}^{-}_{\text{cross}}$. The union of both negative types ensures the probe learns to detect both temporal misalignment and episode mismatch.

Binary cross-entropy training. The probe is trained to classify each pair as aligned ($y = 1$) or misaligned ($y = 0$):

$\mathcal{L}(\theta; \mathcal{P}) = \mathbb{E}_{((\mathbf{w}_1, \mathbf{w}_2), y) \sim \mathcal{P}}\left[-y\log p - (1-y)\log(1-p)\right]$

where $p = \sigma(g_\theta([f_\phi(\mathbf{w}_1), f_\phi(\mathbf{w}_2)]))$ is the probe's predicted alignment probability, and $y \in \{0, 1\}$ is the ground-truth label ($y = 1$ only for pairs in $\mathcal{P}^+$, $y = 0$ otherwise).

What it computes: the standard binary cross-entropy loss. For positive pairs, the term $-\log p$ heavily penalizes the probe if it predicts low probability of alignment. For negative pairs, the term $-\log(1-p)$ penalizes the probe if it predicts high probability of alignment. The expectation averages this loss over all training pairs.

Why this form: binary cross-entropy is the maximum-likelihood objective for a Bernoulli random variable, which is appropriate because the alignment label is binary (the videos either show the same motion or they don't). The alternative—regression on a continuous similarity score—would require defining a meaningful distance metric on video pairs, which is exactly what the probe is supposed to learn. Classifying binary aligned/misaligned is a simpler and better-defined learning problem.

Why synthesized training pairs beat human labels. The paper's finding that human-labeled data underperforms the automatically constructed data (Table 6: 23.5% vs. 26.2%) is important and counterintuitive. The likely explanation is label noise versus label precision. Human labelers, asked to judge "does this generated video show the same motion as this simulator video?", must make subtle comparisons—is the hand 5 cm off or 10 cm off? Does that count as a mismatch? Different labelers will have different thresholds, and the same labeler may be inconsistent across examples. The synthesized training data has perfect labels by construction: positive pairs are guaranteed aligned (same actions, same timing), temporally shifted negatives are guaranteed misaligned (same actions, different timing), cross-episode negatives are guaranteed misaligned (different actions). There is zero label noise, which allows the probe to learn the precise decision boundary.

The training uses the following hyperparameters (Table 7): AdamW optimizer, learning rate 1e-4, batch size 32 (pairs), maximum 50 epochs, clip length $H = 16$ frames, temporal stride 4 frames (meaning clips are sampled every 4 frames along the video), input resolution 256×256 pixels.

Step 3: Filtering at inference time. At inference time, given a neural trajectory sample $(\mathbf{w}_{\text{gen}}, a_{\texttt{IDM}})$, the system:

1. Renders $\mathbf{w}_{\text{sim}}(a_{\texttt{IDM}})$ by executing $a_{\texttt{IDM}}$ in the simulator.
2. Forms the video pair $(\mathbf{w}_{\text{gen}}, \mathbf{w}_{\text{sim}}(a_{\texttt{IDM}}))$.
3. Feeds the pair through the frozen V-JEPA2 encoder and the trained attentive probe.
4. Obtains the alignment probability $p$.
5. Retains the sample only if $p$ exceeds a threshold $c$.

The paper does not explicitly state the threshold $c$ or how it was chosen. Based on standard practice for binary classifiers used as filters, it is likely set by sweeping values on a validation set to balance precision (proportion of kept samples that are actually good) and recall (proportion of good samples that are kept), or simply set at 0.5 (the default decision boundary for binary cross-entropy trained classifiers).

Why this filtering is action-level, not video-level. The key difference from VLM-based approaches is in what is being compared. A VLM sees only $\mathbf{w}_{\text{gen}}$ and judges whether it "looks physically plausible." The attentive probe sees $\mathbf{w}_{\text{gen}}$ and $\mathbf{w}_{\text{sim}}(a_{\texttt{IDM}})$ and judges whether they show the same motion. The simulator video contains the ground-truth physical consequence of $a_{\texttt{IDM}}$, so a mismatch directly implicates the combination of video quality and action correctness. A video that looks perfectly plausible but has wrong actions will be caught because its simulator replay won't align with the generated motion. A video with implausible physics will be caught because the simulator (which enforces physics) will produce a different motion than the generated video.

---

Best-of-N Sampling: Using the Probe as a Generation-Time Critic

In data-scarce settings—specifically, the co-finetuning experiments on the ALLEX humanoid where only 48 real demonstrations exist for the in-distribution task and 50 synthetic episodes are generated per task—it is wasteful to generate data and then discard a fraction of it. The paper introduces a variant where the attentive probe is used during generation rather than as a post-hoc filter.

Procedure. For each target task, the system generates $N$ candidate videos (with different random seeds for the Gaussian noise used in the diffusion sampling process, producing different videos from the same initial frame and instruction). Each candidate video is paired with its IDM-predicted actions and scored via the attentive probe (i.e., the alignment probability $p$ between the generated video and its simulator replay). The system selects the single candidate with the highest alignment score and adds it to the training dataset.

This Best-of-N variant ensures that every generation slot produces exactly one curated trajectory for the policy, with no data discarded. It trades off diversity (all $N$ candidates come from the same initial frame and instruction, varying only through stochastic generation) for quality (the selected candidate is the one most likely to have correct actions).

Why this matters for co-finetuning. In the pre-training setting, the system generates 10K videos and can afford to discard low-quality ones—the policy sees many diverse examples even after filtering. In co-finetuning, the synthetic budget is tiny (48–50 episodes per task), so discarding even 20% of generated data would leave insufficient data to learn novel behaviors. Best-of-N with $N > 1$ effectively multiplies the generation budget: generating 50 videos with $N = 4$ requires 200 total generations but produces 50 curated trajectories. The paper does not specify the exact $N$ used, but the results in Table 3 (47.9% success on seen tasks, 43.8% on novel objects, 25.0% on novel behaviors) suggest the selection effectively improves trajectory quality in this extreme low-data regime.

Connection to the broader idea. Best-of-N sampling is an instance of using a learned critic (here, the attentive probe's alignment score) to guide a generative process at inference time—a common pattern in language model decoding and diffusion model guidance. The probe's score serves as a measure of "how physically consistent and correctly labeled is this synthetic trajectory?", which correlates with downstream policy learning utility.

---

Post-Processing Pipeline: VLM-Based Pre-Filtering

Before the attentive probe performs action-level verification, the paper applies two VLM-based post-processing steps that filter on video-level semantics and coarse physical plausibility. These are described in Appendix B as "inspired by" the GR00T N1 open foundation model.

Step 1: Instruction following and re-captioning. The initial task instructions generated by the VLM can be ill-posed (physically impossible, ambiguous, or inconsistent with the scene). Moreover, the generated video may fail to follow the instruction (e.g., the instruction says "pick up the red cube" but the video shows picking up a blue sphere). The system queries a VLM (Gemini 3 Pro) with 8 video frames and the original instruction to:

1. Identify videos conditioned on implausible instructions and discard them.
2. For videos that correctly execute a different action than instructed, re-caption them with an updated instruction describing what actually happens.

The re-captioning step is practically important: rather than discarding all instruction-following failures (which would discard videos that are otherwise high-quality, just mislabeled), the system repairs the label. This increases the yield of the pipeline without compromising downstream learning—the policy learns from (video, corrected_instruction) pairs that are internally consistent.

Step 2: Trajectory plausibility evaluation. A separate VLM (Gemini 2.5 Flash) evaluates the generated video's physical plausibility on a 1–5 Likert scale, prompted with 8 frames. The evaluation focuses specifically on "whether robot hand trajectories in the generated videos are plausible, exhibiting correct sense of depth and spatial orientation"—cited as common failure modes involving "incorrect approach distances and object interactions." Videos scoring 3 or above are retained. This coarse physical filter catches obvious failures (hands teleporting, passing through objects, completely wrong depth perception) before the attentive probe addresses subtler action-level mismatches.

Why separate VLMs for different checks. Using Gemini 3 Pro for instruction following and Gemini 2.5 Flash for trajectory plausibility likely reflects different capability requirements: instruction following requires sophisticated language understanding to compare the instruction text against the video content, favoring a more capable model. Trajectory plausibility requires visual-spatial reasoning but simpler judgment (a 1–5 rating), and can use a faster, cheaper model. The paper applies the trajectory plausibility filter "only prior to our action-level filtering stage," meaning the attentive probe sees only videos that have passed this coarse physical check, reducing the probe's workload to detecting subtle mismatches in otherwise plausible-looking videos.

Dataset sizing. After post-processing, the dataset is exactly 10K neural trajectory for the pre-training experiments—the post-processing filters from a larger initial pool down to this fixed size, ensuring fair comparison across methods. For co-finetuning, post-processing is still applied but the fixed-size constraint is relaxed given the already small generation budget (48–50 episodes per task).

4. Key Insights and Innovations

Innovation 1: Simulator-Replay Consistency as a Physically Grounded Verification Primitive for Synthetic Robot Data

The paper's most conceptually distinctive contribution is the introduction of simulator-replay consistency—the idea of validating synthetic robot trajectories by replaying predicted actions in a physics simulator and checking whether the resulting motion matches the generated video. This is not an incremental improvement to existing filtering methods; it is a fundamental reframing of what "quality" means for neural trajectory data.

Prior to this work, the field had two approaches to assessing synthetic robot data quality, both of which operate on the video alone. The first approach—exemplified by DreamGenBench and VideoCon-Physics, which the paper directly benchmarks in Table 4—asks a VLM to judge whether a generated video "looks physically plausible" or "follows instructions." The second, implicit in the standard neural trajectory pipeline, is to trust the IDM as a black-box annotator and hope that physically plausible videos produce correct actions. Both share the same critical blind spot: they evaluate the video without any ground-truth reference for what the actions should produce. A VLM can detect that a hand clearly clips through a table, but it cannot detect that the hand stopped 3 cm short of the object—and it is this second failure mode, not the first, that determines whether the policy learns a successful grasp.

RoboCurate dissolves this blind spot by introducing a physical oracle: the simulator. The simulator does not need to match reality visually (it renders with different lighting, textures, and camera properties than the generated video). It only needs to match reality physically—joint torques move the hand to the same Cartesian positions, objects respond to contact forces, gravity acts downward. By replaying the IDM-predicted actions and comparing the resulting video to the generated one, the system gains a reference signal that neither the VLM nor the IDM alone can provide. This converts action verification—a problem for which ground-truth labels do not exist by definition—into video comparison, a problem for which training data can be manufactured from real demonstrations with known actions.

The significance of this move extends beyond the specific attentive probe architecture. It establishes that simulators can serve as verification oracles for neural data, not just as data sources themselves. This is a different relationship between simulation and learning than the dominant paradigm (generate data in simulation, transfer to reality). Here, simulation is used not to generate training data but to validate it—a role that does not require photorealism, only physical fidelity. This decouples the visual quality problem (which video generation models solve better than simulators) from the physical correctness problem (which simulators solve better than video models), allowing each to be used for what it does best.

The empirical support is decisive. Table 4 shows RoboCurate's action-level filtering achieving 38.3% average success rate versus 35.4% for DreamGenBench and 35.2% for VideoCon-Physics—a meaningful gap that persists across both rearrangement and articulated task categories. More telling is the breakdown in Table 6: removing the attentive probe and using simple cosine similarity between frozen video embeddings drops performance from 26.2% to 23.8%, barely above unfiltered data (22.7%). This demonstrates that the signal extracted from simulator-replay comparison is non-trivial—it requires learned attention, not just embedding distance, to separate motion from appearance.

Innovation 2: The Demonstration That Visual Diversity and Action Quality Are Independent, Compoundable Axes of Synthetic Data Utility

A quieter but equally important insight is the paper's experimental demonstration that visual diversity and action quality are not competing objectives—they can be independently controlled and their benefits multiply. This is not an architectural contribution but a diagnostic finding that changes how the community should think about synthetic data generation.

The field's default assumption, visible in the DreamGen baseline and in most prior work on neural trajectory, has been that the primary bottleneck is generating enough task scenarios. The response has been prompt engineering: query a VLM for diverse task instructions, condition video generation on those instructions, and hope that the resulting coverage of (skill, object, placement, hand) combinations produces a useful training distribution. Visual appearance—the specific textures, lighting, and backgrounds of the scenes—was treated as whatever the video model happened to produce, with no explicit control.

RoboCurate's diversity ablation (Table 5) dismantles this assumption along two dimensions. First, it shows that task diversity alone yields monotonic but sub-linear gains: going from 25% task coverage to 100% improves average success from 12.5% to 19.7%, a substantial increase but one that plateaus as coverage saturates. Second, and more importantly, it shows that adding visual diversity on top of full task coverage yields an additional 3.6 percentage points (19.7% → 23.3%, or roughly +18% relative). This means that even when the policy has seen every (skill, object, placement, hand) combination, it still benefits from seeing those same combinations rendered with different tables, different lighting, and different object textures.

The implication is that visual diversity is not just about sim-to-real robustness—it improves policy performance even when evaluation is in the same visual domain. The likely mechanism is that visual variation forces the policy to attend to task-relevant features (object shape, spatial relationships, robot hand pose) rather than exploiting visual shortcuts (a specific texture pattern, a distinctive shadow). This is a form of implicit data augmentation that operates at the level of the policy's learned representations rather than through explicit regularization.

The design choices that enable this—Canny edge conditioning for I2I editing, action-preserving V2V transfer, the explicit constraint that V2V modifies only texture and color while preserving object identity and shape—are engineering contributions, but the conceptual contribution is the demonstration that these choices matter for downstream performance and operate independently of action quality filtering. The paper shows that RoboCurate without filtering (Table 1: 22.7%) still substantially outperforms DreamGen (19.5%), indicating the diversity pipeline alone provides gains. Adding filtering yields further improvement (26.2%), indicating the two mechanisms compound rather than overlap.

Innovation 3: Automatically Constructed Training Data for Fine-Grained Motion Comparison, and the Counterintuitive Finding That It Outperforms Human Labels

The attentive probe's training methodology represents an elegant solution to a problem that initially appears to require expensive human annotation: how do you train a classifier to detect subtle motion mismatches between videos without manually labeling thousands of video pairs? The paper's approach—constructing positive and negative pairs from real demonstrations by exploiting the known mapping between actions and simulator replay—is simple in retrospect but represents a non-obvious application of the simulator's oracle property.

The negative pair construction is particularly clever because it teaches the probe to distinguish aligned from misaligned motion along two orthogonal axes that map directly to the failure modes of neural trajectory. Temporal shift negatives simulate the case where the generated video shows the robot at a different stage of the task than the actions would produce (e.g., the actions say "reach toward object" but the video shows the hand already at the object). Cross-episode negatives simulate the case where the actions are for an entirely different manipulation (e.g., the actions describe a pick-and-place but the video shows a push). By training on both, the probe learns to detect both temporal and semantic motion mismatches—precisely the failures that arise when an IDM mispredicts actions from generated video frames.

The finding that this automatically constructed data outperforms human-labeled data (Table 6: 26.2% for automatic construction vs. 23.5% for 2K human-labeled pairs) is the kind of result that, while not heavily emphasized in the paper, has significant implications beyond robotics. It suggests that for perceptual comparison tasks where ground-truth physical correspondence can be established, synthetic supervision beats human supervision. The likely mechanism is label noise: humans, asked to judge "do these two videos show the same motion?", apply inconsistent and subjective thresholds, especially for subtle mismatches. The synthetic data has perfect labels by construction—positive pairs are guaranteed identical in motion, negative pairs are guaranteed different—eliminating the label noise that limits human-annotated classifier performance.

This result is a specific instance of a broader phenomenon that has been observed in other domains: when a generative process can produce perfectly labeled data (e.g., through known transformations or physical grounding), it often yields better supervision than human annotation, which is inherently noisy for fine-grained perceptual tasks. The paper's contribution is demonstrating this phenomenon for the novel problem of video motion consistency comparison in robot manipulation, and providing a concrete, reproducible recipe for constructing the training data.

Innovation 4: Emergent Novel Behaviors from Curated Synthetic Data Alone, Establishing a Capability Threshold That Uncurated Data Cannot Cross

The paper's most striking result—and the one with the deepest implications for how the field thinks about synthetic data—is the demonstration that curated synthetic trajectories can teach a policy to perform behaviors for which zero real demonstrations exist. On the ALLEX humanoid, the pour-can task achieves 0.0% success with real-data-only training (the policy never sees this behavior in real data) and 12.5% with uncurated DreamGen-style synthetic data (Table 3). RoboCurate's Best-of-N curated data achieves 25.0%. This is not an incremental improvement on a known task; it is the emergence of a qualitatively new motor skill—pouring—that the policy has never seen executed on real hardware.

This result establishes a capability threshold that has been implicitly assumed but never empirically demonstrated: synthetic data can cross a signal-to-noise threshold that enables learning new skills, but only if the data is of sufficient quality. Uncurated data, even when generated in quantity, fails to cross this threshold because the noise from physically implausible videos and incorrect action labels drowns out the signal from the few correct examples. RoboCurate's filtering raises the average quality enough that the policy can extract the underlying skill from the successful trajectories.

The implication is that the relationship between synthetic data quantity and policy capability is not smooth—it exhibits a phase transition at the point where the signal from correct examples exceeds the noise from incorrect ones. This has direct practical consequences: for novel skill acquisition, generating more uncured data may be futile (the policy cannot learn from data where it cannot distinguish correct from incorrect examples), while generating less but higher-quality data—through Best-of-N selection with a reliable critic—may succeed.

The broader conceptual move here is reframing synthetic data generation for robotics from a quantity problem (we need more data, any data) to a signal-to-noise problem (we need better data, specifically data with correct actions). This parallels the shift that occurred in language model training when the community moved from "just scrape more text" to "curate high-quality sources." The robotic analog, enabled by RoboCurate's verification mechanism, is: generate many candidates, physically verify them, and train only on the verified ones.

This innovation is supported by the full pattern of results across Tables 1–3: RoboCurate's gains over DreamGen are largest in the lowest-data regime (300 demos: +70.1% vs. +26.6% relative improvement) and on the hardest generalization challenges (novel behavior: 25.0% vs. 12.5%). As data becomes more plentiful (1000 demos) or tasks become easier (in-distribution pick-and-place), the gap narrows but does not close. This pattern is consistent with the signal-to-noise threshold interpretation: when other sources of signal (real demos, task similarity) are weak, data quality matters most.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The pre-training experiments use a 3K subset of ActionNet (fourier2025actionnet), a bimanual tabletop manipulation dataset with 44-DoF dexterous hand trajectories collected on the Fourier GR1-T1 humanoid from an ego-view camera. The 3K subset is obtained by applying unique prompt filtering to the original 30K episodes to reduce redundancy. Fine-tuning and evaluation are conducted on two simulated benchmarks: GR-1 Tabletop (nvidia2025gr00tn1openfoundation), covering 24 tasks (18 rearrangement, 6 articulated) with dexterous hand manipulation across varied objects and placements, and DexMimicGen (jiang2025dexmimicgen), covering 6 tasks (3 GR-1 Humanoid, 3 Bimanual Panda Arms with dexterous hands) for cross-embodiment evaluation. The co-finetuning experiments use a manually collected ALLEX humanoid dataset with 48 teleoperated demonstrations for one in-distribution pick-and-place task, plus synthetic data for two out-of-distribution tasks (pick-and-place cup, pour can) for which zero real demonstrations exist. All evaluations use 50 trials per task except ALLEX (24 trials across 3 tasks), and results are averaged over 3 random seeds for DexMimicGen.

• Base model(s). The policy backbone is GR00T N1.5 (nvidia2025gr00t), a vision-language-action model with a diffusion-based action head trained with flow matching. For pre-training, the action head is initialized from scratch (no pre-trained weights loaded), while the vision and language components presumably inherit from the GR00T N1.5 checkpoint—though the paper is not explicit about which parts are pre-trained versus randomly initialized. The GR00T N1.5 is chosen because it represents a state-of-the-art VLA architecture with demonstrated performance across manipulation benchmarks, making it a representative testbed for evaluating synthetic data quality. For the ALLEX co-finetuning experiments, a fully pre-trained GR00T N1.5 model is loaded and fine-tuned, which is important because it tests whether curated synthetic data can further improve an already-capable model.

• Metrics. The primary metric is success rate (%), computed as the fraction of evaluation trials in which the policy successfully completes the specified task. For simulation benchmarks (GR-1 Tabletop, DexMimicGen), success is determined automatically by the simulator based on task completion criteria (e.g., object placed at target location, articulated object moved to goal state). For the real-world ALLEX experiments, success is presumably evaluated by human judgment, though the paper does not specify the evaluation protocol in detail. For GR-1 Tabletop, results are reported separately for rearrangement and articulated task categories and then averaged. For DexMimicGen, results are reported per embodiment (GR-1 Humanoid, Bimanual Panda Arms) and averaged. For ALLEX, results are reported per task (in-distribution pick-and-place cube, novel object pick-and-place cup, novel behavior pour can) and averaged.

• Baselines. The paper evaluates against several baselines, organized along two axes: data source and filtering method.

  • Real data only: policy trained exclusively on real demonstrations with no synthetic data, serving as the lower bound for data scaling.
  • DreamGen (jang2025dreamgen): the base neural trajectory pipeline—image-to-video generation from initial frames, IDM pseudo-labeling—without RoboCurate's visual diversity augmentation or action-level filtering. This serves as the "uncurated synthetic data" baseline.
  • RoboCurate without filtering: the full visual diversity pipeline (I2I editing + V2V transfer + task instruction diversification) but without the attentive probe's action-level verification, included in Tables 1 and 2 to isolate the contribution of filtering from diversity.
  • RoboCurate with filtering: the full pipeline, used in all main results tables.
  • DreamGenBench (jang2025dreamgen): VLM-based physical plausibility judgment of generated videos, compared as a filtering baseline in Table 4.
  • VideoCon-Physics (bansal2024videophy): VLM-based video-level physics assessment using the 7B VideoCon model, compared as a filtering baseline in Table 4.
  • Cosine similarity filtering (Table 6): thresholding cosine similarity between frozen V-JEPA2 embeddings of generated and simulator-replay videos.
  • Human-labeled attentive probe (Table 6): the attentive probe trained on 2K manually binary-labeled video pairs instead of the automatically constructed data.
  • For co-finetuning (Table 3), RoboCurate uses Best-of-N sampling as the filtering mechanism, denoted as "RoboCurate*".

• Generation budget / compute accounting. For pre-training experiments, all methods that leverage synthetic data use the same pre-filtering dataset size of 10K neural trajectory—meaning the total number of trajectories passed to the policy for pre-training is fixed at 10K regardless of how many were generated then filtered. This is the key fairness constraint: uncurated methods feed all 10K generated trajectories, while RoboCurate generates a larger pool (the exact size is not specified, but must exceed 10K since some are discarded during filtering) and retains the 10K highest-quality trajectories. For co-finetuning, the generation budget is specified per task: 48 episodes for the in-distribution task, 50 for each OOD task. For Best-of-N sampling, the paper generates N candidate videos per episode (N is not explicitly stated) and selects 1, so the total generation cost before selection is N × 48 and N × 50 for ID and OOD tasks respectively. The I2I:V2V ratio for augmented videos is 2:1 "for efficiency given the generation cost," acknowledging that V2V transfer is more expensive per output since it processes full videos rather than single frames.

• Cross-validation / statistical protocol. The paper reports that DexMimicGen results (Table 2) are averaged over 3 random seeds, with the standard error of the mean likely corresponding to the variability across these seeds. For GR-1 Tabletop (Tables 1, 4, 5, 6), no mention of multiple seeds or cross-validation is made, suggesting results may be from a single training run per configuration—a potential concern for reliability, especially with the modest 50-trial evaluation per task. For real-world ALLEX results (Table 3), 24 trials across 3 tasks means only 8 trials per task, which is an extremely small sample size for estimating success rates, particularly for the novel behavior task where the 25.0% result represents 2 out of 8 successful trials. Confidence intervals are not reported anywhere in the paper, making it difficult to assess whether the reported differences are statistically significant or within the noise floor.

Main Quantitative Results

Pre-training on GR-1 Tabletop and DexMimicGen

Headline result on GR-1 Tabletop (Table 1). RoboCurate with filtering achieves an average success rate of 26.2% with 300 demonstrations (25.4% rearrangement, 28.7% articulated) and 37.9% with 1000 demonstrations (38.2% rearrangement, 37.0% articulated). These represent relative improvements of +70.1% and +25.1% over the real-data-only baseline (15.4% and 30.3%, respectively). The uncurated DreamGen baseline achieves only +26.6% relative improvement with 300 demos (15.4% → 19.5%) and +6.3% with 1000 demos (30.3% → 32.2%), demonstrating that RoboCurate's curation provides substantially more benefit when real data is scarce.

The breakdown of improvements reveals an interesting asymmetry: the articulated task category benefits disproportionately from RoboCurate's curation. With 300 demonstrations, RoboCurate achieves 28.7% on articulated tasks versus DreamGen's 14.7% and the real-only baseline's 13.3%—nearly a 2× improvement over DreamGen and more than doubling the real-only performance. In contrast, rearrangement tasks show a smaller gap: 25.4% (RoboCurate) versus 21.1% (DreamGen) versus 16.1% (real-only). This suggests that articulated manipulation—which involves understanding mechanical constraints (hinges, drawers, doors) rather than just pick-and-place—places higher demands on action quality, making the curated data's correct actions especially valuable.

The diversity-only ablation (RoboCurate without filtering, Table 1: 22.7% at 300 demos) already outperforms DreamGen (19.5%), confirming that the visual diversity pipeline independently contributes +3.2 percentage points beyond base neural trajectory generation. Adding filtering adds a further +3.5 points (22.7% → 26.2%), establishing that diversity and filtering compound rather than substitute for each other.

Headline result on DexMimicGen (Table 2). RoboCurate with filtering achieves an average success rate of 51.8% across 6 tasks—62.7% on GR-1 Humanoid and 40.9% on Bimanual Panda Arms with dexterous hands—trained with 100 demonstrations per task. The relative improvement over real-data-only (44.6%) is +16.1%, versus only +4.0% for DreamGen (46.4%). The cross-embodiment transfer aspect is notable: the pre-training data uses GR-1 humanoid trajectories, yet the fine-tuned policy successfully operates Bimanual Panda Arms, demonstrating that the synthetic data's action priors generalize across embodiments with different kinematics and hand morphologies.

Why the gains are smaller on DexMimicGen. The +16.1% relative improvement on DexMimicGen is substantially smaller than the +70.1% on GR-1 Tabletop, and the paper does not explicitly analyze this difference. Several factors likely contribute: (a) DexMimicGen uses 100 demonstrations per task rather than 300, which is a substantially higher per-task real-data budget (6 tasks × 100 = 600 total demos), leaving less room for synthetic data to improve; (b) the cross-embodiment transfer from GR-1 to Panda Arms may be inherently limited by kinematic differences, capping the value of GR-1 synthetic trajectories; (c) the real-data-only baseline is already strong at 44.6%—roughly 3× higher than GR-1 Tabletop's 15.4%—suggesting the tasks are intrinsically easier or better matched to the base policy.

Co-finetuning on Real-World ALLEX Humanoid

Headline result on ALLEX (Table 3). RoboCurate with Best-of-N sampling achieves an average success rate of 38.9% across 3 tasks—47.9% on the in-distribution pick-and-place cube, 43.8% on novel object pick-and-place cup, and 25.0% on novel behavior pour can. The relative improvement over real-data-only (13.9%) is +179.9%, versus +100.0% for DreamGen (27.8%). Critically, the pour-can task achieves 25.0% with RoboCurate versus 12.5% with DreamGen and 0.0% with real data only, demonstrating that curated synthetic data enables emergent novel behavior learning.

The sample sizes underlying these results are worth scrutinizing: 24 total trials across 3 tasks means 8 trials per task. The pour-can result of 25.0% corresponds to exactly 2 successes out of 8 trials. The pick-and-place cube result of 47.9% (likely ~4/8) and pick-and-place cup of 43.8% (likely ~3-4/8) are similarly coarse estimates. Without confidence intervals, it is impossible to determine whether the 25.0% vs. 12.5% gap on pour can is statistically meaningful or within the variance expected from 8-trial evaluations. However, the consistent pattern across all three tasks—RoboCurate outperforming DreamGen in each case—provides qualitative evidence that the effect is real, even if the precise magnitudes are uncertain.

The Best-of-N variant is the enabler. In this co-finetuning setting, the attentive probe is applied as a generation-time critic, selecting the best candidate among N videos per episode. The paper emphasizes that this avoids discarding data in an already data-scarce regime. The relative gain of RoboCurate* over DreamGen is ~40% on average (38.9% vs. 27.8%, or roughly 1.4×), which is the "additional ~40% relative gain over using unfiltered neural trajectory" that Section 4.2 highlights. This is a substantial improvement for a technique that adds no new data generation—only a smarter selection among generated candidates.

Comparison with VLM-Based Filtering Methods

Headline result on filtering comparison (Table 4). On GR-1 Tabletop with 1000 demonstrations, RoboCurate's action-level filtering achieves 38.3% average success rate (37.7% rearrangement, 40.3% articulated) versus 35.4% for DreamGenBench, 35.2% for VideoCon-Physics, and 32.1% for unfiltered Real + Neural data. The gap of +2.9–3.1 percentage points over VLM-based methods is consistent across rearrangement tasks (37.7% vs. 33.7–33.9%) but smaller on articulated tasks (40.3% vs. 39.0–40.7%), where VideoCon-Physics actually matches RoboCurate's performance. This suggests that for tasks where physical implausibility is the dominant failure mode—as in articulated manipulation where hand-object interactions must follow mechanical constraints—VLM-based physical plausibility checking partially captures the relevant signal, making action-level verification less critical.

The unfiltered baseline of 32.1% is notably lower than the "w/ RoboCurate (no filtering)" entry in Table 1 (34.8% with 1000 demos), likely because Table 4 uses a different neural trajectory generation setup or a different base Real + Neural dataset for fair comparison across filtering methods. The paper states "all methods use the same 10K pre-filtering neural trajectory" for Table 4, but does not clarify whether this dataset is generated identically to the main experiments.

Diversity Analysis

Headline result on diversity ablation (Table 5). With 300 demonstrations on GR-1 Tabletop and a fixed 10K neural trajectory budget, average success rate increases monotonically with task diversity: 12.5% at 25% coverage, 17.3% at 50% coverage, and 19.7% at 100% coverage. Adding visual diversity on top of full task coverage further improves to 23.3%. The rearrangement tasks show a particularly strong response to task diversity, nearly doubling from 12.7% at 25% coverage to 22.2% at 100% + visual diversity. Articulated tasks show a more dramatic step change: 12.0% → 14.7% → 22.0% → 26.7%, with a notable jump of +7.3 percentage points from 50% to 100% task coverage, suggesting articulated tasks have a threshold effect where insufficient task coverage prevents learning the mechanical constraints entirely.

The 23.3% result with full diversity (no filtering) serves as an important calibration point: it is the same configuration as "w/ RoboCurate (w/o filtering)" in Table 1, confirming that the diversity pipeline alone provides substantial gains over DreamGen's 19.5%. The +3.6 percentage point gap between full task coverage without visual diversity (19.7%) and with visual diversity (23.3%) is the quantitative value of the visual augmentation pipeline in isolation, representing roughly a +18% relative improvement over task diversity alone.

Aggregated Performance Across All Settings

Figure 6 provides a visual summary (bar chart of relative improvement over real-data-only baseline across all experimental settings). The pattern is consistent: RoboCurate (green bars) substantially outperforms DreamGen (blue bars) and the real-only baseline (gray, normalized to 0%) across every benchmark and data regime. The largest gains occur in the lowest-data settings: GR-1 Tabletop 300 demos (+70% relative), ALLEX co-finetuning (+180% relative), and ALLEX novel behavior (from 0% to measurable success). The gains narrow but remain meaningful at higher data volumes: GR-1 Tabletop 1000 demos (+25% relative), DexMimicGen (+16% relative).

Ablation Studies and Robustness Checks

Attentive probe architecture vs. simple similarity (Table 6): On GR-1 Tabletop with 300 demonstrations, the full attentive probe achieves 26.2% average success (25.4% rearrangement, 28.7% articulated) versus 23.8% for cosine-similarity-based filtering with the same V-JEPA2 backbone. The gap is entirely driven by articulated tasks (28.7% vs. 17.0%), while rearrangement tasks actually favor cosine similarity (25.4% vs. 26.1%). This asymmetry is important: for articulated tasks requiring precise motion understanding, the learned cross-attention over video embeddings matters substantially; for rearrangement tasks where the motion signal may be coarser, the simpler similarity metric is competitive. This suggests that the attentive probe's value scales with the fineness of the motion discrimination required.

Human-labeled vs. automatically constructed probe training data (Table 6): Training the attentive probe on 2K manually binary-labeled video pairs yields 23.5% average success—worse than both the automatic construction (26.2%) and the cosine similarity baseline (23.8%). The articulated task performance is particularly poor (19.7% vs. 28.7%), suggesting human labelers struggle to provide consistent supervision for fine-grained motion mismatches in complex manipulation scenarios. This negative result validates the paper's central design choice: automatically constructing training pairs from known action-replay correspondences provides cleaner supervision than human annotation, which is noisy for this perceptual comparison task.

Task diversity coverage (Table 5): As discussed above, performance scales monotonically with the percentage of the combinatorial task space covered, from 12.5% (25% coverage) to 19.7% (100% coverage). The relationship is not linear: the jump from 25% to 50% coverage (+4.8 percentage points) is larger than from 50% to 100% (+2.4 percentage points), indicating diminishing returns to task diversity beyond ~50% coverage under this data budget. This saturation point likely depends on the fixed 10K dataset size—with more data, higher coverage would likely continue to yield gains.

Visual diversity on top of task diversity (Table 5): Adding I2I editing and V2V transfer when task diversity is already at 100% yields +3.6 percentage points (19.7% → 23.3%). Notably, the visual diversity benefit is much larger for articulated tasks (+4.7 percentage points: 22.0% → 26.7%) than rearrangement (+3.3 percentage points: 18.9% → 22.2%), consistent with the pattern observed in the attentive probe ablation where articulated tasks are more sensitive to data quality signals.

Filtering vs. no filtering (Tables 1, 2, 4): Across three independent comparisons, adding action-level filtering to the diversity-augmented data consistently improves performance:

• GR-1 Tabletop 300 demos (Table 1): 22.7% → 26.2% (+3.5 points)
• GR-1 Tabletop 1000 demos (Table 1): 34.8% → 37.9% (+3.1 points)
• DexMimicGen (Table 2): 49.3% → 51.8% (+2.5 points)
• GR-1 Tabletop (Table 4, different setup): 32.1% → 38.3% (+6.2 points, though this includes both diversity and filtering versus a different unfiltered baseline)

The filtering benefit is largest in the lowest-data regime (300 demos) and shrinks as real data increases, consistent with the intuition that curation matters most when the policy has few real examples to learn from.

Best-of-N sampling as filtering for co-finetuning (Table 3): In the extreme low-data ALLEX setting (48–50 synthetic episodes per task), Best-of-N selection using the attentive probe yields ~40% relative improvement over unfiltered DreamGen (38.9% vs. 27.8% average, 47.9% vs. 37.5% ID, 43.8% vs. 33.3% novel object, 25.0% vs. 12.5% novel behavior). The paper does not ablate the value of N (number of candidates), so it is unclear whether further gains are possible with larger N or whether the probe saturates quickly.

Pre-training data ratio: The paper specifies (Appendix A.1) that real and neural trajectory are sampled in a 1:1 ratio during pre-training. This ratio is not ablated, leaving open the question of whether different mixing ratios (e.g., 1:2 or 2:1) would yield better performance—particularly relevant since the curated data may be higher quality but less diverse than the full 10K pool, potentially shifting the optimal mixing ratio.

Critical Assessment

Claim 1: RoboCurate yields substantial relative improvements over real-data-only baselines (+70.1% on GR-1 Tabletop, +16.1% on DexMimicGen, +179.9% on ALLEX). These numbers are directly supported by Tables 1, 2, and 3, but they carry important caveats that the paper does not sufficiently discuss. The +179.9% figure on ALLEX is computed from a very small evaluation (24 total trials, ~8 per task). With 8 trials, a single additional success changes the success rate by 12.5 percentage points, meaning the estimate is extremely noisy. The paper does not report confidence intervals or standard errors anywhere, making it impossible to assess whether these differences are statistically significant or could arise from sampling noise. This is a genuine weakness: the real-robot results are the most compelling part of the story (emergent novel behaviors from synthetic data), but they are also the least statistically robust.

Claim 2: Action-level filtering through simulator-replay consistency substantially outperforms VLM-based video plausibility filtering. Table 4 supports this with a 2.9–3.1 percentage point gap between RoboCurate and DreamGenBench/VideoCon-Physics (38.3% vs. 35.4%/35.2%). However, the gap is concentrated in rearrangement tasks (37.7% vs. 33.7%/33.9%) and largely disappears for articulated tasks (40.3% vs. 40.7%/39.0%), where VideoCon-Physics actually matches RoboCurate. The paper does not discuss this asymmetry. A possible interpretation: for tasks where physical constraint violations are the dominant failure mode, VLM-based physics checking captures much of the relevant signal, and action-level verification provides marginal additional value. For tasks where subtle motion errors (wrong grasp position, insufficient approach distance) matter more, the simulator-replay comparison provides unique signal. This is a more nuanced story than "our method beats VLM filtering," and the paper would be stronger for acknowledging it.

Claim 3: Visual diversity and action quality are independent, compoundable axes of synthetic data utility. This claim is well-supported by the diversity ablation (Table 5) and the filtering ablation rows in Tables 1 and 2. The diversity-only improvement over DreamGen (22.7% vs. 19.5% in Table 1) and the filtering-on-top-of-diversity improvement (26.2% vs. 22.7%) demonstrate independent additive effects. However, the paper does not test whether filtering is more or less effective at different levels of diversity—it is possible that filtering becomes more important when diversity is high (because visual variation makes the IDM's job harder) or less important (because diverse scenes implicitly include some high-quality trajectories). An interaction ablation crossing diversity level with filtering on/off would be the definitive test.

Claim 4: Curated synthetic data enables emergent novel behavior learning (pour can: 0.0% → 25.0%). This is the paper's most striking claim and also its most fragile. With 8 trials, the 25.0% success rate (2 successes) versus 12.5% for DreamGen (1 success) is a difference of exactly one trial outcome. A third trial of RoboCurate or a second trial of DreamGen could eliminate or reverse the gap. The paper does not report whether the evaluation protocol involved repeated trials on the same physical setup or variations in object positions/lighting, which would affect generalizability. The claim that synthetic data enables novel behavior is plausible and consistent with the pattern across other results, but the specific magnitude claims on ALLEX should be treated as suggestive rather than definitive.

What is missing. Several experiments would significantly strengthen the paper's conclusions:

1. Confidence intervals and statistical significance tests for all results, especially the ALLEX real-robot evaluation. Without these, it is impossible to distinguish real effects from sampling noise, particularly given the small evaluation sample sizes.

2. Ablation of the attentive probe threshold $c$. The filtering threshold determines the precision-recall tradeoff: a high threshold keeps only the highest-confidence samples (high precision, low recall), while a low threshold keeps more samples (low precision, high recall). The optimal threshold likely depends on the downstream data budget and task difficulty, yet the paper does not disclose it or ablate its effect.

3. Ablation of Best-of-N $N$. For the ALLEX experiments, the paper uses Best-of-N sampling but does not state $N$ or show how performance scales with $N$. This is critical because generating more candidates per episode increases total compute linearly but may yield diminishing quality returns as the probe saturates.

4. Comparison with simulation-based data augmentation. The paper positions neural trajectory as an alternative to simulation, but never directly compares RoboCurate against a simulation-generated dataset of equivalent size. A head-to-head comparison—10K simulator-generated trajectories vs. 10K RoboCurate neural trajectories, both pre-training the same policy—would directly test the paper's claim that neural trajectory's photorealism outweighs simulation's perfect physics.

5. Scaling the amount of curated data. All experiments use a fixed 10K neural trajectory budget for pre-training. It is unknown whether the benefits of curation persist, grow, or shrink with larger synthetic datasets. If the attentive probe's filtering removes a roughly constant fraction of low-quality trajectories, then doubling the pre-filtering pool should yield diminishing returns as the probe's false-positive rate becomes the bottleneck.

6. Cross-task generalization analysis. The paper demonstrates novel behavior learning (pour can) from synthetic data, but does not analyze whether the synthetic pour-can trajectories transfer to variations of the task (different can shapes, different pouring targets) or to related skills (tipping, shaking). This would test whether the policy learned a general pouring skill or a narrow synthetic-data-memorized behavior.

7. Sensitivity to simulator fidelity. The simulator-replay verification assumes the simulator accurately reflects the robot's kinematics. If the simulator model differs from the real robot (joint limits, link lengths, mass properties), the verification could reject correct trajectories or accept incorrect ones. The paper does not discuss simulator fidelity requirements or test sensitivity to modeling errors.

On balance, the experiments support the paper's qualitative claims—curated synthetic data with simulator-replay verification improves policy learning, and the improvement is larger when real data is scarce—but the precise quantitative magnitudes, particularly the headline percentage improvements, should be interpreted with appropriate skepticism given the small evaluation sample sizes and absence of statistical reporting. The ALLEX results are especially vulnerable to this concern and would benefit substantially from larger-scale evaluation in any follow-up work. The ablation studies are well-designed and internally consistent, but the paper leaves several practically important knobs (filtering threshold, Best-of-N $N$, data mixing ratio) unexplored.

6. Limitations and Trade-offs

The Simulator-Replay Verification Requires a High-Fidelity Simulator of the Specific Robot Embodiment

The assumption or constraint. The central verification mechanism—replaying IDM-predicted actions in a simulator and comparing the rendered video to the generated video—requires access to a physics simulator that accurately models the target robot's kinematics, joint limits, contact dynamics, and link geometry. The paper uses ActionNet's GR-1 humanoid and the ALLEX humanoid, both of which have simulation models available (a requirement for collecting the real demonstration data in the first place, since both platforms use simulation for at least some portion of their data pipelines).

The paper does not discuss what happens when the simulator's model differs from the real robot's dynamics. Small kinematic discrepancies—incorrect link lengths, joint offsets that differ from the physical hardware, simplified mass distributions—would cause the simulator replay to produce slightly different joint trajectories from the real robot executing the identical actions. The attentive probe, trained to detect motion mismatches, could flag these physically correct but kinematically mismatched trajectories as "inconsistent," discarding perfectly good synthetic data. Conversely, if the simulator is overly simplified (e.g., no contact friction, idealized joint constraints), it might accept trajectories that are physically impossible on the real hardware.

The consequence. A practitioner deploying RoboCurate on a novel robot platform for which no high-fidelity simulator exists faces a bootstrap problem: they cannot train the attentive probe (which requires real demonstrations with known actions to construct positive/negative pairs), and they cannot verify synthetic trajectories without a simulator. For custom or novel robotic hardware—which is exactly the regime where synthetic data would be most valuable, since real demonstration collection on new platforms is expensive—this requirement is prohibitive.

Even for platforms with simulators, the fidelity requirement is unstated. The paper does not characterize how accurately the simulator must match reality for the verification to remain reliable. A 5% error in link length? A 2 cm offset in the camera extrinsics relative to the real robot's camera? The probe's sensitivity to these errors is unknown, yet they determine whether the filtering actually improves data quality or merely adds a systematic bias against trajectories that deviate from the simulator's imperfect model.

What evidence exists in the paper. The paper does not measure simulator fidelity requirements, ablate simulator quality, or discuss sensitivity to modeling errors. Appendix A briefly mentions that the IDM is trained on "manually collected ALLEX data" (48 demonstrations) for the co-finetuning experiments, and that the video generation model is fine-tuned on this same data, but there is no corresponding description of simulator calibration or validation. The experimental setup implicitly assumes the simulator is accurate enough, but this assumption is never tested. The paper's success on real-world ALLEX tasks (Table 3) provides indirect evidence that the simulator model is sufficient for verification in this specific case, but does not establish a general fidelity threshold.

Mitigation status. The paper does not address this limitation. There is no discussion of simulator requirements, no sensitivity analysis, and no suggestion that the method could work with approximate or learned dynamics models. The authors do not suggest future work on characterizing or reducing the simulator fidelity requirement. This is a significant gap for practitioners considering adoption on new platforms.

---

Difficulty Estimation Is Computationally Expensive and Unaccounted for in the Headline Efficiency Claims

The assumption or constraint. RoboCurate's pipeline requires three computationally expensive stages that produce no immediate contribution to policy training: (1) generating a pool of synthetic videos larger than the final dataset (since filtering discards samples); (2) running the IDM on every generated video to produce candidate action sequences; and (3) replaying every candidate action sequence in the simulator and evaluating it through the attentive probe. The paper acknowledges that the generation cost exceeds the "pre-filtering dataset size of 10K neural trajectory" used in all pre-training experiments, but never quantifies the overhead ratio.

For a concrete estimate: if the attentive probe retains 50% of generated trajectories, the system must generate, IDM-label, and simulate 20K videos to produce 10K curated trajectories. Each video generation requires multiple neural function evaluations through the Cosmos-Predict2-14B diffusion model. Each IDM inference requires denoising a 44-dimensional action sequence through a diffusion transformer. Each simulator replay requires forward dynamics simulation and rendering. The total compute is dominated by video generation (the 14B-parameter diffusion model) and simulator rendering (which can be accelerated but is not costless, particularly for dexterous hand manipulation with complex contact dynamics).

The consequence. The headline efficiency gains (+70.1%, +16.1%, +179.9% relative improvements) are computed by comparing policy performance after training on curated vs. uncurated data of the same final size, without accounting for the additional generation, IDM inference, and simulation cost required to produce the curated data. If generating, labeling, and filtering 20K trajectories costs 2× more than generating 10K uncurated ones, then the fair comparison would be: does training on 10K curated trajectories outperform training on 20K uncurated trajectories? The paper does not perform this comparison. A practitioner deciding whether to adopt RoboCurate needs to know whether the curation cost is better spent on generating more uncured data instead.

The I2I editing and V2V transfer pipeline adds further overhead. The paper specifies an I2I:V2V ratio of 2:1 "for efficiency given the generation cost" (Appendix B), implicitly acknowledging that V2V transfer is expensive. The diversity ablation (Table 5) shows that visual diversity improves performance by +3.6 percentage points, but this must be weighed against the cost of running FLUX.2-dev for image editing and Cosmos-Transfer2.5-2B for video transfer for every augmented trajectory. If visual diversity costs 2× more per trajectory than standard generation, the fair comparison is against generating 2× more standard trajectories.

What evidence exists in the paper. The paper does not report total wall-clock time, GPU-hours, or FLOP counts for any stage of the pipeline. The pre-filtering dataset size of 10K is the only budget constraint stated, but the pre-filtering generation cost is never quantified. The paper explicitly acknowledges that V2V transfer is deprioritized in the 2:1 ratio for efficiency reasons, but does not report what the cost difference is. The Best-of-N variant (Section 3.3) multiplies the generation cost by N (generating N candidates per episode, selecting 1), but N is never specified and the cost overhead is not analyzed.

Mitigation status. The paper partially acknowledges this through its design choices (the 2:1 I2I:V2V ratio, the claim that Best-of-N is "particularly appealing in fine-tuning settings"), but does not systematically account for or ablate the curation overhead against the alternative of generating more uncurated data. This is the most conspicuous missing experiment in the paper: a comparison where the same total compute budget is spent on (a) generating 10K curated trajectories vs. (b) generating 2× or 3× more uncurated trajectories. Until this comparison is made, the paper's claims about the value of curation over simple data scaling remain suggestive rather than conclusive.

---

The Method Assumes Access to a Pre-Trained IDM Trained on Real Data from the Target Embodiment

The assumption or constraint. RoboCurate's entire pipeline depends on the inverse dynamics model to convert generated videos into action-labeled trajectories. The attentive probe verifies these IDM predictions, but it does not replace them—it only filters out the worst ones. If the IDM is fundamentally incapable of producing correct actions for a particular type of manipulation (because its real training data does not cover that behavior), the probe will correctly detect the mismatch and discard the trajectory, but the system has no mechanism to produce a corrected action.

For the pre-training experiments, the paper uses an open-source pre-trained IDM checkpoint for the GR-1 embodiment. For the ALLEX co-finetuning, the IDM is trained from scratch on the manually collected ALLEX dataset (48 demonstrations, 20K gradient steps, batch size 64; Appendix B). Training an IDM from scratch on only 48 demonstrations is an extremely data-scarce regime—the IDM must learn to predict 48-dimensional action sequences from pairs of image observations with fewer training examples than the action dimensionality. If the IDM fails to generalize to novel object configurations or hand poses, the attentive probe will correctly reject its predictions, but no amount of filtering can extract correct actions from an IDM that never produces them.

The consequence. For novel behaviors that are qualitatively different from the real training data, the IDM may be entirely unable to produce correct actions, regardless of how many candidate videos are generated or how stringently they are filtered. The Best-of-N variant (Section 3.3) selects the candidate with the highest alignment score among N generated videos, but if all candidates have poor alignment (because the IDM fails on all of them), the "best" is still bad. A practitioner deploying RoboCurate on a new behavior for which the IDM has no training coverage will see the attentive probe reject most or all synthetic trajectories, yielding no usable data—a silent failure mode where the system appears to be working (generating videos, running the probe, selecting candidates) but produces no training signal for the novel behavior.

The paper's ALLEX results implicitly demonstrate this tension. The pour-can task achieves 25.0% success with RoboCurate versus 12.5% with DreamGen and 0.0% with real data only (Table 3). The fact that DreamGen achieves non-zero success (12.5%) means the IDM does sometimes produce correct actions for pouring, even when trained on only 48 pick-and-place demonstrations. But the 25.0% ceiling (even with Best-of-N selection) suggests the IDM's accuracy on this novel behavior is fundamentally limited—filtering can select the best among imperfect predictions but cannot create correctness where the IDM has none.

What evidence exists in the paper. The paper does not ablate IDM quality or training data volume. All experiments use a fixed IDM (pre-trained for GR-1, trained on 48 demos for ALLEX). There is no experiment showing how policy performance changes if the IDM is trained on more data, a different data distribution, or a different architecture. The paper does not report the attentive probe's acceptance rate broken down by task type—it would be informative to know what fraction of pour-can trajectories pass the filter compared to pick-and-place trajectories, as this would reveal whether the IDM's accuracy degrades on novel behaviors. The ReST experiment (Appendix K) in the paper's reference example shows that revision model training can backfire; the paper does not explore any analogous sensitivity of the IDM to training methodology.

Mitigation status. The paper does not address IDM quality as a bottleneck. The IDM is treated as a fixed component of the pipeline whose errors can be detected but not corrected. The authors do not discuss whether the IDM could be fine-tuned on high-scoring synthetic trajectories (similar to the self-improvement loops discussed in the reference paper's Section 8), or whether the simulator-replay consistency signal could be used to train a better IDM. This is a significant limitation for practitioners: the value of RoboCurate's filtering is bounded above by the IDM's accuracy, and the paper provides no guidance on what IDM quality is sufficient or how to improve it.

---

Evaluation Is Limited to Dexterous Tabletop Manipulation with a Single Model Family on Small Test Sets

The assumption or constraint. All experiments use the MATH analog of a single benchmark and single model: dexterous tabletop manipulation tasks on GR-1 Tabletop (24 tasks), DexMimicGen (6 tasks), and ALLEX humanoid (3 tasks), all trained with the GR00T N1.5 VLA architecture. The tasks share fundamental characteristics: quasi-static tabletop object interaction, bimanual or single-arm dexterous hand control, ego-view camera, rigid objects. The paper provides no evidence that RoboCurate's benefits generalize to other manipulation paradigms (mobile manipulation, in-hand dexterity without table support, tool use), other sensing modalities (depth, force-torque, tactile), other robot morphologies (wheeled bases, parallel-jaw grippers, soft manipulators), or other policy architectures (discrete-action policies, diffusion policies not built on GR00T).

The consequence. A practitioner working on, say, mobile manipulation with a wheeled base and parallel-jaw gripper, using a policy architecture other than GR00T N1.5, has no evidence that RoboCurate will provide any benefit. Several aspects of the method could be architecture-specific:

• The attentive probe is trained on V-JEPA2 video features. V-JEPA2 is a self-supervised video encoder whose representations may be particularly well-suited to the structured motion of dexterous hand manipulation but less informative for other types of robot motion (e.g., mobile base navigation where the ego-motion is the dominant visual signal rather than articulated hand motion).

• The visual diversity pipeline assumes a tabletop workspace with distinguishable objects, a static background, and a fixed camera viewpoint. The Canny edge conditioning that preserves scene structure during I2I editing and V2V transfer may fail for cluttered scenes (where edge maps become dense and uninformative) or dynamic backgrounds (where the edge map includes moving distractors).

• The simulator-replay verification assumes the simulator can render videos from the same camera viewpoint as the generated video. For mobile manipulation where the camera moves with the robot, the simulator must replicate the camera trajectory exactly—any discrepancy in the base motion creates a motion mismatch the probe will flag, even if the arm motion is correct.

The ALLEX real-robot evaluation has the additional weakness of extremely small sample sizes: 24 total trials across 3 tasks, or 8 trials per task. The pour-can result of 25.0% represents 2 successful trials out of 8. The pick-and-place cube result of 47.9% (likely 4/8) and pick-and-place cup result of 43.8% (likely 3–4/8) are similarly coarse. A single trial outcome changing by chance would shift the reported success rate by 12.5 percentage points. Without confidence intervals, none of the ALLEX results can be interpreted as statistically reliable effect sizes.

What evidence exists in the paper. The paper provides no cross-embodiment evaluation beyond the GR-1 to Panda Arms transfer in DexMimicGen (Table 2), which is still within the same dexterous tabletop manipulation domain. There is no evaluation on a non-dexterous embodiment, a mobile platform, or a non-tabletop task. The V-JEPA2 encoder choice is not ablated against other video encoders (e.g., VideoMAE, TimeSformer), leaving open the question of whether the probe's performance is specific to V-JEPA2's representations. The paper does not report confidence intervals, standard errors, or statistical significance for any result. The DexMimicGen results are averaged over 3 seeds, but the standard deviation or standard error is not reported.

Mitigation status. The paper does not claim broader generalizability and does not discuss the limitation. The evaluation scope is typical for a first paper introducing a new data curation method in robot manipulation—the community standard is to demonstrate on 1–3 benchmarks with 1–2 embodiments. But the paper's claims about enabling "emergent success on novel action tasks" (Abstract) and the broader framing as a general synthetic data curation framework require evidence beyond dexterous tabletop manipulation that the paper does not provide. This limits the strength of the paper's conclusions but is addressable in future work.

---

Real Data Remains Essential for Training Every Component, and the Dependency Structure Is Not Characterized

The assumption or constraint. RoboCurate amplifies the value of existing real demonstrations, but it does not reduce the fundamental dependency on them. Real data is required for four distinct components: (1) fine-tuning the video generation model (Cosmos-Predict2-14B is fine-tuned on ActionNet and GR00T-GR1-100 for pre-training, manually collected ALLEX data for co-finetuning; Appendix B); (2) training the IDM (open-source checkpoint for GR-1, from-scratch training on 48 demos for ALLEX); (3) constructing the training data for the attentive probe (positive/negative pairs require real videos with known actions and corresponding simulator rollouts; Section 3.2); and (4) training the policy itself (co-finetuning on ALLEX uses 48 real demonstrations; Table 3).

Each component may have a different scaling relationship with the amount of real data. The video generation model may saturate quickly—once it has enough examples to learn the robot's appearance and typical motion patterns, additional real data may not improve generation quality. The IDM may require broader coverage of the state-action space to generalize to novel behaviors. The attentive probe may require sufficient variation in real trajectories to learn a motion-comparison function that discriminates subtle mismatches. The paper provides no guidance on how much real data is needed for each component, whether one component is the bottleneck, or how these requirements interact.

The consequence. A practitioner with a limited real-data budget (say, 50 demonstrations for a new robot platform) cannot determine whether to allocate it primarily to IDM training, video model fine-tuning, or probe training pair construction. The paper's ALLEX experiments use the same 48 demonstrations for all three, so the components are not independently ablated by data volume. If, for example, the IDM requires 100 demonstrations to produce reasonable actions on novel behaviors but the video generation model needs only 20 to produce plausible videos, the practitioner would benefit from knowing this tradeoff—but the paper provides no such analysis.

More fundamentally, the paper does not establish the minimum real-data threshold below which RoboCurate fails. The ALLEX experiments use 48 demonstrations, which is very small, but the GR-1 Tabletop experiments use 300–1000 demonstrations per task (Table 1), which is substantially more. It is possible that RoboCurate's filtering is most valuable in the regime of 100–500 demonstrations (where the IDM is competent but not perfect, giving the filter meaningful choices) and provides diminishing returns below 50 (where the IDM is too poor for filtering to salvage) and above 1000 (where real data is abundant enough that synthetic data adds little). The paper's results are consistent with this hypothesis but do not test it directly.

What evidence exists in the paper. The paper's two data regimes—300 demos on GR-1 Tabletop and 48 on ALLEX—both show benefits from RoboCurate, but the benefit magnitude differs dramatically (+70.1% vs. +179.9% relative improvement). This could reflect the real-data volume, the task difficulty, or the evaluation domain (simulation vs. real). The paper does not systematically vary the amount of real data used for each component. The Appendix A.1 mentions that "the IDM is trained on ActionNet" for pre-training and "from scratch" on ALLEX data for co-finetuning, but does not report IDM prediction accuracy as a function of training data volume.

Mitigation status. The paper does not address this limitation explicitly. The dependency on real data is inherent to the approach—RoboCurate is a data curation framework, not a zero-shot generation method—but the paper does not characterize the shape of this dependency or provide guidance on allocating a fixed real-data budget across the pipeline's components. This is a practical limitation for deployment that the paper implicitly acknowledges by noting that the video model, IDM, and probe all require real data, but does not analyze further.

7. Implications and Future Directions

How This Work Changes the Landscape

RoboCurate introduces a physically grounded verification primitive for synthetic robot data that fundamentally reframes the quality problem in neural trajectory generation. Before this work, the field had no mechanism to verify whether the actions inferred from a generated video were physically correct—only whether the video itself "looked plausible" to a VLM. This is not an incremental improvement to video filtering; it is a category change in what signal is available for data curation. The paper demonstrates that action-level verification through simulator replay extracts information that VLM-based video assessment fundamentally cannot access, because the VLM lacks a physical reference frame against which to compare the video's motion.

The shift this enables is from generation-centric to verification-centric thinking about synthetic robot data. The dominant paradigm has been: build better video generation models, prompt them more carefully, and hope the resulting trajectories are useful. RoboCurate shows that a different axis—verifying outputs against a physical oracle—yields gains that compound with, rather than substitute for, generation improvements. The diversity ablation in Table 5 demonstrates this concretely: better generation (adding visual diversity) improves performance, and better verification (adding action-level filtering) independently improves it further. The two are not competing investments; they multiply.

This reframing has a specific, testable consequence that changes how research resources should be allocated: improving the verifier (the attentive probe, the simulator fidelity, the training pair construction) is at least as valuable as improving the generator (the video diffusion model, the prompt engineering, the diversity pipeline). The paper's Table 6 shows that the choice of verification architecture matters enormously—switching from cosine similarity to the learned attentive probe improves articulated task performance from 17.0% to 28.7%, a 11.7 percentage point gap that exceeds the benefit of adding visual diversity (Table 5: +3.6 points). This suggests the verification signal, not the generation diversity, is the binding constraint on data quality in the current regime. A field that has invested heavily in better video models should now invest symmetrically in better verifiers.

The paper also establishes that curated synthetic data can cross a capability threshold that uncurated data cannot, as demonstrated by the ALLEX pour-can result (Table 3: 0.0% real-only, 12.5% DreamGen, 25.0% RoboCurate). This is not a smooth scaling curve; it is evidence of a phase transition where data quality, not quantity, determines whether a novel skill is learnable at all. The implication is that for novel behavior acquisition—the hardest and most valuable target for synthetic data—practitioners should prioritize curation over scale. Generating 10× more uncured data may be futile if the signal-to-noise ratio remains below the learnability threshold, while generating less but verified data may cross it. This insight reframes the synthetic data scaling problem from "how much can we generate?" to "what fraction of what we generate is actually useful?"

Finally, the paper reconciles a tension in prior work that has been implicit but undiscussed: the gap between the promise of neural trajectory (visually realistic, text-conditionable, scalable) and its measured performance (marginal gains over real data in DreamGen's +4.0% on DexMimicGen, Table 2). The diagnosis—that uncurated synthetic data is contaminated by physically inconsistent videos with incorrect action labels, and that this noise drowns out the signal—is not surprising in retrospect, but had not been empirically isolated before RoboCurate. The paper shows that closing the gap requires not better generation but better verification, and provides a concrete, reproducible mechanism for doing so.

Follow-Up Research This Work Enables

Characterizing the minimum real-data threshold for each pipeline component independently. RoboCurate depends on real demonstrations for four distinct purposes: fine-tuning the video generation model, training the IDM, constructing probe training pairs, and training the downstream policy. The paper uses the same real data for all four (48 demos for ALLEX, a 3K ActionNet subset for pre-training), leaving open the question of which component is the bottleneck as real data scales down. A rigorous follow-up would hold three components fixed with abundant data while systematically varying the fourth, measuring downstream policy performance across a range of data volumes (e.g., 10, 25, 50, 100, 200 demonstrations). The most practically important relationship to characterize is the IDM's data requirement: if the IDM saturates at 50 demonstrations but the probe requires 100, a practitioner with 80 demonstrations knows to invest in probe training data rather than IDM training data. This experiment would also test whether the probe's automatic training pair construction remains effective at very low real-data volumes, where the diversity of negative pairs may be insufficient to learn a general motion-comparison function.

A head-to-head compute-matched comparison of curated neural trajectory against simulation-generated data of equivalent cost. The paper positions neural trajectory as a complement to simulation but never benchmarks against a simulation-only baseline. The key experiment: fix a total compute budget (GPU-hours for generation + simulation + verification), and compare policy performance after pre-training on (a) N curated RoboCurate trajectories (including all generation, IDM inference, simulation, and filtering overhead) and (b) M simulation-generated trajectories (including scene authoring cost if scripted, or domain randomization overhead if procedural), where M may be larger than N because simulation avoids expensive diffusion model inference. The hypothesis to test is whether neural trajectory's photorealism provides benefits that outweigh simulation's perfect physics and zero filtering overhead. The paper's results suggest that photorealism matters (the visual diversity pipeline provides +3.6 percentage points in Table 5), but the cost accounting is missing. This experiment would directly inform practitioner decisions about whether to invest in neural trajectory pipelines or improved simulation authoring.

Ablation of Best-of-N N to characterize the quality-diversity tradeoff curve in data-scarce settings. The ALLEX experiments (Table 3) use Best-of-N sampling with an unspecified N, selecting the single candidate with the highest attentive probe alignment score per episode. This trades off trajectory diversity (all N candidates come from the same initial frame and instruction, varying only through stochastic generation) against trajectory quality (only the best-aligned candidate is kept). For a fixed generation budget of K total candidates across T tasks, the number of distinct trajectories per task is K / (N × T), while the per-trajectory quality increases with N. Running this experiment for N ∈ {1, 2, 4, 8, 16} on the ALLEX setup would reveal whether quality saturates quickly (N = 2 already captures most of the benefit) or continues to improve (N = 16 is substantially better than N = 4). The practical consequence: if N = 2 is sufficient, practitioners can maintain near-maximal diversity while getting quality benefits; if N = 16 is needed for the pour-can task to reach 25.0%, diversity is being heavily sacrificed and the method may overfit to the specific generation seeds.

Stress-testing the attentive probe against deliberate distribution shifts in simulator fidelity. The paper implicitly assumes the simulator model is accurate enough that the probe's alignment judgments are trustworthy. A systematic stress test would: (a) introduce controlled kinematic errors in the simulator (link length offsets of 1%, 5%, 10%, 20%), (b) measure the probe's precision and recall at each error level against a ground-truth set of verified-correct and verified-incorrect trajectories (constructed by executing the IDM actions on the true hardware or a high-fidelity simulator), and (c) measure the downstream policy performance when trained on data filtered by the probe operating with each level of simulator error. This would establish the simulator fidelity requirement—the maximum modeling error below which the probe remains a net positive for data quality. The hypothesis, based on the probe's success with real-world ALLEX data (Table 3), is that moderate errors (5–10% link length offsets) are tolerable because the probe learns to attend to coarse motion patterns (hand trajectory shape, object displacement direction) rather than precise pixel-level alignment. Proving or disproving this hypothesis would directly determine RoboCurate's applicability to platforms without high-fidelity simulators.

Using the attentive probe's alignment score as a reward for training or fine-tuning the IDM itself. The paper treats the IDM as a fixed component whose errors can be detected (and the corresponding trajectories discarded) but not corrected. However, the probe's alignment score provides a differentiable signal—for a given generated video, higher alignment between the simulation replay of predicted actions and the generated video indicates better action predictions. This signal could be used to fine-tune the IDM on synthetic videos: generate a video, have the IDM predict actions, simulate those actions, compute the probe's alignment score, and backpropagate through the IDM (treating the probe's score as a reward and using a policy gradient or direct preference optimization variant) to shift the IDM toward action predictions that, when simulated, produce motion more closely matching the generated video. The experiment would compare: (a) baseline IDM trained only on real data, (b) IDM fine-tuned on synthetic videos with a reconstruction loss (standard self-training), and (c) IDM fine-tuned with the probe's alignment score as a reward. The paper's ReST reference in the example paper's Appendix K—where RL-based revision training backfired—suggests caution, but the probe's score is a more direct quality signal than the RL reward used in ReST. This direction would transform the probe from a filter (discarding bad data) into a teacher (improving the data generator), potentially crossing the barrier identified in the ALLEX pour-can task where the IDM's limited accuracy caps the filtering benefit.

Extending the diversity pipeline to dynamic camera viewpoints and mobile manipulation. All experiments use a static ego-view camera in a tabletop setting. The Canny edge conditioning that underlies I2I editing and V2V transfer assumes a mostly static background where edges correspond to scene structure. In mobile manipulation, the camera moves with the robot, and edge maps contain ego-motion-induced changes that may confuse the conditioning or cause the diversity models to alter task-relevant motion rather than appearance. A follow-up would adapt the pipeline to a mobile manipulation benchmark (e.g., a mobile base with an arm performing pick-and-place in a room, or a quadruped with a manipulator) by: (a) replacing Canny edge conditioning with optical-flow-based conditioning that captures scene-relative motion rather than absolute edge positions, (b) testing whether the attentive probe's motion-comparison function degrades when camera ego-motion is present, and (c) measuring whether the visual diversity benefits observed in Table 5 (+3.6 points on top of full task diversity) transfer to settings where viewpoint variation is already a confounding factor. The hypothesis to test is whether the probe's benefit is larger (because camera motion makes the IDM's job harder, increasing the value of verification) or smaller (because camera motion is a confounding signal that makes the probe's motion-comparison task noisier).

Practical Applications and Downstream Use Cases

Bootstrapping novel manipulation behaviors on custom robotic hardware with minimal real demonstrations. The ALLEX results (Table 3) demonstrate the core value proposition: with 48 real demonstrations of a single pick-and-place task, RoboCurate enables the robot to perform a qualitatively different behavior (pouring a can) that it has never seen in real data, achieving 25.0% success versus 0.0% without synthetic data. For a robotics lab or company deploying a new custom manipulator—where collecting real demonstrations is expensive and time-consuming but a kinematic simulator model exists—RoboCurate provides a recipe for generating and verifying synthetic data for novel tasks without any additional human teleoperation. The practical workflow: collect ~50 demonstrations of one "anchor" task on the hardware; fine-tune the video generation model and train the IDM and attentive probe on this data (all steps described in Appendix B); generate and filter synthetic trajectories for desired novel tasks using VLM-generated task instructions (Appendix C.1) and Best-of-N selection (Section 3.3); co-finetune the policy on the real anchor data plus the curated synthetic data. The paper's numbers suggest this can turn a 0% success rate on a novel task into a workable starting point for further iterative improvement.

Scaling data for dexterous hand manipulation without proportional scaling of human teleoperation effort. Dexterous hand manipulation is notoriously difficult to collect data for—the 44-dimensional joint space of the GR-1 and 48-dimensional space of the ALLEX make teleoperation interfaces complex and fatiguing. The paper's pre-training results (Tables 1 and 2) show that 10K curated synthetic trajectories substantially improve policy performance across multiple dexterous benchmarks, with the largest gains on articulated tasks (GR-1 Tabletop articulated: 13.3% real-only → 28.7% RoboCurate, Table 1). For organizations developing dexterous manipulation policies, RoboCurate shifts the cost structure: instead of scaling teleoperation linearly with desired task coverage, invest in the fixed cost of building the curation pipeline (video model fine-tuning, IDM training, probe training) and then generate diverse curated data programmatically. The diversity ablation (Table 5) provides concrete guidance: prioritize reaching 100% task coverage first (25% → 100% coverage yields +7.2 percentage points), then add visual diversity (+3.6 additional points). The practical implication is that a small set of real demonstrations covering the robot's basic motion repertoire can be amplified into a diverse training set spanning many task variations and visual conditions.

Quality-gated data generation for self-improving robot policies in production. The Best-of-N sampling variant (Section 3.3) is designed for data-scarce fine-tuning, but it generalizes naturally to a production setting where a deployed policy continuously generates synthetic data for self-improvement. A robot operating in a warehouse or home could: log an initial frame from its camera, generate N candidate videos of a target task using the fine-tuned video model, score each with the attentive probe against simulator replays, and add only the highest-scoring trajectory to its training buffer for periodic policy updates. The probe serves as an automated quality gate that prevents the policy from being contaminated by low-quality self-generated data—addressing a known failure mode in self-improvement loops where models degrade by training on their own erroneous outputs. The paper's Table 4 result—that VLM-based filtering (DreamGenBench, VideoCon-Physics) is substantially weaker than action-level filtering—implies that a simple visual plausibility check would be insufficient as a quality gate; the simulator-replay verification is the key enabler for reliable autonomous data curation.

When to Prefer This Method

The paper explicitly positions RoboCurate against three specific alternatives across its experiments: (a) uncurated neural trajectory generation (DreamGen, Tables 1–3), (b) VLM-based video-level physical plausibility filtering (DreamGenBench and VideoCon-Physics, Table 4), and (c) real-data-only training (all tables). The conditions under which RoboCurate is preferred are clear from the experimental design and results:

• Prefer RoboCurate over uncurated neural trajectory (DreamGen) when real demonstration data is scarce relative to task diversity. The largest gains occur at 300 demos on GR-1 Tabletop (+70.1% relative vs. +26.6% for DreamGen, Table 1) and in the ALLEX co-finetuning setting with 48 demos (+179.9% vs. +100.0%, Table 3). At 1000 demos, the gap narrows (+25.1% vs. +6.3%, Table 1) but RoboCurate still outperforms. The signal is most pronounced for articulated tasks (Table 1: 13.3% real-only → 28.7% RoboCurate vs. 14.7% DreamGen at 300 demos) and novel behavior acquisition (Table 3: 0.0% real-only → 25.0% RoboCurate vs. 12.5% DreamGen on pour can), where curation provides the largest absolute and relative benefits.

• Prefer RoboCurate's action-level filtering over VLM-based video plausibility filtering when task-critical motion precision matters more than coarse physical constraint detection. Table 4 shows RoboCurate outperforms DreamGenBench and VideoCon-Physics by +2.9–3.1 percentage points on average, but the gap is concentrated in rearrangement tasks (37.7% vs. 33.7–33.9%) and minimized on articulated tasks (40.3% vs. 39.0–40.7%), where VLM-based approaches partially capture the relevant signal through physical plausibility checks. For tasks where the dominant failure mode is subtle action errors (incorrect grasp position, insufficient approach distance, wrong wrist orientation) rather than gross physical violations (objects clipping, hands teleporting), the simulator-replay comparison provides unique signal that VLMs cannot access.

• Prefer real-data-only training when the cost of building the RoboCurate pipeline exceeds the cost of collecting additional real demonstrations, or when no simulator model exists for the target embodiment. The paper does not characterize the pipeline's fixed cost, but the dependency on real data for video model fine-tuning, IDM training, and probe training pair construction means there is a minimum real-data investment below which RoboCurate cannot be deployed, and a crossover point above which collecting more real data is cheaper than building the synthetic pipeline. The paper's ALLEX experiments show the pipeline is viable at 48 demonstrations, but the generalization of this number to other embodiments and tasks is unknown. Practitioners without an existing simulator model for their robot—or for whom building one is a multi-month engineering effort—should weigh this cost against the demonstrated benefits.