WMPO: World Model-based Policy Optimization for Vision-Language-Action Models

🎯 Pitch

A VLA model that can only imitate experts breaks the moment it makes a mistake—unless you let it practice inside a video-generating world model. By fine-tuning that world model on the policy's own failures, the robot discovers self-correcting behaviors and learns to recover from errors without ever touching the real world.

1. Executive Summary

This paper introduces World Model-based Policy Optimization (WMPO), a framework that performs on-policy reinforcement learning for Vision-Language-Action (VLA) models entirely within a learned pixel-space video world model, eliminating the need for costly real-robot interactions. Evaluated on four Mimicgen simulation tasks and one real-world manipulation task using OpenVLA-OFT as the base policy, WMPO combines policy behavior alignment (fine-tuning the world model on the policy's own rollout trajectories to capture failure modes), clip-level autoregressive video generation with noisy-frame conditioning and frame-level action control (enabling stable long-horizon imagined trajectories for reliable sparse reward assignment), and on-policy GRPO optimization within the imagined environment. WMPO outperforms both online GRPO and offline DPO baselines by 9.8 percentage points with a small rollout budget (P=128) and by 15.2 points with a larger budget (P=1280), while exhibiting emergent self-correction behaviors and producing 4–6% shorter successful trajectories than the base imitation-learned policy, establishing that model-based RL with pixel-aligned world models can substantially improve VLA robustness and sample efficiency only when the world model is adapted to the policy's own behavioral distribution rather than trained solely on expert demonstrations.

2. Context and Motivation

The Core Problem: Imitation-Learned VLA Policies Are Brittle and Cannot Improve from Failure

The central problem this paper addresses is deceptively simple: VLA models trained via imitation learning work well on what they've seen, but they break when things go wrong — and they have no mechanism to get better. This isn't a minor inconvenience; it's a fundamental limitation of the dominant training paradigm for robotic manipulation.

To understand why, we need to unpack what imitation learning actually does. When you train a VLA model on expert demonstrations, you're teaching it to predict what action a human would take given a particular visual observation and language instruction. The training objective is straightforward: maximize the likelihood of the demonstrated actions given the observed states. This works remarkably well when the robot encounters states that are similar to those in the training data — the policy simply reproduces what it was shown.

But the real world is not a closed set of expert demonstrations. During deployment, small errors in action prediction compound over time. A gripper closes slightly too early, a block is nudged a few millimeters off-center, an approach angle is off by a few degrees. These micro-deviations push the robot into states that never appeared in the expert demonstrations, because experts execute tasks correctly by definition. At this point, the imitation-learned policy has no recourse. Section 1 and Figure 1a capture this pithily: the policy "can take suboptimal actions that lead to compounding errors, making task completion or recovery nearly impossible."

The paper identifies two specific behavioral failures that emerge from this brittleness. First, when the policy encounters a near-failure state (e.g., a collision during a peg-insertion task), it persists in the wrong behavior — continuing to push against the obstacle until the maximum time horizon expires — because the training data contains no examples of recovering from collisions. Second, even in successful executions, the imitation-learned policy often exhibits "stuck" behavior: it hesitates, makes micro-adjustments that don't advance progress, and generally takes longer than necessary to complete tasks. Neither of these problems can be fixed by adding more expert demonstrations, because the data distribution itself excludes the recovery behaviors the policy needs to learn.

This is a well-known problem in imitation learning, formalized as covariate shift or distributional drift in the classic work of Ross et al. [5]. The policy's action distribution during test time diverges from the training distribution, and without access to corrective feedback from those off-distribution states, performance degrades rapidly. The problem is especially acute in fine-grained manipulation tasks — precisely the kind of tasks (inserting pegs, assembling multi-part structures, stacking objects with tight tolerances) that are most valuable for real-world deployment.

Why This Problem Matters

The brittleness of imitation-learned VLAs is not just an academic concern — it has direct, practical consequences for deploying robots in the real world.

Autonomous deployment requires robustness. A robot in a warehouse, a factory, or a home will inevitably encounter situations that differ from its training demonstrations. Lighting changes, object positions shift slightly, components have manufacturing tolerances — these are routine variations, not adversarial edge cases. A policy that fails silently when it encounters any deviation from its training distribution is not deployable without human supervision. The paper's example is instructive: in the Square task (inserting a square block onto a stick), the base imitation-learned policy achieves only 24.2% success rate in simulation (Table 1). More than three out of four attempts fail. A real-world deployment at this reliability level would require constant human intervention.

Failure data is abundant but unusable under imitation learning. When a robot fails, it generates data —trajectories of what not to do. For a human learning a motor skill, watching one's own mistakes is often more informative than watching expert demonstrations. But imitation learning cannot use failure data; it can only learn from positive examples of correct behavior. This means that every failed attempt in a real deployment represents wasted information — the robot has learned nothing from its mistake. As Section 1 notes, IL-trained policies "lack the ability for learning from failures and self-correction."

Scaling expert demonstrations is expensive and unsustainable. The standard approach to improving imitation-learned policies is to collect more expert demonstrations covering a wider variety of scenarios. But expert demonstrations require human teleoperators — skilled individuals who can control the robot to perform the task. This is slow, expensive, and fundamentally limited by what humans can anticipate. You can't demonstrate every possible failure recovery, because you don't know what failures will occur in advance. Moreover, for many tasks (e.g., assembly with tight clearances, deformable object manipulation), even human teleoperators struggle to provide consistently high-quality demonstrations. The paper's lifelong learning experiment (Figure 6) quantifies this: simply adding more expert demonstrations (from 300 to 428 to 556) produces diminishing returns compared to WMPO's self-improvement approach, despite the base policy requiring human-collected trajectories while WMPO uses only self-collected data.

The gap between VLA capabilities and VLA robustness is widening. As VLA models become more capable — integrating web-scale pretraining, handling diverse instructions, generalizing across embodiments — their brittleness becomes the binding constraint. A model that can understand "insert the square block onto the stick" but fails to execute it reliably 75% of the time is not useful. The field has made enormous progress on the breadth of VLA understanding (what tasks the model comprehends) but far less progress on the depth of VLA execution (how reliably it completes them). This paper's contribution sits squarely at this gap: it provides a mechanism for taking a VLA that understands a task (via imitation learning) and making it reliably execute that task (via RL within a world model).

Where Prior Approaches Fall Short

The paper identifies three main strategies that prior work has explored for improving VLA robustness beyond imitation learning, and explains why each is insufficient.

Prior Approach 1: Real-World On-Policy RL

The most direct solution is to apply standard on-policy RL algorithms (PPO, GRPO) directly on the physical robot: the policy interacts with the environment, observes rewards, and updates its parameters to maximize return. This is theoretically sound —on-policy methods provide unbiased gradient estimates, handle exploration-exploitation naturally, and can learn from failures because they receive reward signals (not just demonstration actions) from the environment.

The problem is sample complexity. On-policy RL algorithms typically require hundreds of thousands or millions of environment interactions to learn effectively. On a real robot, each interaction means physically moving the robot, resetting the environment, and executing a full trajectory — a process that takes seconds to minutes per attempt. The paper notes this directly: applying RL to real robots is "notoriously sample-inefficient, requiring millions of interactions that are impractical, unsafe, and time-consuming to collect." An unsafe policy during exploration can damage the robot, the environment, or nearby humans. A time-consuming training process (think: weeks of continuous robot operation) is incompatible with practical deployment timelines.

The paper's own GRPO baseline bears this out (Table 1). With P=128 real trajectories, online GRPO achieves 33.2% mean accuracy across the four tasks — essentially identical to the base policy (33.6%) and far below WMPO (47.1%). Even with P=1280 trajectories, GRPO reaches only 37.1%, lagging WMPO's 57.6% by over 20 percentage points. The problem isn't the algorithm choice —it's that 1280 trajectories provide insufficient data for on-policy RL to learn meaningful improvements, and each additional trajectory costs real robot time.

Prior Approach 2: Human-Guided RL

Several works attempt to bridge the sample efficiency gap by introducing human intervention during RL training. The idea is that when the policy enters a state from which it cannot recover, a human operator takes over, provides corrective actions, and the policy learns from this intervention signal. Methods like SERL [26], ConRFT [8], and human-assisted preference optimization [9] fall into this category. By focusing RL updates on the most informative (and dangerous) states, these methods reduce the number of robot interactions needed.

The limitations are twofold. First, it's labor-intensive. Human supervision during training means someone must be physically present (or connected via teleoperation) throughout the training process. For tasks requiring hours or days of training, this is impractical and expensive. The paper notes this explicitly: the approach "requires continuous human supervision, making it labor-intensive and difficult to scale." Second, it doesn't eliminate the physical interaction bottleneck — it only reduces it. The robot still needs to execute trajectories in the real world, incurring the same hardware wear, safety risks, and wall-clock time constraints. Human guidance makes RL more sample efficient but not sample efficient enough to eliminate the need for real-robot rollouts as the primary training mechanism.

Prior Approach 3: Simulation-Based RL

An alternative is to perform RL in simulation, using a physics simulator (like MuJoCo, Isaac Gym, or Mimicgen) to generate the millions of rollouts needed for on-policy RL, then transfer the learned policy to the real robot via sim-to-real techniques (domain randomization, system identification, etc.). This approach has been successful in locomotion, dexterous hand manipulation, and some manipulation tasks. Recent works like VLA-RL [10] and SimpleVLA-RL [11] apply this strategy specifically to VLA models.

The limitation is simulator fidelity and coverage. Building an accurate simulator requires modeling every object the robot will interact with, every surface property, every contact dynamic, and every visual texture — and doing so for every new task. For simple rigid-body tasks, this is feasible. For tasks involving deformable objects, liquids, granular materials, or fine-grained contact dynamics (the Square task in this paper involves a clearance of only 5mm between the block and the stick in the real-world experiment), simulators either cannot model the physics accurately or require prohibitive engineering effort per task. The paper notes that "constructing accurate simulators for each real-world scenario introduces prohibitive engineering overhead." This is not a solved problem — it's a fundamental tension between simulator generality (can it model everything?) and simulator cost (how much effort per new task?).

Moreover, simulators primarily address the physics of the environment but not the visual realism that VLAs depend on. A VLA pretrained on web-scale real images has visual representations tuned to real-world lighting, textures, occlusions, and camera noise patterns. A simulator's rendered images — even with domain randomization — will differ from real images in ways that can degrade the VLA's pretrained visual understanding. This is the "fundamental mismatch" the paper identifies in Section 1: the VLA's rich visual features, acquired during pretraining, are not optimized for synthetic renderings.

Prior Approach 4: Latent World Models

World models are not new — the model-based RL community has been developing them for years. The dominant paradigm, exemplified by Dreamer [16], DreamerV2 [17], and DreamerV3 [18], learns a compact latent dynamics model using a Recurrent State-Space Model (RSSM). The RSSM encodes observations into a low-dimensional latent state, learns to predict how this latent state evolves under actions, and decodes back to observations for reward computation and visualization. Policy optimization then happens entirely in latent space — the policy maps latent states to actions, and the value function estimates returns from latent states.

This approach is computationally efficient and has achieved impressive results on Atari games, DeepMind Control Suite tasks, and even Minecraft. However, it creates a critical representation mismatch when applied to VLA models. VLAs are pretrained on massive datasets of real images and natural language — their visual encoders have learned rich, high-dimensional representations that capture fine-grained visual details relevant to manipulation (object boundaries, contact points, material properties, spatial relationships). When you project observations into a compact latent space via an RSSM and then train the policy on those latent states, you are effectively forcing the policy to learn a new visual representation from scratch within the latent space, discarding the pretrained features that make VLAs powerful in the first place.

The paper makes this argument explicitly in the introduction: "Many classical model-based RL approaches operate in an abstract latent space, which creates a fundamental mismatch with powerful VLA foundation models that are pretrained on real-world images. The rich, pretrained visual understanding of VLAs cannot be directly applied within a mismatched latent dynamics model."

The key insight is that pixel-space predictions matter not just for realism but for representation alignment. When the world model generates realistic pixel-space images and the VLA processes those images through its pretrained encoder, the VLA's features are operating on data that matches their training distribution. When the world model generates latent states, the VLA's encoder must be either discarded or forcibly adapted to a new representation space, losing the benefit of pretraining.

Prior Approach 5: Offline DPO

A simpler approach that avoids both physical interaction and world model training is to use offline preference-based methods like DPO [28]. The idea: collect a batch of trajectories from the base policy, label which ones succeeded and which failed, and fine-tune the policy to prefer successful trajectories using a contrastive loss (the DPO objective). No simulator, no world model, no online interaction required.

The paper's results (Table 1) show why this is insufficient. DPO improves over the base policy — from 33.6% to 37.3% with P=128 and to 42.4% with P=1280 — but it plateaus. Once the offline data is consumed, DPO cannot generate new data from the updated policy's distribution. The policy being optimized is different from the policy that collected the data, creating an off-policy gap: the DPO objective assumes preferences come from the current policy's own distribution, but in offline settings they come from a stale behavior policy. This gap widens as training progresses, limiting improvement. The lifelong learning experiment (Figure 6) confirms this: while WMPO steadily improves with iterative data collection (from 46.9% base to roughly 65% after two iterations), DPO's performance fluctuates unstably and shows no clear upward trend.

How WMPO Positions Itself

WMPO positions itself as a synthesis that addresses the limitations of all prior approaches simultaneously. Rather than choosing between real-world RL (high fidelity, low sample efficiency), simulation (high sample efficiency, low fidelity), or latent world models (medium fidelity, representation mismatch), WMPO proposes a learned pixel-space world model as the imaginary training environment, with three specific design choices that bridge the gaps left by prior work.

First, pixel-space video generation bridges the VLA representation gap. By operating in pixel space rather than a compact latent space, WMPO ensures that the VLA's pretrained visual encoder receives data that matches its training distribution. The world model generates realistic RGB frames; the VLA processes these frames through the same pretrained vision backbone it uses for real images; the resulting visual features are aligned with the policy's pretrained knowledge. This is the crucial distinction from Dreamer-style approaches: WMPO doesn't ask the policy to learn a new representation inside the world model; it brings the world model's outputs into the policy's existing representation space.

Second, policy behavior alignment adapts the world model to the policy's own distribution. A world model trained only on expert demonstrations suffers from the same distribution mismatch as the imitation-learned policy itself — it can simulate successes but not failures. WMPO addresses this by fine-tuning the world model on trajectories collected from the policy's own rollouts. This is a form of on-policy model learning: the world model is adapted to the current policy's state-action distribution, which includes failure modes, near-misses, and recovery attempts that never appear in expert data. Without this adaptation, the paper argues, "the model's imagination of failure cases remains brittle and unfaithful" — the world model would generate plausible-looking trajectories that don't correspond to what the real policy would actually produce.

Third, on-policy GRPO within the world model achieves what real-world on-policy RL cannot. By moving policy optimization entirely into the world model, WMPO eliminates the physical interaction bottleneck. The policy can generate millions of imagined trajectories, receiving dense feedback from a learned reward model, without ever touching a real robot. Critically, because the world model is adapted to the policy's behavior distribution, these imagined trajectories are on-policy — they represent rollouts from the current policy, not a stale behavior policy. This provides the theoretical benefits of on-policy RL (unbiased gradient estimates, no distributional mismatch between data and policy) while achieving the practical benefits of simulation (arbitrary scale, no hardware constraints). The paper emphasizes that "WMPO naturally supports repeated rollouts from the same initial state, which is difficult to realize in the physical world but crucial for large-scale GRPO training."

The paper's claim is not that world models are a new idea — they explicitly cite the Dreamer lineage [16-18], diffusion-based world models [31, 32], and large-scale video models [12, 13] as precursors. The claim is that the specific combination of pixel-space generation (resolving the VLA representation mismatch), policy behavior alignment (resolving the distribution mismatch), and on-policy GRPO (resolving the off-policy bias) enables model-based RL to work effectively for VLA fine-tuning where prior approaches have failed. The extensive experiments — simulation comparisons showing 20+ point improvements over baselines, real-robot validation, generalization tests, emergent self-correction — are designed to substantiate this specific claim, not to claim world models as a novel invention.

This positioning is important because it clarifies what the paper is and isn't contributing. It is not proposing a new world model architecture (the backbone is OpenSora with SDXL VAE, both off-the-shelf). It is not proposing a new RL algorithm (GRPO is from Shao et al., 2024). It is proposing an integration framework and a set of design choices that together solve a concrete, well-motivated problem: how to make VLA policies learn from their own failures without expensive real-world interaction, through careful alignment between the world model, the policy's representations, and the policy's behavioral distribution.

3. Technical Approach

3.1 Reader Orientation

WMPO is a system that lets you train a robot's VLA policy through trial-and-error learning without ever touching a real robot — instead, the robot "practices" inside a learned video world model that simulates what would happen if it took different actions. The core problem it solves is that imitation-learned VLA policies cannot learn from their own mistakes because they need ground-truth demonstrations, and real-world reinforcement learning requires millions of physical robot interactions; WMPO resolves this by inserting a trainable pixel-space world model between the policy and reality, so the policy can fail millions of times in simulation while the world model adapts to faithfully reproduce the policy's actual behavior.

3.2 Big-Picture Architecture (Diagram in Words)

The WMPO training system has four major components connected in a loop:

A base VLA policy (fine-tuned OpenVLA-OFT via imitation learning on expert demonstrations) that maps visual observations and language instructions to action chunks. This is the policy we want to improve.
A video-generative world model (OpenSora backbone with SDXL VAE) that takes a short history of real or imagined frames and a predicted action chunk, and generates the next sequence of frames showing what would happen. This replaces the physical environment.
A lightweight reward model (VideoMAE encoder with a linear classification head) that takes an imagined trajectory and outputs a binary success/failure label. This replaces hand-crafted reward functions.
A GRPO-based policy optimizer that samples groups of imagined trajectories from the current policy inside the world model, evaluates them with the reward model, and updates the policy to prefer trajectories that are more successful relative to their group.

Information flows in a cycle: (a) from a real initial state, the policy predicts an action chunk; (b) the world model consumes that action chunk and the current frame history to generate the next K frames; (c) steps a–b repeat autoregressively until a full trajectory is imagined; (d) the reward model scores the trajectory; (e) multiple such trajectories are generated as a group, their advantages are computed, and the policy is updated via the GRPO objective; (f) periodically, the updated policy collects new real-world trajectories, and these are used to fine-tune the world model (policy behavior alignment), keeping it synchronized with the policy's evolving behavior distribution.

3.3 Roadmap for the Deep Dive

First, the formal problem formulation (MDP) and the overall objective, because it defines what all subsequent components are built to achieve and introduces the key notation.
Second, the generative world model — its architecture, how it generates imagined trajectories autoregressively, and the two techniques (noisy-frame conditioning and frame-level action control) that make long-horizon generation stable — because the world model is what enables the entire approach.
Third, policy behavior alignment — why and how the world model is fine-tuned on the policy's own rollouts — because without this step, the imagined trajectories would not reflect the policy's actual failure modes.
Fourth, the reward model — its architecture, training, and inference — because it provides the learning signal that replaces environment rewards.
Fifth, the on-policy GRPO algorithm — trajectory sampling, dynamic sampling, advantage computation, and the policy update objective — because this is where the actual policy improvement happens.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and integration paper whose core idea is that VLA policies can be improved through on-policy RL entirely within a pixel-space world model, provided that the world model is adapted to the policy's behavioral distribution and that the generation process is stabilized for long horizons.

Problem Formulation: The MDP for VLA Manipulation

WMPO frames the VLA manipulation task as a formal Markov Decision Process (MDP) $\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R})$ , which provides the mathematical scaffolding for reinforcement learning. The four components of this MDP are defined in Section 3.1 as follows.

State space $\mathcal{S} = \mathcal{I} \times \mathcal{G}$ . This is a product space: $\mathcal{I}$ is the image observation space (sequences of RGB frames $I_{0:K}$ ), and $\mathcal{G}$ is the language instruction space (the natural language command describing the task, e.g., "Insert the square into the stick"). The paper makes an explicit simplifying assumption here: "robot states can be solely defined by their image observations." This means the policy does not receive proprioceptive information (joint angles, end-effector positions, gripper status) or wrist-camera inputs — it works entirely from third-person visual input plus language. This is a practical choice that simplifies the system (no sensor fusion, no proprioception-vision alignment) and tests whether visual information alone suffices for the policy and world model to represent task-relevant state. The paper acknowledges this limitation via a footnote: for partially observable settings (POMDPs), the current formulation would need extension, which is left to future work.

Action space $\mathcal{A}$ . The policy outputs action chunks rather than individual instantaneous actions. An action chunk $a_t \in \mathbb{R}^{K \times D}$ is a sequence of $K = 8$ actions, each being a $D$ -dimensional control vector representing the robot's degrees of freedom. For policy optimization, each dimension of each action is discretized into 256 bins. This means the policy does not predict continuous values; instead, it predicts a categorical distribution over 256 possible values for each action dimension. The log-probability of an action chunk is the sum of log-probabilities across all $K$ actions and all $D$ dimensions, treating each discretized dimension independently. The policy is parameterized as $\pi_\theta(a \mid s)$ — a conditional distribution over action chunks given a state.

Transition function $\mathcal{P} : \mathcal{S} \times \mathcal{A} \to \mathcal{S}$ . In standard RL, this would be the physical environment dynamics. In WMPO, it is replaced by a learned world model $p_\phi(s_{t+1} \mid s_t, a_t)$ — a parameterized generative model that produces the next observation given the current observation and action. The key shift here is that state transitions no longer require physical robot execution; they are simulated by a neural network. The world model is trained to approximate the true environment dynamics, but (crucially) it can be updated as the policy's behavior changes.

Reward function $\mathcal{R}$ . Instead of a hand-designed reward (which would require engineering per-task reward functions and risk reward hacking — where the policy finds unintended ways to maximize the reward signal without actually solving the task), WMPO uses a learned reward model $R_\psi(\tau) \in \{0, 1\}$ that outputs a binary success/failure label given a full trajectory $\tau$ . This is a sparse, outcome-based reward: the model evaluates the entire trajectory (not individual steps) and returns 1 if the task was completed successfully and 0 otherwise. There are no intermediate rewards (shaping) — the policy must learn to achieve the terminal goal.

The imagined trajectory $\tau = \{s_0, \hat{s}_1, \ldots, \hat{s}_T\}$ is generated by iteratively alternating between policy and world model: at each step, the policy samples an action $a_t \sim \pi_\theta(a_t \mid s_t)$ , and the world model samples the next state $\hat{s}_{t+1} \sim p_\phi(s_{t+1} \mid \hat{s}_t, a_t)$ , starting from a real initial state $s_0$ . The hat notation $\hat{s}$ distinguishes imagined (model-generated) states from real states.

The overall learning objective is:

$\max_\theta \mathbb{E}_{\tau \sim \pi_\theta, p_\phi} \left[ R_\psi(\tau) \right]$

where the expectation is over trajectories sampled from the policy $\pi_\theta$ and the world model $p_\phi$ , and $R_\psi(\tau) \in \{0, 1\}$ is the reward model's binary success prediction.

What it computes: the expected success rate of the policy when executed inside the world model. The maximization over $\theta$ means we are searching for policy parameters that produce trajectories the reward model judges as successful. Since $R_\psi$ is binary, this objective is the probability of task success under the imagined dynamics.

Why this form: the expectation couples the policy and the world model — the policy chooses actions, and the world model determines their consequences. This decouples policy optimization from physical reality: all terms in the objective ( $\pi_\theta$ , $p_\phi$ , $R_\psi$ ) are neural networks evaluated on a computer. No real robot steps are required to compute or optimize this objective. The binary reward form avoids reward shaping errors — there is no risk of the policy exploiting intermediate reward signals because there are no intermediate rewards. The binary outcome is unambiguous (the task was either completed or not), which simplifies reward model training and reduces reward hacking compared to continuous reward functions.

A critical design choice is that $p_\phi$ and $R_\psi$ are not static pretrained models — the world model $p_\phi$ is fine-tuned on the policy's own rollouts (policy behavior alignment, Section 3.2), and the reward model $R_\psi$ is trained on real trajectories from the target tasks. This ensures that the objective landscape the policy optimizes over actually corresponds to the outcomes the policy would experience in reality.

Generative World Model: Architecture and Imagined Trajectory Generation

The world model is the central component that makes WMPO possible — it replaces the physical environment and must generate realistic, physically plausible future frames conditioned on actions. Section 3.2 describes both the generation procedure and the architectural choices.

Imagined trajectory generation procedure. The process is autoregressive and operates at the level of action chunks and frame chunks:

Input: $c = 4$ initial conditioning frames $I_{0:c}$ from a real environment state, plus a language instruction $g$ .
Policy step: the policy $\pi_\theta$ takes the most recent $m$ frames $I_{i-m:i}$ and the instruction $g$ as input, and predicts an action chunk $a_{i:i+K} \sim \pi_\theta(I_{i-m:i}, g)$ of length $K = 8$ .
World model step: the world model $p_\phi$ conditions on the last $c = 4$ observed frames $I_{i-c:i}$ and the predicted action chunk $a_{i:i+K}$ to generate the next $K = 8$ frames:

$I_{i:i+K} \sim p_\phi(I_{i-c:i}, a_{i:i+K})$

Repeat: steps 2–3 continue autoregressively — the newly generated frames become the conditioning input for the next iteration — until a maximum length $N$ is reached, yielding a complete imagined trajectory $\tau = \{I_{0:N}, a_{0:N}\}$ .

The paper carefully specifies the frame indexing to avoid confusion: $i$ is used for frame-level indices, $t$ for state-level indices (where a "state" often spans multiple frames), $N$ is the maximum frame-level length, and $T$ is the maximum state-level length.

Why action chunks and frame chunks rather than single actions and frames: generating one frame at a time would require the policy and world model to operate at a much higher frequency, increasing computational cost and making temporal consistency harder to maintain. The chunk size $K = 8$ balances temporal resolution (fine enough to capture manipulation dynamics) with computational efficiency (coarse enough to reduce the number of autoregressive steps). The conditioning length $c = 4$ provides enough temporal context for the world model to infer object velocities, contact states, and action effects without being so long that it becomes computationally expensive or susceptible to accumulating irrelevant history.

World model architecture. The world model is built on top of a video diffusion backbone inherited from OpenSora [35], with two key modifications designed specifically for simulating robot-object interactions.

Base architecture: The diffusion model operates in the latent space of a VAE (Variational Autoencoder) — frames are encoded into a compressed latent representation, the diffusion process adds and removes noise in this latent space, and the final latents are decoded back to pixel-space RGB frames. This is standard for video diffusion: working in latent space is computationally cheaper than working in pixel space directly, and the VAE's compression removes high-frequency noise while preserving semantic content.

Modification 1 — 2D VAE instead of 3D VAE: OpenSora originally uses a 3D VAE (which compresses across spatial and temporal dimensions simultaneously). The authors replace this with the 2D VAE from SDXL [36]. The rationale: 3D VAEs achieve high compression by applying aggressive temporal downsampling, which can introduce temporal distortions — frames may blur together, fast motions may be smoothed out, and fine-grained temporal details (critical for robot-object interactions like contacts, slips, and insertion events) may be lost. A 2D VAE encodes each frame independently (no temporal compression), preserving per-frame spatial detail at the cost of higher latent dimensionality per frame. This trade-off is worth making because robot manipulation depends on precisely capturing when and where objects make contact, how grippers close, and whether an insertion succeeds — all of which require high temporal fidelity.

Modification 2 — frame-level action control via extended AdaLN: To ensure that the generated frames faithfully reflect the commanded actions, the authors extend the standard AdaLN (Adaptive Layer Normalization) mechanism to inject action signals at the frame level, not just globally.

In a standard diffusion transformer, each block contains multi-head attention (MHA) or feed-forward (FFN) sublayers, each preceded by Layer Normalization. AdaLN modulates the normalized features by predicting scale and shift parameters from conditioning information (typically the diffusion timestep). The authors extend this: for each action $a_i$ corresponding to frame $i$ , an MLP produces three modulation coefficients:

$\gamma_i^1$ and $\beta_i^1$ : scale and shift applied to the LayerNorm output.
$\alpha_i^1$ : a scale factor for the residual connection.

The update rule within each transformer block at frame $i$ is:

$x_i = x_i + (1 + \alpha_i^1) \cdot \text{Block}\left( \gamma_i^1 \cdot \text{LayerNorm}(x_i) + \beta_i^1 \right)$

where $x_i$ is the feature representation at frame $i$ , and $\text{Block}(\cdot)$ is either the MHA or FFN sublayer.

What it computes: the action signal $a_i$ is converted (via an MLP) into per-frame modulation parameters that control how strongly each transformer sublayer transforms its input and how much of that transformation is added to the residual stream. When the action indicates large movement, the modulation can amplify the block's output; when the action indicates stillness, the modulation can suppress it. This is applied at every frame independently, so frame 3 can be modulated differently from frame 4 based on their respective actions in the action chunk.

Why this form: global conditioning (e.g., a single action embedding injected at the start of the sequence) would make it difficult for the model to learn which frames correspond to which actions — the model would need to infer temporal alignment internally, which is unreliable. Frame-level conditioning makes the alignment explicit: the action signal for frame $i$ directly influences the features at frame $i$ . This is inspired by IRASim [37], which demonstrated that frame-level action control significantly improves action-frame alignment in video generation for robotics. The residual scaling factor $(1 + \alpha_i^1)$ follows the standard AdaLN formulation — when $\alpha_i^1 = 0$ , the block behaves normally; when $\alpha_i^1 > 0$ , the block's contribution is amplified; when $-1 < \alpha_i^1 < 0$ , it is attenuated. This gives the model fine-grained control over how much each action influences each frame's features.

Noisy-frame conditioning for long-horizon stability. A fundamental challenge in autoregressive generation is error accumulation: the world model generates frame 5 conditioned on generated frame 4 (which already contains errors), then generates frame 6 conditioned on frame 5 (which now contains compounded errors), and so on. Over long horizons (hundreds of frames), these errors compound and the generated trajectory diverges from anything physically plausible — objects drift, contacts are lost, the scene degrades into noise.

The paper's solution is noisy-frame conditioning. During training, when the world model is asked to generate frames conditioned on previous frames $I_{i-m:i}$ , those conditioning frames are not provided clean. Instead, they are perturbed with diffusion noise at approximately 50 out of 1000 timesteps (so approximately 5% of the full noise schedule). This means the model learns to generate plausible next frames even when the conditioning frames are imperfect — it learns to be robust to the kind of errors it will encounter at test time when conditioning on its own generated outputs.

The paper describes the training procedure as: "during training, conditional frames $I_{i-m:i}$ are perturbed with diffusion noise at 50/1000 steps rather than kept clean." The ratio 50/1000 corresponds to a moderate noise level — not so high that the conditioning frames become unrecognizable (which would destroy the signal), but high enough that the model must learn to handle imperfections.

Why this works: this is a form of input-level regularization or data augmentation specific to diffusion models. By training the model to denoise from partially-noisy conditioning, we are effectively teaching it to perform error correction — to look at slightly corrupted conditioning frames and still infer the correct dynamics. At test time, when the conditioning frames contain autoregressive errors, the model treats those errors as noise and denoises through them, maintaining stable long-horizon generation. The paper reports that with this technique, the world model "achieves stable long-horizon generation, producing trajectories of hundreds of frames without noticeable quality loss." Without it, the quality would degrade progressively.

Policy Behavior Alignment: Adapting the World Model to the Policy's Distribution

A world model pretrained on expert demonstrations alone faces a critical distribution mismatch: expert demonstrations contain only successful trajectories, but the policy being optimized will produce failures, near-misses, and recovery attempts that look very different from expert behavior. If the world model is asked to simulate a trajectory where the policy makes a mistake, but the world model has never seen that kind of mistake during training, its predictions will be unreliable — it might generate physically impossible outcomes, ignore the action entirely, or hallucinate a successful outcome when the real robot would fail. As the paper states: "without this adaptation, the model's imagination of failure cases remains brittle and unfaithful."

What policy behavior alignment is. It is the process of fine-tuning the world model on trajectories collected from the current policy (not from experts). Specifically:

Collect real trajectories: deploy the current policy $\pi_\theta$ on the real robot (or in simulation), executing full task attempts. These trajectories include both successes and failures — they represent the policy's actual behavior distribution.
Fine-tune the world model: continue training $p_\phi$ on these policy-collected trajectories, using the same diffusion objective and architecture. The model learns to reproduce the visual dynamics of what actually happens when this specific policy acts, including failure modes.

The paper uses $P = 128$ and $P = 1280$ as rollout budgets — these are the numbers of real trajectories collected and used for fine-tuning.

Why this is necessary. Consider what happens if you skip this step. The world model is pretrained on Open X-Embodiment [21] — a large dataset of diverse robot demonstrations across many embodiments, tasks, and environments. This gives it broad knowledge of robot dynamics. But those demonstrations are almost exclusively successful trajectories. When the current policy attempts a task and makes a mistake (e.g., it grips the object too loosely and drops it, or it approaches from the wrong angle and collides), the pretrained world model has no training examples of the visual consequences of those mistakes. It might:

Generate frames showing the object magically staying in the gripper because that's what always happens in expert data.
Generate physically incoherent frames because the state-action pair is out of distribution.
Generate a smooth but incorrect trajectory that doesn't correspond to reality.

In any case, the policy receives a misleading learning signal: it sees itself "succeeding" in imagination when it would actually fail in reality. This destroys the entire premise of model-based RL — the policy cannot learn to avoid failures if the world model never shows failures.

Why fine-tuning on only 128–1280 trajectories suffices. The world model is pretrained on millions of trajectories from OXE, which gives it strong priors about how robots and objects move in general. Fine-tuning on the policy's rollouts doesn't need to teach basic physics from scratch — it only needs to adapt the model to this specific policy's behavioral idiosyncrasies (e.g., how this policy tends to fail, what its typical action distribution looks like, what visual patterns precede failures). This is a form of domain adaptation: the coarse dynamics are shared across policies, but the fine details of the state-action distribution are policy-specific.

The paper's experiments in Table 1 validate this design: with $P=128$ (only 128 real trajectories), WMPO already achieves substantial improvements (47.1% vs. 33.6% base), showing that a modest amount of policy-specific data is sufficient for effective alignment.

Reward Model: Learned Sparse Binary Success Classification

The reward model $R_\psi$ provides the learning signal for policy optimization. Instead of hand-crafting a reward function (which would require per-task engineering and risks reward hacking, where the policy finds unintended ways to maximize the reward signal), WMPO learns a binary success classifier from real trajectory data.

Training data construction. Given a real trajectory $\tau = \{I_{0:N}\}$ (a sequence of frames from a real robot execution), the training data is constructed as follows:

Positive samples: the terminal clip of a successful trajectory — a sequence of $L = 8$ frames at the end of the trajectory, $c_N = I_{N-L:N}$ , showing the completed task state. This captures what "success looks like": the object in the goal configuration, the assembly completed, the peg inserted.
Negative samples: two sources. First, clips $\{c_i = I_{i-L:i} : L \leq i \leq N-L\}$ from anywhere except the very end of successful trajectories — these show intermediate states that, while on the path to success, don't yet show the completed task (e.g., the robot reaching for the object, mid-grasp, approaching the target). Second, arbitrary clips from failed trajectories — these show states that lead to or represent failure.
Class balancing: the numbers of positive and negative samples within each training batch are equalized to address class imbalance (in practice, there are typically more negative clips than positive ones).

Model architecture. The reward model uses a VideoMAE [39] encoder — a Vision Transformer pretrained on video with masked autoencoding, which learns spatio-temporal representations from video data. A linear classification head is placed on top of the encoder to produce a single scalar logit, which is passed through a sigmoid to produce a success probability.

Training objective. The model is trained with binary cross-entropy loss:

$\mathcal{L}_\psi = -\left[ y \log(\hat{y}) + (1 - y) \log(1 - \hat{y}) \right]$

where $y \in \{0, 1\}$ is the binary ground-truth label (1 for success clips, 0 for failure clips) and $\hat{y} \in [0, 1]$ is the model's predicted success probability.

What it computes: the standard binary cross-entropy between the predicted probability and the true label. This trains the classifier to output probabilities that are well-calibrated — high for clips showing task completion, low for clips showing failure or non-terminal states.

Why this form: binary cross-entropy is the maximum-likelihood objective for a Bernoulli target. It strongly penalizes confident mistakes (predicting 0.99 when the label is 0, or 0.01 when the label is 1), which is appropriate because confident misclassifications of success or failure would directly mislead the policy.

Inference procedure. At inference time, the reward model processes a full trajectory $\tau$ by applying a sliding window over the frame sequence. For each clip of length $L = 8$ extracted with stride $s = 1$ (so adjacent clips overlap by 7 frames), the model produces a success probability. The trajectory is classified as successful if any clip's probability exceeds a threshold $\tau_{\text{thr}}$ . The threshold is selected via validation experiments.

Why sliding window with max-over-clips: the alternative would be to feed the entire trajectory to the model and get a single prediction. But trajectories can be long (hundreds of frames), and the successful completion state may occupy only a small fraction of the total frames. A model processing the full trajectory would need to identify the relevant temporal region amid a sea of intermediate frames, which is a harder learning problem. The sliding window approach decomposes the problem: each short clip is independently classified, and the trajectory succeeds if any clip looks like a completion state. This also makes the reward model more robust — if the trajectory includes a successful completion followed by irrelevant post-completion frames, the completion clip will still get a high score. The stride of 1 ensures dense coverage so no short success moment is missed.

The paper reports that the reward model achieves "an F1 score above 0.95 across all tasks, reliably distinguishing success from failure and effectively mitigating reward hacking." This high accuracy is crucial — if the reward model were inaccurate, the policy would optimize for a corrupted signal and learn unintended behaviors.

On-Policy GRPO within the World Model

The final component is the reinforcement learning algorithm that actually updates the policy parameters $\theta$ to maximize the objective from Section 3.1. WMPO uses Group Relative Policy Optimization (GRPO) [30], an on-policy algorithm that has been shown to be stable and scalable in large language model training [22]. The entire optimization loop happens within the world model — no real robot interactions are needed during the policy update steps.

Why GRPO and not PPO or DPO. The paper explicitly motivates this choice (Section 3.4). On-policy methods (like PPO and GRPO) are "often favorable for better performance" because they avoid the value estimation bias that plagues off-policy methods [43] — off-policy algorithms must estimate the value of actions from data collected by a different (older) policy, which introduces systematic errors. DPO is an offline method that cannot generate new on-policy data, limiting its improvement ceiling. GRPO specifically is chosen because it "provides stable and scalable training in settings with sparse rewards" — it doesn't require a learned value function (unlike PPO, which needs a critic network to estimate state values), making it simpler and more robust to value estimation errors. Additionally, GRPO naturally handles the group-based comparison structure (comparing trajectories within a group rather than against a value baseline), which is well-suited to the binary success/failure reward structure.

Trajectory sampling procedure. For each policy update, a batch of imagined trajectories is generated:

Sample initial states: from the real environment dataset $\mathcal{D}$ , sample initial frames $I_{0:c}$ and language instructions $g$ . These are real initial states (the first few frames of a task attempt), ensuring the imagined trajectories start from realistic configurations.
Sample a group of $G = 8$ trajectories: from the current policy $\pi_{\theta_{\text{old}}}$ (the policy parameters before the update), generate $G$ independent imagined trajectories $\{\tau_1, \ldots, \tau_G\}$ by autoregressively alternating between policy action prediction and world model frame generation, as described in Section 3.2.
Score trajectories: the reward model $R_\psi$ processes each trajectory and outputs binary success/failure labels $R_i \in \{0, 1\}$ .
Dynamic sampling filter: if all $G$ trajectories in a group are predicted as successful (all $R_i = 1$ ) OR all are predicted as unsuccessful (all $R_i = 0$ ), the entire group is discarded. The process repeats — more groups are sampled from new initial states — until a full batch of $B = 64$ trajectories (across all kept groups) is populated.
Precompute reference log-probabilities: for each action chunk $a_t$ in each kept trajectory, compute and store the log-probability under the old policy $\pi_{\theta_{\text{old}}}$ . This is needed for the probability ratio computation during the policy update.

The log-probability of an action chunk is the sum across all $K$ actions in the chunk and all $D$ degrees of freedom:

$\log \pi_{\theta_{\text{old}}}(a_t \mid s_t) = \sum_{i=1}^{K} \sum_{j=1}^{D} \log \pi_{\theta_{\text{old}}}\left( a_t^{i,j} \mid s_t \right)$

where $a_t^{i,j}$ is the $i$ -th action in the $j$ -th degree of freedom within the action chunk at time $t$ . This factorization assumes each discretized action dimension is conditionally independent given the state (a common assumption in autoregressive action prediction).

What dynamic sampling achieves and why it matters. Groups where all trajectories have the same outcome (all success or all failure) provide no relative comparison signal for the policy. If every trajectory in a group succeeded, the policy has no information about which actions were better — it only knows that its current behavior works, but not whether some variations work better. If every trajectory failed, the policy has no positive examples to learn from — it only knows that its current behavior doesn't work, but not which direction to move. By discarding these uninformative groups, dynamic sampling ensures that every group used for training contains both successes and failures, providing a meaningful contrastive signal. This is analogous to the filtering used in DAPO [44] and addresses the problem of vanishing gradients that can occur when the advantage is zero for all trajectories in a group.

Advantage computation. For each trajectory $\tau_i$ in a kept group, the advantage $\hat{A}_i$ is computed using group-based normalization:

$\hat{A}_i = \frac{R_i - \text{mean}(\{R_i\}_{i=1}^{G})}{\text{std}(\{R_i\}_{i=1}^{G})}$

where $R_i = R(\tau_i) \in \{0, 1\}$ is the binary reward, and the mean and standard deviation are computed over the $G = 8$ trajectories in the group.

What it computes: the advantage measures how much better or worse trajectory $\tau_i$ is compared to the average trajectory in its group, expressed in units of standard deviation. A successful trajectory in a group where most trajectories failed will have a large positive advantage; a successful trajectory in a group where most succeeded will have a small positive advantage; a failed trajectory in a mostly-successful group will have a large negative advantage.

Why group normalization instead of a learned value baseline. PPO and other actor-critic methods compute advantage as $R - V(s)$ , where $V(s)$ is a learned state-value function. This requires training a separate critic network and introduces value estimation errors. GRPO replaces the learned value baseline with the sample mean of the group — a simple, hyperparameter-free baseline that is always unbiased within the group. The division by standard deviation normalizes the scale of advantages across groups, stabilizing training. This is particularly appropriate for sparse binary rewards: with only two possible values (0 and 1), the group mean is a natural reference point, and the advantage reflects the relative frequency of success within the group.

Policy update objective. The policy parameters $\theta$ are updated to maximize a clipped surrogate objective, following the standard PPO/GRPO formulation:

$\mathcal{J}(\theta) = \mathbb{E}_{s_0 \sim \mathcal{D}, \{\tau_i\}_{i=1}^{G} \sim \pi_{\theta_{\text{old}}}} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{T} \sum_{t=0}^{T} \min\left( r_{i,t}(\theta) \hat{A}_i, \ \text{clip}(r_{i,t}(\theta), 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}}) \hat{A}_i \right) \right]$

where:

$\tau_i$ is the $i$ -th trajectory in a group of size $G = 8$ .
$T$ is the trajectory length (number of action chunks).
$\hat{A}_i$ is the group-normalized advantage for trajectory $\tau_i$ .
$r_{i,t}(\theta)$ is the probability ratio between the new and old policies for action chunk $t$ in trajectory $i$ :

$r_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t} \mid s_{i,t})}{\pi_{\theta_{\text{old}}}(a_{i,t} \mid s_{i,t})}$

$\epsilon_{\text{low}} = 0.20$ and $\epsilon_{\text{high}} = 0.28$ are asymmetric clipping thresholds.
The outer expectation is over initial states sampled from the real dataset and trajectories sampled from the old policy.

What it computes: the objective is a per-timestep weighted average over trajectories, where the weight for each trajectory is its advantage $\hat{A}_i$ . The probability ratio $r_{i,t}(\theta)$ measures how much more (or less) likely the new policy is to take action $a_{i,t}$ in state $s_{i,t}$ compared to the old policy. Multiplying by the advantage means: if the advantage is positive (the trajectory was better than group average), we want to increase the probability of the actions in that trajectory; if negative, we want to decrease it.

The $\min$ and $\text{clip}$ operations implement a trust region: the probability ratio is clipped to the range $[1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}}] = [0.80, 1.28]$ . This prevents the new policy from deviating too far from the old policy in a single update. When the ratio exceeds the clipping range, the gradient is zero for that action (the $\min$ selects the clipped value, which has no dependence on $\theta$ ), preventing overly aggressive updates.

Why asymmetric clipping ( $\epsilon_{\text{low}} = 0.20$ , $\epsilon_{\text{high}} = 0.28$ ). Standard PPO uses symmetric clipping ( $\epsilon = 0.2$ in both directions). The paper follows DAPO [44] in using asymmetric thresholds, allowing slightly larger increases in probability (up to 28% per update) than decreases (up to 20% per update). The intuition: when the policy finds a good action, we want to reinforce it strongly; when it finds a bad action, we want to suppress it but not so aggressively that we collapse exploration. The asymmetry also reflects the fact that in discrete action spaces, probability mass must sum to 1 — increasing one action's probability necessarily decreases others, so asymmetric clipping provides finer control over this redistribution.

No KL divergence regularization. The paper explicitly states that it "remove[s] the KL divergence regularization so that no reference model is required during training." This is a departure from standard GRPO implementations (which include a KL penalty against a reference policy to prevent catastrophic forgetting) and follows the DAPO variant [44]. The benefit is practical: "reducing memory consumption and encouraging the policy to explore novel behaviors." Without a reference model, the GPU memory footprint is smaller (no second copy of the policy in memory), and the policy is free to deviate further from its initialization, which is desirable because the goal is precisely to learn behaviors (self-correction, recovery) that the base imitation-learned policy does not exhibit. The trust region clipping provides sufficient stability without the KL penalty.

Training hyperparameters. The GRPO configuration is specified in Table 4 of the paper:

Optimizer: AdamW with $\beta_1 = 0.9$ , $\beta_2 = 0.999$ .
Learning rate: $5 \times 10^{-6}$ .
Training batch size: $B = 64$ trajectories (after dynamic sampling).
Group size: $G = 8$ .
Mini-batch size: $M = 128$ .
Clip ratios: $\epsilon_{\text{low}} = 0.20$ , $\epsilon_{\text{high}} = 0.28$ .
Temperature: $1.6$ (for the policy's action sampling distribution — higher temperature increases exploration by making the action distribution more uniform).

Why a temperature of 1.6. The base policy is trained via imitation learning, which typically uses low sampling temperatures (close to 0) to produce deterministic, expert-like behavior. During RL, higher temperature is needed for exploration — the policy must try actions it wouldn't normally take to discover whether they lead to better outcomes. A temperature of 1.6 makes the action distribution substantially more stochastic than greedy decoding, encouraging the kind of variation that dynamic sampling needs (diverse outcomes within a group).

The training loop in full (Algorithm 1 in Appendix A). The overall procedure alternates between:

Sampling phase: for each initial state sampled from the real dataset, generate $G = 8$ imagined trajectories from $\pi_{\theta_{\text{old}}}$ inside the world model. Apply dynamic sampling to discard uninformative groups. Continue until a batch of $B$ trajectories is accumulated.
Update phase: compute reference log-probabilities, advantages, and the GRPO objective $\mathcal{J}(\theta)$ . For $E$ epochs (the paper doesn't specify $E$ explicitly but it is standard in PPO/GRPO to use a few epochs, typically 1–5), iterate over mini-batches of size $M$ and update $\theta$ via gradient ascent on $\mathcal{J}(\theta)$ .
Synchronization: set $\theta_{\text{old}} \leftarrow \theta$ .
Periodic world model alignment: after some number of policy updates, collect new real trajectories from the updated policy and fine-tune the world model on them, maintaining the policy behavior alignment.

This loop can iterate indefinitely, enabling the continuous self-improvement demonstrated in the lifelong learning experiment (Figure 6), where alternating between policy updates and world model updates leads to steady performance gains without any additional expert demonstrations.

4. Key Insights and Innovations

Innovation 1: Pixel-Space World Models as Representation-Aligned Training Environments — Not Just Realistic Simulators

The dominant prior paradigm in model-based RL for visual control, exemplified by the Dreamer lineage [16–18], operates on a compact latent state learned by a Recurrent State-Space Model (RSSM). In those frameworks, the policy ingests abstract latent vectors—compressed representations that discard most of the pixel-level detail of the original observations. This is computationally convenient (latent rollouts are cheap) and has proven effective for Atari and continuous control from pixels. However, when applied to VLA models, this approach creates what the paper identifies as a "fundamental mismatch": VLAs are pretrained on web-scale real images and possess rich, high-dimensional visual representations tuned to real-world textures, lighting, object boundaries, and spatial relationships. Forcing a VLA to operate on RSSM latents means either discarding its pretrained visual encoder entirely or training it to interpret a representation space it was never designed for.

WMPO's central intellectual move is to flip this relationship. Instead of asking the policy to adapt to the world model's representation (by projecting observations into a compact latent space), WMPO demands that the world model produce outputs in the representation space the policy already understands — raw pixel-space RGB frames. The world model generates realistic images via a video diffusion backbone (OpenSora with SDXL VAE, Section 3.2); the VLA processes these images through its frozen pretrained vision encoder; the resulting features align with the distribution the VLA was trained on. The world model becomes a drop-in replacement for a camera feed, not a new representation to learn.

What makes this distinctive at the idea level. The insight is not "pixel-space generation looks better" — it's that the value of pixel-space generation lies in representation alignment, not visual fidelity per se. A latent world model could generate realistic-looking frames via a decoder without ever exposing the policy to pixels during training (Dreamer does this for visualization). The critical difference in WMPO is that the policy sees those pixels during optimization, through its own pretrained encoder, and therefore the gradients flowing back through the policy update reflect the VLA's actual visual processing pipeline. Prior work on diffusion world models for RL [31, 32] had shown that pixel-level fidelity matters for Atari with Gaussian policies, but those policies were trained from scratch on the generated images. WMPO's contribution is showing that for pretrained vision-language policies, pixel-space generation serves a different purpose: it prevents the representational knowledge acquired during large-scale pretraining from being discarded or misaligned at the world model interface.

Comparison to prior assumptions. The Dreamer lineage implicitly assumes that a compact latent state is a sufficient representation for control — that whatever visual details matter for decision-making will be captured in the latent bottleneck. This is a reasonable assumption when the policy learns its visual features from scratch inside the world model (as Dreamer agents do). But for VLAs, which invest enormous pretraining compute into learning general-purpose visual representations from internet-scale data, the assumption breaks. The VLA's features encode distinctions (material properties, fine contact geometries, subtle spatial relationships) that a compact RSSM latent — optimized primarily for reconstruction and reward prediction — may not preserve. By keeping the world model's outputs in pixel space, WMPO lets the VLA's own feature extractor determine what visual information is task-relevant, rather than the world model's latent bottleneck making that determination implicitly.

Significance beyond performance. This is fundamentally a representation-level diagnostic. It tells the field that when moving from training-control-policies-from-scratch to fine-tuning-pretrained-foundation-models, the interface between the world model and the policy matters as much as the world model's accuracy. The 20+ percentage point improvements over baselines (Table 1) are downstream evidence, but the deeper contribution is identifying why latent world models were insufficient for VLAs specifically: not because they modeled dynamics poorly, but because they severed the VLA's connection to its pretrained visual knowledge. This diagnostic has implications beyond robotics — any domain where a pretrained foundation model must be fine-tuned via model-based RL (medical imaging, autonomous driving, scientific experimentation) faces the same representation-alignment question.

Anchoring evidence. The paper does not directly ablate pixel-space vs. latent-space world models (this would be a natural experiment, and its absence is a limitation). The evidence for this claim is therefore architectural: the entire WMPO framework is built around the premise that the VLA processes generated frames through its pretrained encoder, and the strong results (47.1% vs. 33.6% base at P=128; 57.6% at P=1280, Table 1) provide indirect validation. The real-world experiment (Figure 7) further supports this: the world model generates pixel-space trajectories that, when processed by the VLA's vision encoder during policy optimization, lead to a policy that transfers successfully to the physical robot (70% success vs. 53% base).

Innovation 2: Policy Behavior Alignment as a Principled Solution to the Expert-Demonstration Distribution Mismatch in World Model Training

World models for robotics are typically trained on datasets of successful task executions — expert demonstrations from humans or scripted policies. This is natural: demonstration data is abundant (Open X-Embodiment alone contains millions of trajectories [21]), and it captures the dynamics of tasks being performed correctly. But this creates a subtle and severe problem when the world model is used for policy optimization rather than planning: the world model has never seen what happens when the policy makes mistakes. During RL training, the policy will frequently take suboptimal actions — it will drop objects, collide with obstacles, approach from wrong angles, and generally behave in ways that never appear in expert data. If the world model is asked to simulate the visual consequences of these off-distribution actions, its predictions will be unreliable; it may hallucinate physically impossible outcomes, default to expert-like behavior that ignores the policy's mistake, or produce trajectories that diverge from what the real robot would experience. In all cases, the policy receives a corrupted learning signal — it "succeeds" in imagination when it would fail in reality, or "fails" for reasons that don't correspond to actual failure modes.

This is not a hypothetical concern. It is a specific instance of the well-known covariate shift problem in imitation learning [5], projected onto the world model: the state-action distribution of the optimizing policy drifts away from the state-action distribution of the expert demonstrations on which the world model was trained. Prior model-based RL approaches have addressed this through iterative data collection (Dyna-style: interact with real environment, update model, repeat) or by using uncertainty estimates to avoid model exploitation. But these solutions assume access to the real environment for data collection — exactly what WMPO is designed to minimize.

WMPO's innovation is to recognize that the distribution mismatch can be addressed through a surprisingly lightweight mechanism: fine-tune the world model on a small number of trajectories (P=128 to 1280) collected by the current policy itself. These trajectories include both successes and failures — they represent the policy's actual behavioral distribution, not an idealized expert distribution. The world model, pretrained on millions of diverse trajectories from OXE, already understands general robot-object dynamics; fine-tuning adapts it to the specifics of this policy's tendencies: how it grips, when it drops objects, what angles it approaches from, what failure modes look like visually.

What makes this distinctive at the idea level. The key intellectual move is recognizing that the world model doesn't need to be rebuilt from scratch for each new policy — it just needs to be aligned. The world model's pretraining on OXE provides a strong prior over physical dynamics that transfers across embodiments, tasks, and even policy behaviors. Fine-tuning on a small set of policy-specific trajectories adapts this prior to the policy's idiosyncratic distribution. This is conceptually analogous to how foundation models in NLP are fine-tuned on small domain-specific datasets: the pretrained model captures general linguistic competence; fine-tuning adapts it to the target distribution.

Prior work either avoided the distribution mismatch entirely by training world models from scratch on on-policy data (impractical for VLAs due to data requirements), accepted it and hoped the policy would stay close enough to the expert distribution (which fails when RL induces significant behavioral changes), or used ensemble uncertainty to avoid out-of-distribution regions (which restricts exploration). WMPO's alignment approach splits the difference: pay a modest real-world data cost (128–1280 trajectories) to adapt a pretrained world model, then use it for arbitrary-scale imagined rollouts.

Comparison to prior assumptions. The Dreamer family sidesteps this issue by training the world model simultaneously with the policy, collecting new real-environment data at each iteration. This ensures the model stays on-policy, but at the cost of continuous real-world interaction — exactly the bottleneck WMPO aims to remove. Offline model-based RL approaches (like MOPO, COMBO) use uncertainty penalties to keep the policy within regions where the model is reliable, but they are fundamentally conservative: they prevent the policy from exploring regions where the model might be wrong, which limits the behavioral innovations (self-correction, recovery) that WMPO's emergent behaviors demonstrate (Figure 3, Section 4.3). The paper's DPO baseline (Table 1) represents the alternative extreme: no world model at all, just offline preference learning on a fixed dataset — this plateaus because it cannot generate new on-policy data to continue improving.

Significance beyond performance. This innovation establishes policy behavior alignment as a first-class design requirement for model-based RL with foundation models. It says: if your world model was trained on expert data and you want to use it for policy optimization, you must adapt it to your policy's behavior distribution, or the imagined rollouts will be unreliable. The modest data requirement (128 trajectories) makes this practical — it's not asking for millions of on-policy rollouts, just enough to capture the policy's characteristic failure modes and action distribution. This is a fundamental contribution to the methodology of model-based RL for pretrained policies, not just an algorithmic trick.

Anchoring evidence. Table 1 shows that with P=128 real trajectories, WMPO achieves 47.1% mean success (vs. 33.6% base), and with P=1280 it reaches 57.6%. The fact that performance scales with the alignment budget (more policy-specific data → better world model fidelity → better policy optimization) confirms that alignment quality matters. The lifelong learning experiment (Figure 6, Section 4.5) provides the strongest evidence: WMPO improves iteratively (from 46.9% base to ~65% after two iterations) by alternating between policy updates (inside the world model) and world model updates (on new policy-collected data) — each alignment step synchronizes the world model with the evolving policy distribution, enabling the next round of improvement. DPO, lacking this alignment mechanism, shows unstable performance and no clear upward trend.

Innovation 3: The Verifier-Over-Optimization Analogy for World Model Fidelity — Distribution Mismatch as the Test-Time Compute Bottleneck Transposed to Robotics RL

There is a revealing structural analogy between WMPO and the test-time compute scaling literature (as analyzed in the reference example paper on compute-optimal test-time scaling). In that literature, the central bottleneck is verifier over-optimization: as you allocate more test-time compute to search against a learned reward model (PRM), the search process eventually finds solutions that score highly under the verifier but are actually incorrect — the verifier's training distribution (human-labeled correct solutions) diverges from the search distribution (model-generated candidates optimized against the verifier). The solution is difficulty-aware allocation: use aggressive search only on problems where the verifier signal is reliable (medium difficulty), and back off to weaker optimization on problems where the verifier is vulnerable to exploitation (easy difficulty).

WMPO faces an isomorphic problem, transposed to the robotics domain. The "verifier" in WMPO is the learned reward model $R_\psi$ , which provides the binary success/failure signal for policy optimization. The "search process" is the GRPO-based policy optimization that iteratively generates imagined trajectories and updates the policy to increase the reward model's predicted success rate. The over-optimization risk is that the policy learns to produce imagined trajectories that look like successes to the reward model but would not correspond to actual task completion in reality — a form of adversarial exploitation of the reward model's blind spots.

But WMPO faces a deeper and more fundamental version of this problem. The reward model is only one component of the "verifier" — the world model $p_\phi$ is also part of the evaluation pipeline. If the world model's simulation of the policy's actions is inaccurate (because the world model was trained on expert demonstrations and has never seen the policy's failure modes), then the imagined trajectory itself is corrupted. The policy optimizes against a joint (world model, reward model) system, and distribution mismatch in the world model is a more severe failure mode than reward model over-optimization because it corrupts the state transitions rather than just the reward signal.

What makes this distinctive at the idea level. WMPO's policy behavior alignment can be understood as a world model over-optimization mitigation strategy, structurally analogous to the difficulty-aware compute allocation in the test-time compute scaling literature. The insight is that the world model's training distribution (expert demonstrations) and the policy's rollout distribution (mixture of successes and failures) diverge, and this divergence is the primary bottleneck preventing effective model-based RL for VLAs — not the RL algorithm, not the reward model, not the policy architecture. Just as the test-time compute paper showed that verifier robustness, not search algorithm sophistication, is the binding constraint on inference-time scaling, WMPO shows that world model distribution alignment, not RL algorithm design, is the binding constraint on model-based VLA fine-tuning.

The paper does not make this analogy explicitly, but the structural parallel is clear upon analysis. Prior model-based RL work focused on algorithmic solutions to distribution mismatch: ensemble uncertainty penalties (MOPO), conservative value estimates (COMBO), or iterative on-policy data collection (Dyna, Dreamer). WMPO's innovation is to treat the problem as fundamentally one of model training data rather than algorithm design — fix the model's data distribution (by adding policy-specific rollouts) rather than constraining the policy to stay within the model's training distribution. This is a simpler and more direct solution, and the paper's strong results suggest it is also more effective.

Comparison to prior assumptions. The dominant assumption in model-based RL has been that world model inaccuracy is an epistemic uncertainty problem (the model doesn't know what will happen in unfamiliar states) that should be addressed through uncertainty-aware planning or conservative optimization. WMPO's approach treats it as a data coverage problem (the model hasn't seen the right kind of data) that should be addressed through targeted data collection. This is a conceptual reframing with practical consequences: collecting 128 trajectories from the current policy is cheaper and more scalable than implementing sophisticated uncertainty estimation or conservative RL algorithms, and it directly addresses the root cause rather than compensating for its symptoms.

Significance beyond performance. This insight generalizes beyond WMPO to any domain where a learned dynamics model is used for policy optimization. It suggests a design principle: align your world model's training data with your policy's behavioral distribution before attempting optimization. The amount of alignment data needed may be surprisingly small (128 trajectories sufficed in WMPO) if the world model has been pretrained on broad domain data. This principle has implications for model-based RL in autonomous driving (align simulation with the specific driving policy's style), dexterous manipulation (align with the policy's grasp attempts and failure modes), and any domain where the optimizing policy's behavior differs systematically from the data on which the world model was trained.

Anchoring evidence. The ablation is implicit but clear: the base policy achieves 33.6% (Table 1). Online GRPO, which uses real-world trajectories directly (no world model, no distribution mismatch), achieves 33.2% at P=128 — no improvement, because 128 real trajectories are insufficient for on-policy RL without a world model to amplify them. DPO, which uses offline data without a world model, achieves 37.3% at P=128 — modest improvement from preference learning, but plateaus. WMPO achieves 47.1% at P=128 — substantial improvement, possible only because the world model (aligned via those same 128 trajectories) amplifies the RL signal by generating many imagined rollouts per real trajectory. The fact that WMPO dramatically outperforms methods using the same real-data budget demonstrates that the world model alignment is what unlocks the value of the real data, not the data quantity itself.

Innovation 4: Emergent Self-Correction as Evidence That Model-Based RL Can Induce Behaviors Absent from Demonstration Data

A persistent criticism of imitation learning is that it can only reproduce behaviors present in the training data — by definition, it cannot invent new strategies. Reinforcement learning, in principle, can discover novel behaviors through exploration and reward maximization. However, in practice, RL for robotics often struggles to produce qualitatively new behaviors because (a) exploration is expensive and dangerous on real robots, and (b) reward shaping often constrains the policy to stay near known behaviors. WMPO provides a particularly clean demonstration that model-based RL, with sufficient exploration budget and a well-aligned world model, can induce emergent problem-solving strategies that are entirely absent from the demonstration data used to initialize the policy.

The specific evidence is the self-correction behavior documented in Figure 3 and Section 4.3 for the Square task. When the base policy (imitation-learned from expert demonstrations) encounters a collision — the square block contacts the stick at the wrong angle — it persists in pushing forward until the maximum time horizon expires. This is exactly what you'd expect from a policy trained on expert data: collisions never occur in expert demonstrations, so the policy has no training signal for what to do when one happens. It defaults to continuing its current action, which leads to failure.

The WMPO-trained policy, by contrast, learns to lift the square, realign it above the stick, and then insert it correctly. This is a recovery behavior: the policy detects that it is in a failure state (the square is jammed against the stick) and executes a multi-step correction sequence (lift, reposition, reinsert) that leads to task success. This behavior was never demonstrated — the expert data shows only smooth, collision-free insertions. It emerged through RL within the world model because:

The world model, after policy behavior alignment, accurately simulates the visual consequences of collisions (the square stays jammed, the robot doesn't magically succeed).
The reward model correctly labels such trajectories as failures (reward = 0).
GRPO's exploration (temperature 1.6, multiple trajectories per group) causes the policy to occasionally try lifting the square after a collision, just by random variation.
Some of those variation trajectories lead to successful insertion (the square gets realigned and inserted), earning a reward of 1 and receiving a positive advantage.
The policy update reinforces the recovery action sequence, making it more likely in future collisions.

What makes this distinctive at the idea level. This is not a claim that WMPO invented self-correction — recovery behaviors have been observed in RL-trained policies before. What's distinctive is the completeness of the evidence chain. The paper provides (a) a clear behavioral contrast between the base policy and the WMPO-trained policy on the same task (Figure 3, specific frames showing collision and recovery), (b) a mechanistic explanation for why imitation learning cannot produce this behavior while model-based RL can, and (c) a quantitative corroboration: the WMPO-trained policy's successful trajectories are 4–6% shorter on average (Figure 5, Section 4.3), indicating more efficient execution without the "stuck" behavior that characterizes the base policy. The shorter trajectory length is not just a convenience — it's evidence that WMPO has learned to avoid the indecisive, time-wasting micro-adjustments that the base policy exhibits, another emergent behavior not present in the demonstration data.

Comparison to prior assumptions. Prior work on RL for robotics often evaluates success rate as the primary metric, treating behavioral quality as a secondary concern. WMPO's analysis goes beyond "it works better" to "it works differently" — the policy has acquired qualitatively new strategies, not just become more reliable at executing its existing strategy. This matters because it demonstrates that the improvement isn't merely from better calibration or reduced action noise; it's from genuine behavioral innovation. The paper's observation that the policy "rarely becomes 'stuck' in suboptimal states" (Section 4.3) and that trajectories are "faster and smoother" points to a form of temporal efficiency learning — the policy has learned to minimize time-to-completion, not just maximize success probability, because the sparse binary reward incentivizes reaching the goal state efficiently rather than dawdling.

Significance beyond performance. This finding has implications for how we think about the relationship between demonstration data and learned behavior. It suggests that imitation learning provides a useful initialization — a policy that understands the task and can sometimes succeed — but that RL within a world model can transcend that initialization, discovering strategies that the demonstrators never exhibited. This is important for practical deployment: human teleoperators may not think to demonstrate recovery behaviors (because they naturally avoid collisions), but a robot operating autonomously needs those behaviors. WMPO shows that they can be learned without explicit demonstration, as long as the world model faithfully simulates the consequences of mistakes.

Anchoring evidence. Figure 3 provides the qualitative visual evidence: specific frames from both the base policy and WMPO showing the divergent behaviors at the collision point. Figure 5 provides the quantitative corroboration: WMPO trajectories are 4–6% shorter than the base policy, and also shorter than DPO and GRPO trajectories. The fact that all three comparison methods (base, DPO, GRPO) produce longer trajectories than WMPO suggests the efficiency gain is specific to the model-based RL training, not a general property of any post-training optimization.

Innovation 5: Degenerate Solution Filtering via Dynamic Sampling — Importing a Training-Stability Insight from LLM RL to Robotics

The final innovation is more technical but conceptually significant: WMPO's adoption of dynamic sampling from the LLM RL literature [44] represents a diagnostic transfer between domains that solves a specific failure mode of sparse-reward policy optimization.

The problem: when optimizing a policy with binary success/failure rewards, many groups of trajectories will be uninformative — all successes (the policy consistently solves the task from that initial state, providing no contrast between better and worse rollouts) or all failures (the policy consistently fails, providing no positive signal to imitate). Standard RL algorithms handle this by averaging over large batches, but when the batch contains mostly uninformative groups, the signal-to-noise ratio degrades and training becomes inefficient or unstable.

Dynamic sampling (Section 3.4) addresses this by simply discarding groups where all G=8 trajectories have the same outcome, sampling additional initial states until a full batch of B=64 informative trajectories is accumulated. This is a conceptually trivial operation — just filter out degenerate groups — but it has a non-trivial effect on training dynamics: every group that contributes to the gradient update contains both successes and failures, ensuring that the advantage $A_i$ reflects a meaningful contrast between better and worse behavior.

What makes this distinctive at the idea level. This is a case of domain transfer rather than invention. Dynamic sampling was introduced in DAPO [44] for LLM training, where groups of model outputs are scored by a reward model (e.g., correctness on math problems) and uninformative groups are filtered. WMPO recognizes that the identical problem structure exists in model-based RL for robotics: groups of imagined trajectories are scored by a binary reward model, and groups with uniform outcomes provide no learning signal. The transfer is natural because, in both settings, the "environment" for RL is actually a learned model (the LLM's rollout distribution in one case, the world model in the other), and the reward signal is sparse and binary.

What makes this transfer significant rather than trivial is that it reveals a deeper structural similarity between LLM RL and model-based robotics RL that has not been widely recognized. Both domains face the challenge of optimizing a policy against a learned reward signal in a simulated environment, with batch data that may contain large fractions of uninformative samples. Techniques developed for one domain (dynamic sampling, asymmetric clipping, KL-free GRPO) may transfer productively to the other. WMPO demonstrates this empirically: the GRPO configuration borrowed from LLM training (group size 8, batch size 64, asymmetric clipping, no KL penalty) works effectively for robotic manipulation without substantial modification.

Comparison to prior assumptions. Prior model-based RL for robotics has typically used algorithms designed specifically for continuous control: PPO with learned value functions, SAC with replay buffers, or Dreamer's actor-critic with imagined latent rollouts. These algorithms were developed for dense reward settings (e.g., forward velocity for locomotion, distance-to-goal for reaching) where advantage estimation is straightforward and uninformative batches are less of a concern because partial progress provides a continuous signal. WMPO's setting — sparse binary rewards from a learned classifier — is fundamentally different, and the paper's insight is that algorithms from the LLM RL literature are better suited to this reward structure.

Significance beyond performance. This innovation is incremental in its technical content (dynamic sampling is a simple filter) but significant in its methodological implication: as robotics RL increasingly adopts learned reward models (because hand-crafted rewards don't scale across diverse tasks), the reward structure shifts from dense and continuous to sparse and binary. This brings robotics RL closer to the LLM RL regime, where binary outcome rewards (correct/incorrect, helpful/unhelpful) are the norm. WMPO demonstrates that importing training-stability techniques from that regime is not just feasible but beneficial, opening the door for further cross-pollination between these previously separate RL communities.

Anchoring evidence. The paper does not provide an ablation of dynamic sampling (i.e., WMPO with vs. without the filtering). This is a limitation — we cannot quantify how much of the performance gain is attributable to this specific component versus the broader GRPO + world model architecture. However, the paper's strong results compared to GRPO baselines that do not have the benefit of world model amplification (Table 1: WMPO 47.1% at P=128 vs. GRPO 33.2%) suggest that the combination of world model, dynamic sampling, and on-policy GRPO — not any single component — drives the improvement. The dynamic sampling contribution should be understood as part of the overall training stability recipe rather than a standalone performance booster.

5. Experimental Analysis

Evaluation Methodology

Dataset. All simulation experiments use the Mimicgen benchmark [23], specifically four fine-grained manipulation tasks: Coffee_D0, StackThree_D0, ThreePieceAssembly_D0, and Square_D0. For evaluation, 128 different initial states are tested per task and average success rate is reported. Real-world experiments use a custom "Insert the square into the stick" task with a 5mm clearance between the square block and the stick, evaluated over 30 trials.
Base model(s). The base policy is OpenVLA-OFT [24], a state-of-the-art open-source VLA model fine-tuned via imitation learning on 300 expert demonstrations per simulation task (200 for the real-world task). The paper explicitly simplifies OpenVLA-OFT by removing proprioceptive state and wrist camera inputs, relying solely on third-person visual observations and language instructions. This tests whether visual information alone suffices for both the policy and the world model to represent task-relevant state.
Metrics. The primary metric is task success rate (%), defined as the fraction of evaluation trials where the robot successfully completes the specified task. Success is determined by environment-specific criteria (e.g., the square block is fully inserted onto the stick, all three blocks are stacked stably). For the reward model, F1 score is reported to measure classification accuracy on success/failure prediction. For behavioral analysis, relative average trajectory length is reported, normalized so the base policy equals 100% — this quantifies execution efficiency.
Baselines. Three baselines are compared throughout:
- Base policy: OpenVLA-OFT fine-tuned via imitation learning on expert demonstrations, with no additional RL optimization.
- Online GRPO [30]: An on-policy RL baseline where GRPO is applied directly using trajectories collected from the real environment (simulation), without a world model. Hyperparameters follow the same configuration as WMPO's GRPO (Table 4), with batch size tuned between 8 and 64 (Appendix B).
- Offline DPO [28]: An offline preference-based baseline where a preference dataset is constructed from trajectories collected by the base policy (pairing successful vs. failed trajectories), and the policy is optimized using the standard DPO objective without any online interaction or world model.
Generation budget / compute accounting. The rollout budget P is the primary compute-matching mechanism: all methods (WMPO, GRPO, DPO) are allocated the same number of real-world trajectories (P=128 for data-efficiency experiments, P=1280 for scaling experiments) from which to learn. For WMPO, these trajectories are used to fine-tune the world model (policy behavior alignment) and train the reward model, after which an arbitrary number of imagined trajectories can be generated for GRPO updates without additional real-world cost. For online GRPO, the same P trajectories are generated through real environment interaction and used directly for policy updates. For DPO, the P trajectories are collected once and reused repeatedly as offline preference data. This ensures that all methods operate under identical real-world interaction budgets, isolating the effect of the world model in amplifying limited real data.
Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported for the main results. The reward model's F1 score threshold is selected via validation experiments (Section 3.3). For evaluation, a fixed set of 128 initial states per task is used across all methods, providing a deterministic comparison. For the real-world experiment, 30 trials are conducted per method. The lifelong learning experiment (Section 4.5) uses iterative data collection with fixed intervals (P=128 per iteration), evaluating at each stage.

Main Quantitative Results

Comparison Against Online and Offline Baselines (Table 1)

The central quantitative result appears in Table 1, which compares WMPO against GRPO and DPO across all four simulation tasks at two rollout budgets (P=128 and P=1280). The headline numbers are:

At P=128 (small budget, data-efficiency regime):

Base policy: 33.6% mean success rate across all four tasks (individual task ranges: 19.5% on ThreePieceAssembly to 46.9% on StackThree, revealing substantial task difficulty variation).
Online GRPO: 33.2% — essentially identical to the base policy, demonstrating that 128 real trajectories are insufficient for on-policy RL to produce meaningful improvement without a world model.
Offline DPO: 37.3% — a modest 3.7 percentage-point improvement over the base policy, showing that offline preference learning can extract some signal from static data but plateaus quickly.
WMPO: 47.1% — a 13.5 percentage-point improvement over the base policy and 9.8 points over the strongest baseline (DPO). The per-task improvements are: Coffee (43.8% → 61.7%), StackThree (46.9% → 56.3%), ThreePieceAssembly (19.5% → 37.5%), and Square (24.2% → 32.8%).

The critical comparison is WMPO vs. GRPO: both use the same GRPO algorithm and the same 128 real trajectories, but WMPO amplifies those trajectories through the world model to generate many imagined rollouts per real trajectory, while GRPO must learn directly from the 128 real rollouts. The 13.9 percentage-point gap (47.1% vs. 33.2%) quantifies the value of the world model amplification.

At P=1280 (larger budget, scaling regime):

Base policy: unchanged at 33.6% (no additional data).
Online GRPO: 37.1% — improves modestly with 10× more real data but still lags far behind WMPO at P=128 (47.1%), confirming that real-world interaction is fundamentally sample-inefficient even with 1280 trajectories.
Offline DPO: 42.4% — continues to improve with more offline data, but the gains are diminishing (3.7 point improvement from P=128 to P=1280 vs. 3.7 points from base to P=128 initially).
WMPO: 57.6% — a 24.0 percentage-point improvement over the base policy, demonstrating robust scaling with additional policy behavior data. Per-task: Coffee reaches 75.0%, StackThree 64.1%, ThreePieceAssembly 46.1%, Square 45.3%.

The gap between WMPO and the strongest baseline widens from 9.8 points at P=128 to 15.2 points at P=1280, indicating that WMPO leverages additional trajectories more effectively than DPO's static data reuse. This is consistent with the on-policy advantage: as the budget grows, WMPO can perform more world model alignment updates and more GRPO iterations, continuously generating fresh on-policy imagined trajectories, while DPO is constrained to the same offline dataset regardless of budget.

A notable pattern in the per-task results: ThreePieceAssembly and Square are substantially harder tasks (base success rates 19.5% and 24.2%) than Coffee and StackThree (43.8% and 46.9%). WMPO roughly doubles success rates on each hard task at P=1280 (19.5% → 46.1%, 24.2% → 45.3%), suggesting the approach is particularly valuable for tasks where the base policy is unreliable and has more room to learn from failures.

Reward model accuracy: The paper reports that the reward model achieves "an F1 score above 0.95 across all tasks" (Section 4.2), indicating near-perfect classification of success vs. failure in imagined trajectories. This high accuracy is essential — if the reward model were noisy, the GRPO updates would optimize for a corrupted signal, and the policy would not reliably improve.

Emergent Behavior Analysis (Figures 3 and 5, Section 4.3)

The paper provides both qualitative and quantitative evidence for emergent behaviors unique to WMPO-trained policies.

Qualitative: Self-correction in the Square task. Figure 3 shows a frame-by-frame comparison between the base policy and WMPO on the Square task when a collision occurs (the square block contacts the stick at the wrong angle during attempted insertion). The base policy, "trained only on expert demonstrations, has never observed collisions during training; it continues to push the square against the stick until the maximum time horizon is reached, resulting in failure." This is the classic covariate-shift failure mode: when the policy encounters a state absent from the training distribution, it defaults to repeating its current action, which compounds the error.

The WMPO-trained policy, by contrast, "autonomously learns to lift the square, realign it, and then insert it correctly, ultimately succeeding in the task." The specific recovery sequence involves: (1) recognizing the collision state (the square is jammed), (2) lifting the square upward to disengage from the stick, (3) adjusting the lateral position to center the square above the stick, and (4) descending to complete the insertion. This multi-step correction was never demonstrated in the expert data — it emerged purely from RL within the world model.

Quantitative: Trajectory length reduction. Figure 5 shows the relative average trajectory length of successful trials for each method, normalized so the base policy equals 100%. WMPO produces trajectories approximately 4–6% shorter than the base policy. DPO and GRPO also produce shorter trajectories than the base policy but remain longer than WMPO. The paper attributes this to WMPO "discouraging stuck behaviors, which often result in failures due to timeouts" — the binary reward signal penalizes trajectories that reach the time limit without succeeding, incentivizing the policy to complete tasks efficiently. The fact that WMPO trajectories are shorter than both GRPO and DPO trajectories suggests this efficiency gain is specifically a consequence of the world-model-based training, not a general property of RL fine-tuning.

The combination of qualitative self-correction and quantitative efficiency improvement is significant because it demonstrates that WMPO is not merely making the base policy more reliable (fewer failures) but is actually changing how the policy executes tasks (different behavior when encountering obstacles). This is a stronger claim than improved success rate alone — it shows genuine behavioral innovation.

Generalization to Novel Disruptions (Table 2, Figure 4)

Section 4.4 evaluates whether policies trained with WMPO generalize to visual conditions not seen during training. Three disruption scenarios are tested, each on a different task:

Position Disruption (Square task): The stick's position is randomized within a rectangular region, rather than being fixed as in training. This tests spatial generalization — can the policy handle the target object at novel locations?
Background Disruption (StackThree task): The tabletop background is changed to a gray texture instead of the training background. This tests robustness to visual appearance changes that don't affect the physics.
Texture Disruption (ThreePieceAssembly task): The red base is replaced with a dark wooden base, changing the visual texture while preserving the geometry. This tests whether the policy relies on specific visual features vs. generalizable manipulation skills.

Table 2 reports success rates for each method under each disruption, with the following key results:

Base policy: 14.1% (Pos.), 46.1% (Bg.), 10.9% (Tex.) — mean 23.7%. Position and texture disruptions cause substantial degradation (from 24.2% in-distribution to 14.1% and 10.9%, respectively), revealing that the imitation-learned policy overfits to specific visual and spatial configurations from training.
GRPO: 15.6% (Pos.), 47.7% (Bg.), 10.9% (Tex.) — mean 24.7%. Essentially identical to the base policy across all disruptions, showing that online GRPO with limited real trajectories provides no generalization benefit.
DPO: 16.4% (Pos.), 34.4% (Bg.), 7.8% (Tex.) — mean 19.5%. Notably worse than the base policy under background and texture disruptions. The paper interprets this as DPO learning "reliance on spurious visual cues rather than transferable manipulation skills" — the preference optimization may amplify sensitivity to visual features that correlate with success in the training distribution but don't generalize.
WMPO: 22.3% (Pos.), 50.0% (Bg.), 16.4% (Tex.) — mean 29.6%. Consistently best across all disruptions, with the largest margins under position disruption (22.3% vs. 16.4% for DPO) and texture disruption (16.4% vs. 10.9% for GRPO/base).

The paper attributes WMPO's generalization advantage to training "entirely in the world model," which "captures more generalizable strategies." The mechanism is plausible: the world model, pretrained on OXE with diverse visual conditions, generates imagined trajectories that inherently include visual variation (different lighting, backgrounds, textures from the OXE data). The policy, by training on this diverse imagined data, learns manipulation strategies that are robust to visual appearance changes. The position disruption benefit likely comes from the world model's autoregressive generation producing trajectories with slightly different spatial configurations than the exact training demonstrations, effectively providing data augmentation.

A critical observation: DPO's degradation under background and texture changes is a negative result that highlights a risk of offline preference learning. DPO optimizes the policy to prefer trajectories labeled as successful, but without a world model to generate diverse visual conditions, it may learn to associate success with specific visual patterns (e.g., "the background is this particular texture") rather than with the actual task completion state. When those visual patterns change at test time, the learned preference degrades. WMPO avoids this failure mode because the world model's generated trajectories naturally decorrelate visual appearance from task success.

Lifelong Learning (Figure 6, Section 4.5)

The lifelong learning experiment on the StackThree_D0 task tests whether WMPO can sustain improvement through iterative data collection and policy optimization, simulating a deployment scenario where the robot continuously improves from its own experience.

The protocol: starting from the base policy (46.9% success rate), WMPO collects P=128 real trajectories, fine-tunes the world model, optimizes the policy via GRPO, and then uses the updated policy to collect the next batch of 128 trajectories. This cycle repeats for three iterations. DPO is run with the same iterative data collection, using each new batch as additional offline preference data. As a reference, the base policy is also trained with increasing numbers of expert demonstrations (300 → 428 → 556) to compare WMPO's self-improvement against simply collecting more expert data.

Results from Figure 6 (exact values not numerically specified in text but readable from the plot):

Base policy with more expert demos: modest improvement from 300 (46.9%) to ~50% at 428 demos to ~52% at 556 demos. The slope is shallow — each additional expert demo yields diminishing returns.
DPO with iterative data: performance fluctuates unstably, starting at 46.9%, rising to approximately 50% after the first iteration, then remaining roughly flat or oscillating with no clear upward trend. The paper states DPO "fails to improve iteratively due to unstable training."
WMPO with iterative data: steady improvement from 46.9% (iteration 0) to approximately 56% (iteration 1, after 128 self-collected trajectories) to approximately 65% (iteration 2, after 256 self-collected trajectories). Each iteration adds ~8–9 percentage points of improvement.

Three findings stand out. First, WMPO with 128 self-collected trajectories (iteration 1: ~56%) outperforms the base policy with 128 additional expert demonstrations (428 total: ~50%). Self-collected data — which includes failures the expert never demonstrated — is more valuable for improvement than expert data of the same quantity. This directly validates the paper's core motivation: learning from failures provides information that imitation learning from successes cannot.

Second, WMPO sustains improvement over multiple iterations without plateauing, demonstrating that the world model alignment step (fine-tuning on the updated policy's new trajectories) successfully tracks the evolving policy distribution. If the world model failed to adapt, the imagined trajectories would become increasingly off-policy and the GRPO updates would lose effectiveness. The continued improvement through iteration 2 provides indirect evidence that policy behavior alignment is working as designed.

Third, DPO cannot achieve iterative improvement despite receiving the same data. This confirms the offline limitation: DPO optimizes against a fixed reference policy (the initial base policy), and as the optimized policy diverges, the preference data becomes increasingly off-policy, introducing bias that destabilizes training. WMPO's on-policy GRPO, by generating imagined trajectories from the current policy, avoids this off-policy gap entirely.

Real-World Validation (Section 4.6)

The real-world experiment on the "Insert the square into the stick" task — a fine-grained manipulation with 5mm clearance between the square block and the stick — provides the strongest test of whether WMPO's simulation-trained policy transfers to physical reality.

The setup: 200 expert demonstrations are used to fine-tune OpenVLA-OFT as the base policy (achieving 53% success rate over 30 trials). The base policy then collects 128 additional real trajectories, which are used to fine-tune the world model and optimize the policy within WMPO. DPO is trained on the same 128 trajectories as an offline baseline.

Results: base policy achieves 53%, DPO achieves 60%, and WMPO achieves 70% success rate over 30 evaluation trials. The 17 percentage-point improvement from base to WMPO (53% → 70%) represents a 32% relative reduction in failure rate — from failing nearly half the time to failing only 30% of the time.

This real-world result is significant for several reasons. First, it validates that the world model, trained primarily in simulation (Mimicgen) with only 128 real-world trajectories for alignment, can support effective policy optimization that transfers to physical execution. This is the central practical claim of the paper: you don't need millions of real-robot interactions; you need a pretrained world model and a modest number of policy-specific trajectories.

Second, the real-world experiment uses a task with extremely tight tolerances (5mm clearance), where visual fidelity matters critically — a small misalignment in the world model's imagined trajectories could make an insertion look successful when it would actually fail. The fact that WMPO improves real-world performance suggests the world model is capturing fine-grained visual dynamics accurately enough for the policy to learn alignment-sensitive behaviors in imagination and transfer them to reality.

Third, DPO's more modest improvement (53% → 60%) compared to WMPO (53% → 70%) mirrors the simulation results: offline preference learning provides some benefit but plateaus, while world-model-based on-policy RL provides substantially larger gains from the same real-world data budget.

The paper also provides qualitative evidence in Figures 7, 8, and 9, showing side-by-side comparisons of real-world trajectories and their corresponding imagined trajectories from the world model. Figure 7 shows a successful trajectory where the world model accurately predicts the insertion sequence from the same initial state. Figure 8 shows the world model successfully predicting a failure case — it has "learned that when the square and the stick are misaligned, the square cannot be inserted into the stick." Figure 9 shows a rare failure case of the world model itself: although predictions remain accurate until the final frame, "subtle perturbations prevent the model from faithfully capturing the moment when the square gets stuck in the stick." The paper notes that "such failures are relatively rare on the validation set," indicating the world model is generally reliable but not perfect — an honest assessment of the approach's limitations.

Ablation Studies and Robustness Checks

Reward model accuracy (Section 4.2, no standalone figure): The reward model achieves an F1 score above 0.95 across all four simulation tasks. This is not presented as an ablation but as a validation that the learned reward signal is reliable. The high accuracy is essential for two reasons: it ensures the policy is optimizing for genuine task success rather than exploiting reward model blind spots (reward hacking), and it validates the training data construction (terminal-clip positives, mid-trajectory and failure-trajectory negatives with class-balanced batches). The paper does not report how sensitive policy optimization is to reward model accuracy — e.g., whether an F1 of 0.90 vs. 0.95 produces noticeably different policy improvements. This is a missing ablation that would quantify the margin for error in reward model training.

World model architecture choices (Section 3.2, implicit ablations): The paper describes several architectural decisions as motivated by specific failure modes but does not present controlled experiments quantifying their contribution:

2D VAE vs. 3D VAE: The paper states that the 2D VAE from SDXL "better preserves fine-grained motion details and avoids temporal distortions caused by excessive compression" compared to OpenSora's default 3D VAE. This is motivated by the need to capture fine robot-object interactions (contacts, slips, insertions) that 3D VAE temporal compression might smooth out. No ablation comparing 2D vs. 3D VAE on task success or trajectory fidelity is presented. Such an ablation would strengthen the claim that temporal fidelity is the binding constraint for robotic world models.
Noisy-frame conditioning: The paper introduces this technique to mitigate error accumulation in long-horizon autoregressive generation, stating that noise at 50/1000 steps "improves robustness to imperfect conditioning" and enables "stable long-horizon generation, producing trajectories of hundreds of frames without noticeable quality loss." No ablation is provided comparing generation quality with vs. without noisy-frame conditioning, nor is there a sensitivity analysis of the noise level (50/1000 steps). Given that error accumulation is a well-known failure mode of autoregressive video generation, quantifying the benefit of this technique would be valuable.
Frame-level action control: The extended AdaLN mechanism is described as improving action-frame alignment over global conditioning. No ablation compares frame-level vs. global action conditioning. The paper cites IRASim [37] for inspiration, but an internal ablation would quantify how much of the world model's accuracy depends on this specific design choice.

These missing architectural ablations are understandable given the paper's scope (an systems contribution rather than an architectural contribution), but they leave open the question of which design choices are essential for the approach to work and which are incidental.

Dynamic sampling (Section 3.4, no standalone figure): Dynamic sampling filters out trajectory groups where all G=8 trajectories have the same outcome (all success or all failure), ensuring that every group contributing to the gradient update contains both successes and failures. The paper does not ablate this component — i.e., WMPO with vs. without dynamic sampling, or a sensitivity analysis of the group size G. Given that dynamic sampling is imported from DAPO [44] and its contribution to LLM RL training stability is documented in that prior work, the lack of ablation is partly justified by prior evidence. However, the transfer to robotics RL with a world model is not guaranteed — the distribution of group outcomes (how often groups are all-success or all-failure) likely differs between LLM reasoning tasks and robotic manipulation tasks. An ablation or at least a report of the filtering rate (what fraction of sampled groups are discarded) would clarify dynamic sampling's importance in this specific setting.

GRPO configuration choices (Table 4, no standalone figure): The GRPO hyperparameters include asymmetric clipping (ε_low=0.20, ε_high=0.28), no KL divergence regularization, a temperature of 1.6, a group size of 8, and a batch size of 64. These follow DAPO [44] and the broader LLM RL literature. No sensitivity analysis or ablation of any hyperparameter is presented. In particular:

The choice to remove KL divergence regularization is motivated by reducing memory consumption and encouraging exploration, but no comparison to KL-regularized GRPO is provided. The risk of removing KL regularization is catastrophic forgetting of the base policy's capabilities — the fact that the policy continues to improve suggests this risk did not materialize, but quantifying the stability-accuracy tradeoff would strengthen confidence in the choice.
The temperature of 1.6 is relatively high, encouraging substantial exploration. No sweep over temperatures (e.g., 1.0, 1.3, 1.6, 2.0) is reported. The optimal temperature likely depends on task difficulty and the base policy's initial stochasticity, so the generalizability of 1.6 to other tasks is unknown.
Asymmetric clipping is motivated by allowing larger probability increases than decreases, but the specific values (0.20 vs. 0.28) are taken from DAPO without task-specific tuning. An ablation comparing symmetric vs. asymmetric clipping, or different asymmetry ratios, would quantify the benefit.

Policy behavior alignment budget (Table 1, implicit ablation): The comparison between P=128 and P=1280 serves as an implicit ablation of the alignment budget — how much data is needed for effective policy behavior alignment? The improvement from 47.1% to 57.6% (P=128 to P=1280) shows that more alignment data helps, but with diminishing returns: the first 128 trajectories produce a 13.5 percentage-point gain over the base policy; the next 1152 trajectories produce an additional 10.5 points. The paper does not explore intermediate budgets (e.g., P=256, P=512) to characterize the scaling curve more finely. Understanding the shape of this curve — where does it start to plateau? — would guide practitioners in choosing the alignment budget for new tasks.

Base policy quality (no explicit ablation): All experiments use a base policy fine-tuned on 300 expert demonstrations per simulation task. The paper does not vary the quality of the base policy to test whether WMPO's improvement depends on the initial policy's competence. This is a potentially important factor: if the base policy is very poor (e.g., near-zero success rate), the policy behavior data collected for world model alignment would consist almost entirely of failures. The world model might struggle to simulate failures accurately if it never sees successes during fine-tuning, and the reward model would have no positive examples in the collected data. Conversely, if the base policy is very good (e.g., 80%+ success rate), the room for improvement is limited and WMPO's benefits might be smaller. Understanding these boundary conditions would clarify when WMPO is most valuable.

Real-world world model fidelity (Figures 7-9, Appendix C): The real-world experiment includes qualitative evidence of world model accuracy but no quantitative metric (e.g., FVD, PSNR, SSIM between real and imagined trajectories). The paper shows examples of successful predictions (Figures 7-8) and acknowledges rare failures (Figure 9: the model fails to capture the square getting stuck due to "subtle perturbations"). A quantitative evaluation of world model fidelity on real-world data — even on a small validation set — would provide a more objective assessment of how reliable the imagined environment is for policy optimization. The 70% real-world success rate provides indirect validation (the policy improved, so the world model must have been good enough), but doesn't quantify the margin for error.

Critical Assessment

Claim 1: WMPO substantially improves sample efficiency compared to model-free baselines.

What was tested: The comparison in Table 1 shows that WMPO with P=128 real trajectories achieves 47.1% mean success rate vs. 33.2% for online GRPO and 37.3% for offline DPO, using the same real-world data budget. At P=1280, WMPO achieves 57.6% vs. 37.1% (GRPO) and 42.4% (DPO).

What this actually demonstrates: The experiments convincingly show that WMPO extracts more value per real-world trajectory than the baselines. However, "sample efficiency" is a relative claim — the absolute sample complexity is still substantial. 128 trajectories for four tasks (effectively 32 per task if equally distributed) is a non-trivial amount of real-world data, especially for real-robot deployment where each trajectory takes minutes. The paper does not explore whether even smaller budgets (e.g., P=32, P=64) would suffice, nor does it characterize the minimum number of trajectories needed for the world model alignment to be effective. The 128-trajectory budget was chosen, not discovered as a minimum. Additionally, the world model's pretraining on OXE (millions of trajectories) is not counted in the sample efficiency comparison — WMPO benefits from this pretraining in a way the baselines do not, making the comparison somewhat asymmetric. The sample efficiency claim is best understood as: "given a pretrained world model, WMPO is more sample-efficient for downstream task adaptation than training model-free RL from scratch on those tasks."

Missing experiments: An ablation varying the alignment budget (P=32, 64, 128, 256, 512) with a fixed evaluation protocol would characterize the sample efficiency curve and identify the minimum viable budget. A comparison against a baseline that also benefits from the world model pretraining (e.g., using the world model for data augmentation in DPO or behavioral cloning) would isolate the contribution of on-policy GRPO vs. the world model itself.

Claim 2: WMPO achieves stronger overall performance.

What was tested: WMPO outperforms all baselines across all four simulation tasks at both budget levels (Table 1), and outperforms the base policy and DPO in the real-world experiment (70% vs. 53% and 60%). The generalization experiments (Table 2) show WMPO outperforming baselines under position, background, and texture disruptions. The lifelong learning experiment (Figure 6) shows WMPO sustaining improvement over multiple iterations while DPO plateaus.

What this actually demonstrates: The performance improvement is consistent and substantial — this claim is well-supported by the data. The per-task breakdown reveals an important nuance: WMPO's improvement is largest on the hardest tasks. On Coffee (the easiest task, base success 43.8%), WMPO at P=1280 reaches 75.0% — a 31.2 point gain. On ThreePieceAssembly (hardest, base 19.5%), WMPO reaches 46.1% — a 26.6 point gain. In absolute terms the improvement is similar, but in relative terms the hardest tasks remain below 50% success. This is not a weakness of the paper — it's an honest characterization — but it means "stronger overall performance" still leaves substantial room for improvement on the most challenging tasks.

Missing experiments: The paper does not compare WMPO against a hypothetical upper bound: what success rate would the policy achieve if trained with unlimited real-world GRPO (i.e., what is the asymptotic performance of on-policy RL on these tasks)? Without this bound, we cannot assess how much of the performance gap between WMPO and optimal performance is due to world model inaccuracy vs. fundamental task difficulty. Additionally, the paper does not compare against alternative model-based approaches (e.g., DreamerV3 adapted for VLA, or diffusion-world-model approaches like World4RL [32]) that also use learned dynamics models — the baselines are exclusively model-free, making it unclear whether WMPO's advantage comes from using a world model or from using this specific world model design.

Claim 3: WMPO exhibits emergent self-correction behaviors.

What was tested: Figure 3 shows a qualitative comparison of the Square task, where the base policy continues pushing after a collision while WMPO lifts, realigns, and reinserts. Figure 5 shows WMPO trajectories are 4–6% shorter than the base policy, indicating more efficient execution without "stuck" behavior.

What this actually demonstrates: The qualitative evidence for self-correction is compelling but limited to a single example from a single task. The paper does not quantify how frequently self-correction occurs — is this a consistent behavior across many trials, or a cherry-picked example? The trajectory length reduction (Figure 5) provides indirect quantitative corroboration: if the policy is self-correcting more often, its successful trajectories should be shorter because it recovers from near-failures rather than persisting until timeout. However, shorter trajectories could also result from the policy simply being more decisive (less hesitation, faster movements) without involving explicit recovery from collisions. The paper's claim that WMPO "discourag[es] stuck behaviors" is plausible but the causal link between self-correction observations and trajectory length reduction is not directly established.

Missing experiments: A systematic analysis of failure modes and recovery rates would strengthen this claim substantially. For example: what fraction of trials that encounter collisions result in successful recovery for WMPO vs. the base policy? Does WMPO exhibit recovery behaviors on tasks other than Square? Are there types of failures from which the policy cannot recover, and if so, what characterizes them? The paper could also have performed targeted interventions — e.g., deliberately inducing collisions at specific trajectory points and measuring the recovery rate — to isolate self-correction from other sources of performance improvement.

Claim 4: WMPO demonstrates robust generalization and lifelong learning capabilities.

What was tested: Three disruption scenarios (Table 2) test generalization to spatial, background, and texture variations. The lifelong learning experiment (Figure 6) tests sustained improvement over three iterations of alternating data collection and policy optimization.

What this actually demonstrates: The generalization results are convincing within their scope: WMPO outperforms baselines under visual disruptions on the same tasks. However, these are relatively mild distribution shifts (same task, same objects, same robot, different visual appearance or object position). The paper does not test broader generalization: new tasks not seen during world model fine-tuning, new objects with different geometries, or different robot embodiments. The generalization claim should be understood as "robust to visual appearance variations of the training task" rather than "generalizes to novel tasks." The lifelong learning result is promising but limited to three iterations on one task (StackThree) with a small real-world data budget per iteration (P=128). The paper does not report whether improvement continues beyond three iterations or plateaus, nor does it test lifelong learning across multiple tasks (e.g., alternating between Square and Coffee).

Missing experiments: Cross-task generalization (train WMPO on task A, test on related task B) would test whether the learned improvement strategies transfer. Generalization to substantially different initial conditions (e.g., objects starting in configurations not seen in the alignment data) would test whether the world model can extrapolate. Longer lifelong learning runs (5–10 iterations, multiple tasks) would characterize whether WMPO can sustain improvement without catastrophic forgetting or world model degradation.

Overall Assessment

The experiments provide strong support for WMPO's central thesis: that on-policy RL within a pixel-space world model, with policy behavior alignment, can substantially improve VLA policy performance while requiring far fewer real-world interactions than direct model-free RL. The consistency of results across four simulation tasks, two budget levels, multiple baselines, and a real-world transfer experiment builds a convincing case.

The primary limitation of the experimental evaluation is its scope: single model family (OpenVLA-OFT), single simulation benchmark (Mimicgen), four simulation tasks all involving rigid-body manipulation, and one real-world task. The paper's claims are well-supported within this scope, but the generalizability to other VLA architectures (RT-2, π0.5), other task types (deformable objects, mobile manipulation, multi-step assembly with tool use), and other robot embodiments is unverified. The missing architectural ablations (2D vs. 3D VAE, noisy-frame conditioning, frame-level action control, dynamic sampling) mean we cannot assess which components are essential vs. incidental to WMPO's performance — a practitioner attempting to replicate the approach would need to guess at which design choices matter most.

The world model's pretraining on OXE is both a strength and a confound. It enables WMPO's sample efficiency (the world model already understands general robot dynamics), but it also means the approach inherits whatever limitations exist in OXE's coverage — tasks, embodiments, or visual conditions poorly represented in OXE may not benefit from WMPO. The paper does not investigate how world model pretraining quality affects downstream policy optimization, which would be valuable for understanding when WMPO is applicable.

Finally, the reported improvements, while substantial in relative terms, leave absolute success rates below 60% on three of four simulation tasks and 70% on the real-world task. This is not a weakness — it's honest science — but it means WMPO is a partial solution to VLA robustness, not a complete one. The hardest tasks (ThreePieceAssembly, Square) remain below 50% success even after optimization, indicating that significant challenges remain in achieving reliable autonomous manipulation.

6. Limitations and Trade-offs

Generalization Boundaries: Single Model Family, Single Benchmark, One Real-World Task

The assumption or constraint. All experiments in this paper use exactly one VLA architecture (OpenVLA-OFT fine-tuned via imitation learning), one simulation benchmark (Mimicgen with four manipulation tasks), and one real-world task (Square insertion). The paper does not test WMPO with other VLA architectures (RT-2, π0.5, Octo), other simulation platforms (RLBench, ManiSkill, LIBERO), or other task types (deformable object manipulation, mobile manipulation, multi-step assembly requiring tool use). The authors do not explicitly acknowledge this scope limitation as a boundary condition; instead, they state in Section 4.1 that OpenVLA-OFT was chosen as the base policy and Mimicgen as the simulation environment, without claiming broader generalizability.

The consequence. A practitioner considering WMPO for a different VLA architecture faces substantial uncertainty. The pixel-space world model design is motivated by the claim that VLA pretraining confers rich visual representations that a latent world model would discard — but this claim's validity depends on the specific VLA's visual encoder architecture, pretraining data, and feature quality. A VLA with a weaker visual encoder (e.g., one pretrained primarily on navigation data rather than manipulation data) might not benefit from pixel-space alignment to the same degree. Similarly, the world model's OXE pretraining covers many but not all manipulation scenarios; tasks involving dynamics poorly represented in OXE (e.g., highly dynamic tossing, in-hand dexterous manipulation with rolling contacts, tasks requiring precise force estimation from visual cues alone) may exceed the world model's capabilities.

For the generalization disruption experiments (Table 2, Section 4.4), WMPO's advantage appears under visual appearance shifts (background, texture) of the same task — but this says nothing about whether WMPO-trained policies generalize to new tasks with different physical dynamics. The paper never tests cross-task transfer (train WMPO on task A, evaluate on task B), which would be required to claim that WMPO produces generalizable manipulation skills rather than task-specific robustness.

What evidence exists in the paper. The evidence for generalization is entirely within the scope tested: four simulation tasks using one VLA, one real-world task using the same VLA. The generalization experiments (Table 2, Figure 4) are limited to visual appearance variations of the training tasks. No cross-task, cross-embodiment, or cross-VLA-architecture experiments are reported. The lifelong learning experiment (Figure 6) is on a single task (StackThree) for three iterations. This does not constitute evidence against broader generalizability, but it provides zero evidence for it — the scope of evaluation is too narrow to support generalization claims beyond the tested conditions.

Mitigation status. Not addressed. The authors do not propose cross-architecture or cross-benchmark validation as future work, nor do they discuss which aspects of the approach might be VLA-specific vs. general. Section 5 states WMPO is "a scalable and generalizable paradigm for advancing VLA RL" — but this is an aspirational claim, not one supported by evidence spanning multiple VLAs or benchmarks. A responsible interpretation is that WMPO has been demonstrated to work for one VLA on one benchmark plus one real task; whether it works for other VLAs on other tasks is an open question.

Difficulty Estimation Cost Is Unaccounted For: World Model Pretraining and Alignment Overhead

The assumption or constraint. WMPO's headline sample efficiency numbers (Table 1: 47.1% at P=128, 57.6% at P=1280) count only the real trajectories used for policy behavior alignment — they do not include the cost of world model pretraining on OXE (12M training steps, Table 3) or the cost of world model fine-tuning (3M steps per downstream task). The paper is transparent about this architecture (Appendix A reports training hyperparameters), but presents "sample efficiency" as a comparison against baselines using the same real-world trajectory budget, without noting that WMPO benefits from an enormous pretraining investment that the baselines do not share.

To put this in perspective: the world model is pretrained on the Open X-Embodiment dataset (millions of trajectories across dozens of embodiments and tasks) using 32 H100 GPUs for 12 million training steps. This pretraining cost, amortized across downstream tasks, may be acceptable in a research lab with sufficient compute — but it represents a multi-thousand GPU-hour investment that a practitioner adopting WMPO must either replicate (prohibitively expensive) or rely on publicly released pretrained weights (if available). The paper does not release pretrained world model weights (the project page exists but the paper does not specify model availability).

The consequence. If world model pretraining must be replicated from scratch, the total compute cost of WMPO — including pretraining, fine-tuning, and policy optimization — is orders of magnitude larger than the baselines it is compared against. A fair FLOPs-matched comparison would ask: given the total compute budget required to pretrain the world model + collect P trajectories + fine-tune + run GRPO, what success rate could be achieved by simply spending that same compute budget on more real-world GRPO rollouts? The paper provides no such comparison.

This is structurally analogous to the test-time compute paper's difficulty estimation cost issue (flagged in Section 6 of the reference example) — the overhead that enables the method is excluded from the efficiency calculation. In WMPO's case, the overhead is amortizable (pretrain once, fine-tune for many tasks), but for a practitioner with a single downstream task, the pretraining cost cannot be amortized and may dominate the total compute budget.

Additionally, the world model's fine-tuning cost (3M steps, 32 H100 GPUs) is incurred per downstream task. This is not included in the P=128/P=1280 trajectory count — the trajectories are used as training data, not as a measure of GPU-hours. A practitioner with limited compute might find that the 3M fine-tuning steps cost more wall-clock time and energy than simply collecting more real trajectories.

What evidence exists in the paper. The training hyperparameters in Table 3 and Appendix A transparently report the computational scale: 12M pretraining steps, 3M fine-tuning steps, 32 H100 GPUs, batch size 128. The paper does not hide these costs, but it does not factor them into efficiency comparisons or discuss the amortization assumption explicitly. The sample efficiency claim in Section 4.2 ("strong data efficiency") is presented as a property of the P=128 vs. P=1280 comparison, without contextualizing the pretraining investment.

Mitigation status. Partially addressed by transparency (the costs are reported) but not by analysis. The authors do not provide FLOPs-matched comparisons, do not report total GPU-hours, do not discuss whether pretrained world model weights will be released, and do not analyze the tradeoff between world model pretraining cost and downstream sample efficiency. A practitioner reading the paper must infer the amortization assumption from context and calculate the total cost themselves.

World Model Fidelity Is Assessed Qualitatively, Not Quantitatively — No Guarantee on Simulation Accuracy

The assumption or constraint. The entire WMPO framework depends on the world model faithfully simulating the visual consequences of the policy's actions. If the world model produces trajectories that diverge from what would happen in reality — objects slipping that shouldn't, contacts not being registered, insertion failures being missed — the policy will optimize for behaviors that succeed in imagination but fail in the real world. This is the model-based RL analogue of reward hacking, but more severe because it corrupts the state transitions, not just the reward signal.

The paper provides qualitative evidence of world model fidelity (Figures 7-9 in the real-world experiment) but no quantitative metrics of simulation accuracy: no FVD (Fréchet Video Distance), no PSNR, no SSIM, no action-conditioned prediction error, no success-rate correlation between imagined and real trajectories. The reward model is quantitatively evaluated (F1 > 0.95), but the world model — the more complex and failure-prone component — is not.

The consequence. Without quantitative fidelity metrics, a practitioner cannot assess whether their own world model (trained on different data, with different hyperparameters) is "good enough" to support policy optimization. The paper's implicit claim is that the world model must be trained with noisy-frame conditioning, frame-level action control, and policy behavior alignment — but it provides no diagnostic for when these techniques have succeeded vs. when the world model remains unreliable. Figure 9 shows a failure case where the world model misses the square getting stuck due to "subtle perturbations" — but without a quantitative metric, we cannot estimate how common such failures are (the paper says "relatively rare" without quantification) or whether they correlate with downstream policy optimization failures.

This also makes reproducibility difficult. If another group attempts to replicate WMPO and achieves worse policy improvement, they cannot distinguish between "our world model is worse" and "our GRPO configuration is suboptimal" because there is no quantitative world model evaluation to isolate the source of the gap.

What evidence exists in the paper. The only quantitative world model evaluation is the improvement it enables (the policy gets better, therefore the world model must be good enough) — which is circular as a fidelity metric. The real-world experiment (Section 4.6) provides the strongest implicit evidence: WMPO improves real-world success from 53% to 70%, which would be impossible if the world model were systematically misleading. But this is an outcome-level validation, not a component-level diagnostic. The qualitative comparisons in Figures 7-9 show individual examples but support no statistical claims.

Mitigation status. Not addressed. The paper provides no quantitative world model evaluation methodology, no fidelity metrics, and no sensitivity analysis showing how policy improvement degrades as world model accuracy decreases. The reward model's quantitative evaluation (F1 > 0.95) highlights the absence of equivalent evaluation for the world model. This is an important gap because world model fidelity is arguably the binding constraint on the approach — if the world model could be made perfect, policy optimization would be straightforward; if it is unreliable, no amount of GRPO tuning will help.

Hard Problems Remain Largely Unsolved — WMPO Cannot Compensate for Fundamentally Insufficient Base Policy Capability

The assumption or constraint. WMPO improves a VLA policy through RL within a learned world model. Both components — the world model and the RL algorithm — assume the policy has a non-trivial probability of producing successful trajectories under some action sequences. If the base policy's success rate is near zero (the policy fundamentally does not understand the task, produces random or systematically wrong actions), RL within the world model cannot create capability from nothing — there are no successful trajectories to reinforce, and the world model cannot invent new physics that would make incorrect actions succeed.

The paper does not explicitly state this bound, but it is visible in the data. The base policy's success rates span from 19.5% (ThreePieceAssembly) to 46.9% (StackThree) — all substantially above zero. WMPO's improvements are larger in absolute terms on tasks where the base policy is already moderately capable: Coffee (43.8% → 75.0%) gains 31.2 points, while ThreePieceAssembly (19.5% → 46.1%) gains 26.6 points. Neither task approaches 100%, and the hardest tasks remain below 50% even after optimization.

The consequence. For tasks where the base VLA's success rate is very low (e.g., <10%), WMPO is unlikely to help. The world model alignment data would consist almost entirely of failures, providing little signal for the reward model to learn what success looks like, and the GRPO updates would have few or no positive examples to reinforce. This is a fundamental capability bound: WMPO amplifies existing capability but does not create it from nothing. A practitioner with a VLA that rarely succeeds on a task cannot expect WMPO to make it reliable — they would need to first improve the base policy through more imitation learning or a better VLA architecture.

This is structurally identical to the capability bound identified in the test-time compute scaling paper (Section 6, reference example): on the hardest problems (difficulty bin 5, where the base model's pass@1 is near zero), "no method makes meaningful progress" regardless of test-time compute budget. WMPO exhibits the same pattern: the Square task starts at 24.2% base success and reaches 45.3% at P=1280 — improved but still failing more than half the time. The approach does not eliminate the need for better pretrained VLAs; it extracts more value from VLAs that already have some competence.

What evidence exists in the paper. The per-task results in Table 1 clearly show that post-optimization success rates correlate with pre-optimization success rates: tasks that start higher end higher. At P=1280, the rank order of tasks by success rate is unchanged from the base policy (Coffee > StackThree > Square > ThreePieceAssembly before and after optimization). This suggests WMPO provides a roughly multiplicative improvement rather than closing the gap between easy and hard tasks. The paper does not test WMPO on any task with base success <10%, so the lower bound of applicability is unknown — the approach might degrade gracefully or fail catastrophically below some threshold.

Mitigation status. Not addressed. The paper does not discuss the relationship between base policy quality and WMPO improvement, does not test a range of base policy qualities (e.g., by varying the number of expert demonstrations used for initial fine-tuning), and does not characterize the lower bound at which WMPO ceases to help. The lifelong learning experiment (Figure 6) shows WMPO sustaining improvement, but starting from a base policy at 46.9% — we do not know whether a policy starting at 10% would also improve, or at what rate. The authors present WMPO as a general post-training improvement method without specifying the capability floor required for it to work.

The Revision Model Is Not Integrated with PRM Search — WMPO Leaves Complementary Mechanisms Unexplored

The assumption or constraint. WMPO studies policy optimization via on-policy GRPO within a pixel-space world model. However, the paper explicitly treats the world model as a drop-in replacement for the physical environment during RL — it does not explore using the world model for planning (e.g., model-predictive control, tree search over imagined action sequences) or for data augmentation beyond the GRPO rollout mechanism. More critically, it does not combine the world model's ability to simulate future states with the VLA's ability to generate action proposals — a natural integration that could exploit the complementary strengths of model-based planning (exploring future possibilities) and policy-based action generation (efficiently proposing good actions).

This is analogous to the test-time compute paper's finding that PRM search and iterative revisions have complementary, difficulty-dependent strengths but were never combined — and that combining them is an obvious next step (Section 8 of that paper). WMPO's GRPO-based optimization leverages the world model for data amplification (generating many imagined rollouts per real trajectory), but not for search (exploring alternative action sequences from a given state to find better trajectories than the policy would naturally produce).

The consequence. WMPO's policy improvement is bounded by the quality of trajectories the current policy can generate, even with high-temperature exploration. If the optimal behavior requires a sequence of actions that is extremely unlikely under the current policy (e.g., a precise multi-step recovery maneuver that the policy has never sampled), GRPO may never discover it because it only reinforces actions the policy actually produces. Model-based planning — using the world model to simulate the consequences of candidate action sequences and selecting the best — could find such behaviors even if the policy would not naturally sample them.

The paper's qualitative evidence of self-correction (Figure 3) shows the policy learning a recovery behavior that was absent from demonstrations but present in the policy's exploration distribution (lifting the square was a possible but unlikely action under the stochastic policy). A more complex recovery — e.g., backing away from the stick, rotating the square 90 degrees, approaching from a different angle — might require an action sequence so unlikely under the policy that GRPO never generates it, even with temperature 1.6. Model-based search could discover such behaviors by explicitly simulating candidate action sequences, but WMPO does not exploit this capability of the world model.

What evidence exists in the paper. The paper's architecture (Section 3.2, Figure 2) uses the world model exclusively for autoregressive rollout generation: policy predicts action chunk → world model generates next frames → repeat. There is no mechanism for the policy (or a planner) to query the world model with "what if" counterfactuals — trying multiple action candidates from the same state and selecting the best. The GRPO algorithm (Section 3.4) generates multiple trajectories per initial state (G=8), but each trajectory follows a single action sequence from that state; there is no branching or search within a trajectory. The paper does not ablate a planning-based variant, nor does it discuss why planning was not explored.

Mitigation status. Not addressed. The authors do not identify search-based planning as a missed opportunity or future direction. The lifelong learning experiment (Figure 6) and generalization results (Table 2) suggest WMPO's current approach works well for the tested tasks, but they provide no evidence about whether adding search would provide additional gains or whether the current GRPO-only approach is near-optimal.

Latency and Online Deployment Constraints: Sequential Autoregressive Generation Is Inherently Serial

The assumption or constraint. WMPO's imagined trajectory generation (Section 3.2) is fundamentally sequential: the policy predicts an action chunk, the world model generates the next K frames conditioned on those actions and previous frames, the policy observes those new frames and predicts the next action chunk, and so on. Each step depends on the output of the previous step — there is no parallelism across timesteps. For a trajectory of T action chunks (each generating K=8 frames), the generation process requires T sequential forward passes through both the policy and the world model.

The world model itself is a diffusion model — each frame generation step requires multiple denoising iterations (the paper inherits OpenSora's diffusion configuration; the number of denoising steps is not explicitly reported but is typically 50–250 for video diffusion models). Each iteration is a forward pass through a large transformer. Combined with autoregressive rollout over potentially hundreds of frames, the latency for generating a single imagined trajectory is substantial.

The consequence. During training, this latency is acceptable — imagined trajectories are generated offline, batched across many initial states and GPUs. But WMPO's lifelong learning scenario (Section 4.5) imagines iterative improvement during deployment: the robot collects trajectories, updates the world model, re-optimizes the policy, and deploys the improved policy. The policy optimization step requires generating many imagined trajectories (G=8 per initial state, batch size B=64, potentially multiple epochs), which may take hours on 32 H100 GPUs (the configuration reported in Appendix A). This is not real-time adaptation — if a robot encounters a novel failure mode during deployment, it cannot update its policy on-the-fly and immediately retry. The improvement cycle is offline and computationally intensive.

More fundamentally, the autoregressive generation bottleneck limits how much imagined data can be produced per unit of wall-clock time, which constrains the scale of GRPO training. If generating one imagined trajectory takes 10 seconds on a GPU, generating the millions of trajectories that would be needed for very large-scale RL (analogous to the scale at which LLM GRPO operates) would be impractical. The paper's GRPO operates with B=64 trajectories per update — sufficient for the tested tasks, but potentially insufficient for tasks requiring more exploration or more complex behaviors.

What evidence exists in the paper. The training configuration in Appendix A reports using 32 H100 GPUs for world model training and policy optimization, indicating the computational scale required. The paper does not report wall-clock time for trajectory generation, GRPO update frequency, or total training time. No latency analysis is provided. The lifelong learning experiment (Figure 6) shows improvement over three iterations with P=128 trajectories each, but does not report how long each iteration takes or whether the latency would be acceptable in a deployment setting.

Mitigation status. Not addressed. The paper presents WMPO as a training methodology and does not discuss deployment-time constraints. The latency of imagined trajectory generation is not analyzed, and no techniques for reducing it (e.g., consistency models for faster diffusion sampling, parallel exploration of multiple action sequences, reduced denoising steps) are proposed. A practitioner considering WMPO for a setting where policy updates must happen quickly (e.g., a robot adapting to a new environment within minutes) would need to independently assess whether the computational requirements are compatible with their latency constraints.

7. Implications and Future Directions

How This Work Changes the Landscape

WMPO introduces a methodological shift in how the field should think about training VLA policies: it reframes the problem from "collect more expert demonstrations" or "do RL on real robots" to "build a high-fidelity pixel-space world model once, then use it as a reusable training environment for many tasks and many rounds of policy improvement." This is not a paradigm shift in the sense of inventing world models or on-policy RL — both are well-established — but it is a diagnostic reframing that identifies what specifically was preventing model-based RL from working for VLA fine-tuning and provides a concrete, validated solution.

The central diagnostic is this: prior model-based RL for visual control (the Dreamer lineage) operated in compact latent spaces, which works when the policy learns its visual representations from scratch inside the world model. But for VLA foundation models — which invest enormous pretraining compute into learning general-purpose visual representations from web-scale data — the latent bottleneck severs the policy's connection to its pretrained knowledge. WMPO's core contribution is identifying this representation-alignment problem and solving it through a specific architectural choice (pixel-space video generation + the VLA's own pretrained encoder) rather than through algorithmic complexity. The 20+ percentage-point improvement over baselines (Table 1: 47.1% vs. 33.6% base at P=128, 57.6% at P=1280) quantifies how much this diagnosis matters in practice.

This reframing changes the research landscape in several concrete ways. First, it makes world model pretraining a first-class research investment for the robotics community, analogous to how large-scale language model pretraining became a shared infrastructure for NLP. The paper demonstrates that a world model pretrained on Open X-Embodiment (millions of trajectories across dozens of embodiments) can be fine-tuned with as few as 128 policy-specific trajectories to support effective RL — a compelling argument for investing in a shared, open-source world model foundation model that individual labs can adapt to their specific robots and tasks. This shifts the cost-benefit calculation: pretraining a world model from scratch is prohibitively expensive for most labs, but fine-tuning a publicly available pretrained world model is tractable. The paper does not release pretrained weights (a notable gap), but it makes the intellectual case for such an infrastructure investment.

Second, it reconciles conflicting intuitions about the role of simulation fidelity in robot learning. One camp has argued that simulation must be physically accurate (high-fidelity physics engines, system identification, domain randomization) to transfer to reality; another camp has argued that visual realism matters more than physical accuracy because modern vision-based policies are robust to visual variation if trained with sufficient diversity. WMPO provides a nuanced resolution: physical fidelity matters for the world model (it must accurately simulate the consequences of actions, including failures), but visual fidelity matters for the policy interface (the VLA needs pixel-space inputs that match its pretraining distribution). The world model's OXE pretraining provides broad physical priors; fine-tuning on policy-specific data captures the relevant failure modes; the 2D VAE preserves the temporal detail needed for manipulation while the pixel-space output keeps the VLA's visual encoder in-distribution. This decomposition — physical fidelity from data, visual alignment from architecture — is a more productive framing than the "simulation vs. reality" binary.

Third, it redirects research attention from RL algorithm design to world model data quality. The paper's GRPO configuration is straightforward (mostly borrowed from DAPO without task-specific tuning), yet it achieves strong results because the world model is well-aligned with the policy's behavior. This suggests that for VLA fine-tuning, the binding constraint is not algorithmic sophistication (better advantage estimators, more complex exploration strategies) but distribution alignment between the world model's training data and the policy's behavior. This is analogous to the test-time compute scaling literature's finding that verifier robustness, not search algorithm design, is the primary bottleneck — and it has the same practical implication: invest in better world model training data (broader coverage of failure modes, more policy-specific alignment trajectories) rather than more complex RL algorithms. Prior work that attempted sophisticated model-based RL for robotics but achieved disappointing results may have been limited by this distribution mismatch rather than by their algorithms.

Fourth, it demonstrates that emergent behaviors can arise from model-based RL without explicit reward shaping or demonstration of those behaviors. The self-correction behavior in Figure 3 — where the WMPO-trained policy lifts, realigns, and reinserts a misaligned block, a recovery sequence never present in the expert demonstrations — is not just a performance improvement; it is a qualitatively new capability. This has implications for how we think about data requirements for robot learning: if a world model can faithfully simulate the consequences of mistakes, the policy can learn to recover from those mistakes through trial and error, without anyone needing to demonstrate recovery behaviors. This makes the case that failure simulation is as important as success simulation in world model design — a world model that only simulates successful task executions (trained on expert data alone) would never enable the policy to learn recovery, because the policy would never experience failure in imagination and thus never need to recover.

Finally, WMPO makes lifelong learning through iterative self-improvement a credible research direction for VLA-based robotics. The lifelong learning experiment (Figure 6) shows that alternating between policy optimization (in the world model) and world model alignment (on new policy-collected data) produces steady improvement without additional expert demonstrations. This is a concrete instantiation of the "self-improving robot" vision: the robot gets better at its tasks through its own experience, with the world model serving as the bridge between limited real-world data and the large-scale imagined practice needed for RL. The fact that DPO fails to achieve iterative improvement (its performance fluctuates unstably) while WMPO sustains gains validates the on-policy + world model approach as the enabling mechanism — offline methods cannot track the evolving policy distribution, and online methods without a world model cannot amplify limited real-world data.

Follow-Up Research This Work Enables

Releasing and benchmarking a shared pretrained world model foundation model for robotics. The single largest barrier to replicating and extending WMPO is the absence of publicly available pretrained world model weights. The paper reports pretraining on OXE for 12M steps on 32 H100 GPUs — an investment of thousands of GPU-hours that few academic labs can replicate. A natural and high-impact follow-up would be to release the pretrained world model (or train a larger, more capable version on expanded datasets including OXE, DROID, and other large-scale robot manipulation corpora) as a community resource. The key evaluation would be: how many downstream tasks can this shared world model support with modest fine-tuning (P=128–1280 trajectories)? Does performance scale with world model size (parameters, pretraining data) in a predictable way? Such a release would transform WMPO from a method that requires massive pretraining infrastructure into an accessible tool that individual labs can fine-tune for their specific robots and tasks, analogous to how VLAs like OpenVLA are fine-tuned from shared pretrained checkpoints. A benchmark suite — perhaps 10–20 diverse manipulation tasks across multiple embodiments with standardized evaluation protocols — would enable apples-to-apples comparisons of world model quality and downstream policy improvement.

Combining WMPO's world model with model-based planning (search over imagined action sequences). WMPO uses the world model exclusively for autoregressive rollout generation during GRPO training — the policy generates an action sequence, the world model simulates its consequences, and the reward model scores the result. This is data amplification but not planning. A natural extension is to use the world model for explicit search over action sequences: from a given state, sample multiple candidate action chunks, simulate their consequences with the world model, evaluate each imagined outcome with the reward model, and select the best action to execute. This is model-predictive control (MPC) with a learned video world model, and it would exploit a capability (counterfactual simulation) that the world model clearly possesses but WMPO does not use. The experiment would compare: (a) the WMPO-trained policy executing open-loop, (b) the same policy augmented with MPC at test time (replanning after every K steps using the world model), and (c) a policy trained with GRPO on trajectories generated by MPC (a Dyna-style inner loop). The hypothesis: MPC enables recovery from errors that compound during open-loop execution, particularly on tasks requiring precise alignment (Square, ThreePieceAssembly) where small early errors cause later failure. This would strengthen the connection between WMPO and the model-based RL literature while potentially pushing success rates on the hardest tasks beyond the ~45% ceiling observed at P=1280.

Stress-testing the lower bound: at what base policy success rate does WMPO stop helping? The paper tests WMPO with base policies achieving 19.5–46.9% success rates (Table 1), and shows improvement across this range. But the lower bound is unknown: if the base policy succeeds only 5% of the time, the world model alignment data is dominated by failures, the reward model has few positive examples, and GRPO has almost no successful trajectories to reinforce. Does WMPO degrade gracefully (providing small but nonzero improvement) or fail catastrophically (performance drops below the base policy due to reward model noise or world model inaccuracy on out-of-distribution states)? This is a critical stress test because it determines WMPO's applicability envelope. The experiment would train base policies at systematically varied quality levels (e.g., by limiting the number of expert demonstrations: 10, 25, 50, 100, 200, 300) and measure WMPO improvement at each level on a fixed task (e.g., Square). The result would be a curve relating base policy quality to WMPO gain, identifying the threshold below which practitioners should first improve their base VLA via more imitation learning rather than applying WMPO. A negative result at very low base quality would not diminish WMPO's value — it would clarify its boundary conditions and prevent misapplication.

Cross-embodiment and cross-task transfer: does WMPO-trained robustness generalize beyond the training distribution? The paper's generalization experiments (Table 2) test visual appearance variations of the same task on the same robot. A more ambitious test would evaluate whether a policy optimized via WMPO on task A (e.g., Square) shows improved robustness on a related but different task B (e.g., a peg-in-hole insertion with different geometry), compared to a base policy trained on task A alone. The mechanism: WMPO's world model, pretrained on diverse OXE data, generates imagined trajectories with varied dynamics and visual conditions; the policy, by learning from this diverse imagined data, might acquire generalizable manipulation skills (alignment, insertion, recovery from misalignment) rather than task-specific strategies. The experiment would train WMPO on a source task, then evaluate zero-shot on a held-out target task requiring similar manipulation primitives but with different objects. A positive result (nonzero transfer) would significantly expand WMPO's value proposition beyond single-task optimization. A negative result would clarify that WMPO's improvements are task-specific and that generalization requires explicit multi-task training in the world model.

Quantifying and improving world model fidelity with standardized metrics. The paper provides no quantitative evaluation of world model accuracy (no FVD, PSNR, action-conditioned prediction error, or success-rate correlation between real and imagined trajectories). This is a significant barrier to reproducibility and iterative improvement: a practitioner cannot diagnose whether their world model is good enough. A follow-up study would establish a battery of world model evaluation metrics for robotic manipulation, including: (a) action-conditioned video prediction accuracy (how well do generated frames match real held-out trajectories under the same action sequence?), (b) success-rate correlation (do trajectories that the world model + reward model predict as successful actually succeed when executed on the real robot?), and (c) failure-mode coverage (does the world model generate the same types of failures — drops, collisions, misalignments — that the real policy produces, and at similar frequencies?). These metrics would be computed on a held-out set of policy-collected trajectories not used for fine-tuning. The study would then ablate WMPO's architectural choices (2D vs. 3D VAE, noisy-frame conditioning vs. clean, frame-level vs. global action conditioning) against these metrics to determine which components are essential for fidelity. This would transform WMPO from a monolithic system into a set of individually validated design decisions, making it easier for others to adapt, improve, or simplify.

WMPO for domains beyond rigid-body manipulation. All of WMPO's experiments involve rigid objects with stable geometries (blocks, sticks, coffee pods, assembly pieces). An important extension would test WMPO on tasks involving deformable objects (cloth folding, rope manipulation, dough shaping), granular materials (scooping, pouring), or tasks requiring force estimation from visual cues (insertion with extremely tight clearances, snap-fit assembly). These domains stress-test the world model's ability to simulate fine-grained physical interactions that are harder to capture from video data alone. The experiment would select 2–3 deformable or high-precision tasks from existing benchmarks (e.g., RLBench, ManiSkill, or custom real-world setups), train WMPO with the same protocol (OXE pretraining + P=128–1280 task-specific trajectories), and measure whether the approach transfers. A negative result on deformable tasks would not invalidate WMPO but would clarify that pixel-space video generation, as currently architected, has limitations in simulating physics that requires modeling material properties beyond visual appearance. A positive result would dramatically expand WMPO's applicability to household and manufacturing tasks that involve non-rigid objects.

Practical Applications and Downstream Use Cases

Cost-efficient fine-tuning of VLA models for specific robot deployments. A robotics company deploying a VLA-based manipulation system (e.g., a warehouse pick-and-place robot, a lab automation assistant, a household robot) currently faces a choice: collect thousands of expert demonstrations per new task to fine-tune the VLA via imitation learning, or accept the brittleness of a general-purpose VLA that was never optimized for their specific environment and objects. WMPO offers a third path: deploy the robot, let it attempt the task autonomously for 128–1280 trials (collecting both successes and failures), fine-tune a pretrained world model on this data, and run GRPO within the world model to produce a policy optimized for the specific task and environment. The cost savings come from two sources. First, self-collected data requires no human teleoperation — the robot collects it autonomously, and failures are valuable training signal rather than wasted attempts. Second, the 4× to 10× improvement in success rate demonstrated in Table 1 (e.g., Coffee from 43.8% to 75.0% at P=1280) means the robot wastes fewer attempts in production, directly reducing per-task completion time and increasing throughput. The paper's lifelong learning result (Figure 6) further suggests this improvement can be sustained over multiple deployment cycles, with the robot continuously getting better at its tasks without additional human effort.

Data generation for self-improving VLA training pipelines. Large-scale VLA training (e.g., for general-purpose home robots or flexible manufacturing systems) requires enormous quantities of high-quality trajectory data. Currently, this data is collected via human teleoperation, which is slow, expensive, and biased toward successful executions. WMPO enables a self-improving data flywheel: start with a moderate amount of expert data to train a base VLA; deploy the VLA to collect autonomous trajectories (including failures); use WMPO to optimize the VLA; use the optimized VLA to collect higher-quality trajectories (higher success rate, more efficient execution, novel recovery behaviors); add these to the training corpus; retrain or fine-tune the next generation of VLA on this expanded dataset. Each iteration improves both the policy and the training data, creating a positive feedback loop that reduces dependence on human demonstrators over time. The paper's lifelong learning experiment (Figure 6: from 46.9% to ~65% with only self-collected data) provides a proof of concept for this flywheel, and scaling it to more tasks and longer horizons is a natural engineering extension.

Pre-deployment robustness validation in an imagined environment. Before deploying a VLA policy on a physical robot in a safety-critical or high-cost setting (e.g., surgical assistance, nuclear decommissioning, space robotics), it is essential to validate that the policy can handle a wide range of failure scenarios. Physical testing is limited by time and safety constraints — you cannot test thousands of collision scenarios on a real surgical robot. WMPO provides an imagined validation environment: after aligning the world model to the deployment policy, generate thousands of imagined trajectories from diverse initial states, including deliberately induced perturbations (object displacements, visual occlusions, unexpected obstacles), and measure the policy's success rate and failure modes. Because the world model simulates both successes and failures (as demonstrated qualitatively in Figures 7–8), this validation can surface brittleness (e.g., specific collision geometries from which the policy cannot recover) before the robot ever encounters them in reality. The reward model's high accuracy (F1 > 0.95) ensures that success/failure classification in imagination is reliable. This is not a replacement for physical testing, but a complement that dramatically expands the set of scenarios that can be evaluated before deployment, potentially catching dangerous failure modes that would be too rare or too dangerous to test physically.

When to Prefer This Method

Prefer WMPO over real-world on-policy RL (GRPO, PPO) when the real-world interaction budget is severely constrained (fewer than ~1000 trajectories, typical for real-robot deployments) and a pretrained world model (or the resources to fine-tune one from a shared checkpoint) is available. The paper shows WMPO achieves 47.1% mean success with 128 real trajectories vs. 33.2% for online GRPO (Table 1) — a 13.9 percentage-point gap from the same real-world data.
Prefer WMPO over offline preference methods (DPO) when sustained iterative improvement is needed. DPO plateaus after consuming its offline dataset (Figure 6: unstable performance, no clear trend), while WMPO improves steadily over multiple iterations because it generates fresh on-policy imagined data each round. For deployments where the robot will operate continuously and should improve from its own experience, WMPO's on-policy mechanism is essential.
Prefer WMPO when the task requires learning recovery behaviors absent from demonstration data. The paper's self-correction result (Figure 3) shows WMPO learning to recover from collisions — a behavior never demonstrated. For tasks where failure modes are predictable but recovery strategies are complex and hard to demonstrate (e.g., untangling, regrasping after slippage, realigning misassembled components), WMPO's model-based RL can discover these behaviors through trial and error in imagination.
Prefer scaling pretraining (better base VLA) over WMPO when the base policy's success rate is below some critical threshold (likely ~10–15%, though the lower bound is not characterized in the paper). WMPO amplifies existing capability but does not create it from nothing — if the VLA fundamentally does not understand the task or never produces successful trajectories, the world model alignment data lacks positive examples and GRPO has nothing to reinforce. The paper does not establish this threshold quantitatively, but the per-task results (Table 1) show that tasks starting at 19.5% (ThreePieceAssembly) improve less in relative terms than tasks starting at 43.8% (Coffee), suggesting diminishing returns as base quality drops.