ArXiv: 2602.22010

🎯 Pitch

Predicting raw future observations is too noisy for robot control, while compact latent plans lose the precision needed for fine-grained manipulation—WoG bridges this gap by learning an action-conditioned predictive space that gives Vision-Language-Action models exactly the guidance they need for 42% better real-world pick-and-place.

1. Executive Summary

This paper introduces WoG (World Guidance), a framework that compresses future observations into compact conditions and injects them into the action inference pipeline of Vision-Language-Action models, then trains the VLA to predict these conditions alongside actions at test time. Evaluated on the SIMPLER simulation benchmark and real-world robotic manipulation tasks using a Prismatic VLM backbone with a DiT action head, WoG achieves consistent improvements over existing world action models and latent action models — most notably reaching 70.9% average success on Google Robot tasks and demonstrating a 42% improvement on real-world pick-and-place when augmented with UMI data. The method establishes that future observation modeling can enhance fine-grained action generation with strong generalization across visual distribution shifts, but its gains diminish on tasks requiring precise spatial reasoning — confirming that the condition space amplifies existing capabilities of visual encoders without compensating for their intrinsic deficiencies in fine-grained geometry.

2. Context and Motivation

The Core Problem: The Representation-quality Trade-off in World-Action Models

The fundamental question this paper addresses is deceptively simple: if a Vision-Language-Action (VLA) model is asked to predict future information to improve its own action generation, what kind of information should it predict? Current approaches oscillate between two extremes, each failing in a distinct way. On one side, methods that predict rich, explicit future observations (depth maps, images, video frames) are crippled by redundancy: the raw visual signal contains enormous amounts of detail irrelevant to the manipulation task, increasing computational overhead and impeding efficient learning. On the other side, methods that compress future dynamics into sparse latent representations for high-level planning lose fine-grained precision, capturing only coarse motion trends insufficient for the precise end-effector positioning required in real manipulation. This is the representation-quality trade-off — and the paper's central claim is that existing methods occupy the two endpoints of this spectrum, with no one occupying the critical middle ground.

The problem is most acute in settings that demand both capabilities simultaneously. Consider a pick-and-place task with an obstacle in the path: the robot must plan around the obstacle (requiring global trajectory reasoning that benefits from future-aware dynamics) while also precisely positioning the gripper for the grasp (requiring fine-grained spatial accuracy). A representation that prioritizes compactness loses the precision; a representation that prioritizes richness overwhelms the model with noise. The paper's key insight is that the optimal space for world-action modeling is an action-condition space — one whose information content is defined by what action prediction actually needs, rather than by whatever the visual encoder happens to capture.

Why This Matters: The Stakes for Generalist Robot Policies

This gap is not a theoretical curiosity — it has direct, tangible consequences for the field's trajectory toward generalist robot policies. The current dominant paradigm for building VLAs involves pretraining on large-scale, heterogeneous robot datasets like Open X-Embodiment (OXE), then fine-tuning on target tasks with modest amounts of expert demonstrations. In this pipeline, the quality of the representations learned during pretraining is the single biggest determinant of downstream performance and generalization. If the pretrained model learns representations that are either too sparse (losing precision) or too redundant (losing efficiency), no amount of fine-tuning can fully recover the missing capability.

The generalization challenge makes this representation-quality trade-off even more acute. Consider what happens under visual distribution shifts — new backgrounds, different lighting, novel object appearances. A model that has overfit to the redundant visual details present in video prediction (as many world-action models do) will be highly brittle to these shifts, because its learned representations will have coupled manipulation-relevant dynamics with task-irrelevant texture and lighting information. Conversely, a model relying on compressed latent actions that capture only the grossest motion signals will fail when the task requires adapting to new object geometry — the representation simply doesn't encode enough information to distinguish between valid and invalid grasps on a novel object shape. The real-world robotic deployment results in this paper (Section 5) bear this out empirically: baseline methods show pronounced degradation under background and object changes, while the paper's condition-space approach maintains performance with minimal drop across all out-of-distribution scenarios tested.

There is also a data scalability dimension to why this problem matters. The paper explicitly investigates whether world-action modeling can benefit from large-scale human manipulation videos — a data source that is far more abundant than robot demonstrations but lacks action annotations in most cases. For a method to exploit this data effectively, it needs a predictive target that (a) can be supervised from video alone (no action labels required), (b) captures manipulation-relevant information shared across human and robot embodiments, and (c) is compact enough that learning from noisy, uncurated video data is tractable. Raw video prediction violates (c) — it's too expensive and unstable to train on large-scale unlabeled videos. Latent action compression violates (a) and (b) — it requires action labels and is often embodiment-specific. The paper's condition-space prediction framework is designed to satisfy all three criteria simultaneously, making it a natural fit for the data-scaling regime the field is moving toward.

Prior Approaches and Where They Fall Short

The paper categorizes existing methods for incorporating future information into VLA models into two streams, each with distinct limitations.

Stream 1: World Action Models — Rich but Redundant

World action models follow the intuition that if an agent can anticipate what will happen next, it will act better. The implementation of this intuition varies: some methods (VPP, GeACT) inject intermediate features from video generation models directly into the action module; others (UniPi, WorldVLA, RYNNVLA) train VLM backbones to generate future images internally; still others (DreamVLA, FlowVLA) predict explicit modalities like depth, optical flow, or the latent representations of foundation vision models like DINOv2.

The common thread across all these variants is that they attempt to predict task-agnostic visual representations — features designed for general image understanding rather than specifically for manipulation. This is the source of their redundancy problem. As the paper notes in the Introduction, prior work (UniVLA, LAM-Learn, STAMO) has shown that these generic semantic spaces "often contain substantial redundancy for downstream manipulation tasks." A DINOv2 feature map encodes object boundaries, semantic categories, and part-level structure — all useful for general vision — but for a pick-and-place task, the critical information is much narrower: the spatial relationship between the gripper and the target object, the anticipated trajectory through free space, and the contact dynamics at the grasp point. Everything else in the feature map is noise relative to the action prediction objective. This redundancy has concrete consequences: it impedes pretraining efficiency (the model must learn to ignore most of the signal it's predicting), and it limits cross-scenario scalability (because the redundant details couple to specific visual appearances, making the predictions fragile under distribution shifts).

The DreamVLA method represents the closest prior approach to the paper's own, since it also uses DINOv2 features as a predictive target. However, DreamVLA directly regresses the full, uncompressed DINOv2 representations rather than querying a compact, action-relevant subset. This distinction — full feature map vs. queried condition — is central to the paper's contribution, and the ablation study in Section 4.4 directly tests it by comparing the full WoG framework against variants that predict uncompressed visual features.

Stream 2: Latent Action Models — Compact but Coarse

Latent Action Models (LAMs) were developed to address a different challenge: training VLA models effectively from large-scale, heterogeneous datasets collected across different robot embodiments. The core assumption is that despite differences in joint configurations, gripper types, and control frequencies, there exist high-level action representations that capture "skill-related but embodiment-agnostic" motion patterns. These representations are typically discrete (learned via vector quantization) and are supervised through visual reconstruction objectives — the model compresses an action sequence into a latent code, then a decoder reconstructs future observations from that code, forcing the latent to capture the visible effects of the action.

The methods in this stream differ in the reconstruction target: LAPA uses generative reconstruction, UniVLA uses discriminative representations, but all share the goal of producing compact, accurate latent action spaces. The problem, as the paper identifies it, is that "the compression performed by LAMs often resembles PCA-like extraction of maximum-variance signals, yielding coarse planning representations and, in some cases, introducing noise from scenarios that is weakly correlated with actions." This is a precise diagnosis: because the visual reconstruction objective cares about pixel-level accuracy, the latent code is incentivized to capture the largest sources of visual variation — which may include movements of the robot arm, lighting changes, or background motion — rather than the specific control-relevant dynamics. The result is a representation that is good enough for high-level planning (the robot knows roughly where to go) but insufficient for fine-grained execution (it doesn't know exactly how to position the gripper for a successful grasp).

Recent work has attempted to patch this limitation. Methods like VITA, XR1, and CLAP incorporate explicit action reconstruction into the LAM training objective alongside visual reconstruction. Others like VITA and ViPRA introduce video generation as a co-training objective to compensate for the lack of fine-grained guidance. These are steps in the right direction, but the paper argues they still miss the core insight: rather than trying to recover fine-grained information by adding auxiliary objectives (which increases training complexity and can introduce instabilities), one should define the predictive space from the start to be exactly as rich as action prediction requires and no richer. The video generation co-training in VITA and ViPRA is particularly telling — it's essentially trying to back-door fine-grained information into the latent space through a secondary pathway, which the paper's approach addresses directly by making the primary predictive target itself action-relevant.

The Unoccupied Middle: Why Prior Work Missed the Condition Space

The paper channels a deeper theoretical perspective that prior work implicitly overlooked. Both world action models and latent action models define their predictive targets based on the properties of the input data — rich visual features in the first case, compressed visual-motion features in the second. Neither approach asks: what information does the action head actually need? The paper reframes the problem through what one might call the sufficient condition criterion: the predictive space should satisfy the property that its information serves as a sufficient and effective condition for action generation. This reframes the problem from "what visual information is available?" to "what information does the action model treat as causally relevant?"

The paper argues that this condition space has a natural discovery mechanism: if you inject future observations directly into the action inference pipeline (via a cross-attention mechanism), the model will learn to extract precisely the information from those observations that improves action prediction. The representation encoded through this pipeline is the desired condition space — by construction. This is the conceptual innovation: rather than designing a predictive target and hoping it aligns with action needs, the paper lets the action prediction objective itself define what's worth predicting.

How the Paper Positions Itself

The paper positions WoG not as a new VLA architecture or a better vision encoder, but as a new predictive framework that restructures what a world-action model predicts. The key distinction is captured in the paper's own language: "Our method stands out by learning a condition space optimized for action generation, designed to provide more efficient support for enhancing model performance" (Section 2.1). This is a statement about optimization objectives, not about architectures. The architectural components — Q-former encoders, Prismatic VLM backbones, DiT action heads — are all off-the-shelf. The novelty is in how they are connected and what target they are trained to predict.

The two-stage training curriculum is essential to this positioning. Stage I discovers the condition space by directly injecting future information into action generation — the model uses future observations as auxiliary conditions to improve its predictions. This stage jointly optimizes both the encoder (which compresses future observations into conditions) and the action head (which learns to exploit those conditions). Stage II then freezes the encoder and trains the VLM backbone to predict those conditions from current observations alone, alongside predicting actions. This decouples the model from needing future observations at test time while preserving the benefits of future awareness — the VLM learns to internalize the predictive competence that the Future Encoder provided externally.

This two-stage design positions WoG in relation to several broader research threads. Relative to knowledge distillation, it is a form of self-distillation where the "teacher" is not a larger model but a different information source (future observations). Relative to JEPA-style predictive architectures, it shares the philosophy of predicting in a learned representation space rather than in raw observation space, but differs in that the representation space is defined by action relevance rather than by self-supervised consistency. Relative to planning methods that use learned world models, WoG learns to predict a compressed state that is sufficient for action selection rather than a generative model of full future observations — it trades off the ability to visualize futures (which it cannot do) for the efficiency of predicting only what actions need.

The paper also positions itself explicitly in the data scaling landscape. By designing the condition prediction target to be both embodiment-agnostic (capturing object dynamics rather than robot-specific kinematics) and learnable from video alone (requiring no action labels), the framework is engineered to benefit from large-scale human manipulation videos — a data source that neither pure world action models (too expensive to train on uncurated video) nor pure latent action models (require action labels) can exploit effectively. The experiments in Section 5.4, where models trained with unannotated human videos show improved generalization even on robot tasks, are the empirical realization of this positioning.

Finally, the paper is careful to delineate what WoG does not address. It acknowledges that modeling fine-grained spatial constraints (precise relative positioning for stacking, drawer alignment) remains a persistent challenge "independent of the future encoder" and that solving this "requires dedicated spatial mechanisms or historical observation modeling, which lies beyond the scope of this work." This honesty about scope limits positions the contribution as solving one part of the representation-quality trade-off (making world-action modeling both efficient and generalizable) without claiming universality.

3. Technical Approach

3.1 Reader Orientation

WoG is a training framework that teaches a Vision-Language-Action model to predict compact, action-relevant summaries of future visual observations alongside actions, so that at test time the model can internally anticipate what will happen without needing access to actual future frames. The core idea is to first inject future observations directly into the action prediction pipeline to let the model discover what information from those observations is actually useful for action generation (Stage I), then freeze that discovered representation and train the model to predict it from current observations alone, alongside the actions themselves (Stage II). This two-stage curriculum solves the representation-quality trade-off: the predictive target is compact (avoiding the redundancy of full visual features) yet rich enough to carry fine-grained manipulation dynamics (avoiding the coarseness of latent action codes).

3.2 Big-Picture Architecture (Diagram in Words)

The system has seven major components arranged in a two-stage pipeline:

  1. VLM Backbone (Prismatic VLM) — a pretrained vision-language model that encodes the current RGB observation and language instruction into a latent representation $\boldsymbol{z}$. Adopted from OpenVLA's architecture.

  2. Frozen Vision Foundation Models (DINOv2 + Wan VAE Encoder / SigLIP) — pretrained and kept frozen throughout all training. They encode future observations into high-dimensional feature maps carrying semantic, spatial, and temporal information.

  3. Q-Former-Based Future Encoder (Trainable in Stage I, Frozen in Stage II) — a learnable module with $N = 16$ query tokens that cross-attends into the frozen vision features and compresses them into a compact condition vector $\boldsymbol{O}^c \in \mathbb{R}^{32}$. This is the module that discovers the action-relevant condition space.

  4. DiT Action Head — a diffusion transformer that takes the VLM's latent $\boldsymbol{z}$ and (in Stage I) the future condition $\boldsymbol{O}^c$ as input, and generates action sequences via rectified flow denoising. In Stage II it receives only $\boldsymbol{z}$.

  5. Learnable Query Embeddings (Stage II only) — a set of 16 learnable tokens that cross-attend to the final hidden states of the VLM output, producing a predicted condition representation $f_q(\boldsymbol{O}, \boldsymbol{l})$ that is supervised to match the frozen $\boldsymbol{O}^c$.

  6. Rectified Flow Denoising Loss — the objective function for action generation, predicting velocity fields in the action space rather than clean actions directly, enabling iterative refinement.

  7. Cosine Similarity Alignment Loss (Stage II only) — the objective that forces the VLM's internal representations to encode the same information as the Future Encoder's condition vectors, teaching the model to anticipate future dynamics from current observations.

Information flow, Stage I: Current observation + language instruction → VLM Backbone → latent $\boldsymbol{z}$. In parallel, future observations → Frozen Vision Encoders → feature maps → Q-Former Future Encoder → compact condition $\boldsymbol{O}^c$. Both $\boldsymbol{z}$ and $\boldsymbol{O}^c$ enter the DiT Action Head via cross-attention, where $\boldsymbol{O}^c$ modulates the denoising process to produce actions $\boldsymbol{A}_{t:t+T}$. The loss supervises only action prediction; the Q-Former learns by backpropagation through the action head what information from future observations is useful.

Information flow, Stage II: Current observation + language instruction → VLM Backbone → latent $\boldsymbol{z}$. The Q-Former Future Encoder is frozen and produces target conditions $\boldsymbol{O}^c$ from future observations, but these never enter the DiT Action Head. Instead, learnable query embeddings cross-attend to the VLM's final hidden states, producing predicted conditions $f_q(\boldsymbol{O}, \boldsymbol{l})$. The DiT Action Head receives only $\boldsymbol{z}$ and predicts actions. Two losses are applied simultaneously: the rectified flow loss on actions and the cosine similarity loss between predicted conditions $f_q(\boldsymbol{O}, \boldsymbol{l})$ and target conditions $\boldsymbol{O}^c$. At test time, only the VLM Backbone and DiT Action Head are used — the model internally anticipates future conditions without any external future observations.

3.3 Roadmap for the Deep Dive

  • First, the formal problem statement (Section 3.1) establishes the probabilistic framework — what the model must compute at test time, why future observations are available during training but not at inference, and how the joint distribution factorization in Equation 1 motivates the two-stage training design. This is essential for understanding why the training curriculum is structured as it is.
  • Second, Stage I: World Guidance (Section 3.2) — how future observations are encoded, compressed, and injected into the action head, including the architecture of the Q-Former Future Encoder, the choice of frozen vision foundation models, and the rectified flow loss (Equation 2). This stage answers: how does the system discover what information in future observations matters for action generation?
  • Third, Stage II: World Inference (Section 3.3) — how the discovered condition space is frozen and used as a predictive target for the VLM backbone, including the learnable query embeddings, the cosine similarity alignment loss, and the joint training objective (Equation 3). This stage answers: how does the system transfer future-awareness into the model so it works at test time without future observations?
  • Fourth, the mechanism for learning from human manipulation videos (Section 3.4) — two complementary strategies for incorporating large-scale human video data (with and without action annotations) into the WoG framework, exploiting the condition prediction objective as a bridge between human and robot embodiments.
  • Fifth, the UMI data integration pathway (Section 3.4) — how WoG's embodiment-agnostic condition space enables learning from egocentric, kinematically-different robot data collected via UMI devices, validated in Section 5.5.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that the optimal predictive target for world-action models is not raw visual features or compressed latent actions, but rather a condition space discovered by directly injecting future observations into the action prediction pipeline and then teaching the model to predict those conditions from current observations.

Formal Problem Statement and Probabilistic Framework

What the model must do at test time. A VLA model receives, at time step $t$, a current RGB observation $\boldsymbol{O}_t$ and a language instruction $\boldsymbol{l}$. The VLM backbone encodes these into a latent representation $\boldsymbol{z} \leftarrow f(\boldsymbol{O}_t, \boldsymbol{l})$, which is then fed into an action head to generate a sequence of $T$ future actions $\boldsymbol{A}_{t:t+T}$ by maximizing the likelihood $\boldsymbol{P}(\boldsymbol{A}_{t:t+T} \mid \boldsymbol{z})$. At test time, the model has access only to current observations — it cannot peek into the future.

What information is available during training. During training, the system has access to future observations for the next $T$ time steps because it is learning from recorded trajectories. The key question is: how should these future observations be used to improve the model's action prediction capability, given that they will not be available at test time?

The discovery mechanism. In Stage I, WoG compresses future observations from the next $T$ time steps into a compact condition vector $\boldsymbol{O}^c_{t:t+T}$. This condition is injected alongside the current latent $\boldsymbol{z}$ into the action head, so the model learns to generate actions by modeling:

P(At:t+T∣z,Ot:t+Tc)\boldsymbol{P}(\boldsymbol{A}_{t:t+T} \mid \boldsymbol{z}, \boldsymbol{O}^c_{t:t+T})

where $\boldsymbol{P}(\cdot \mid \cdot, \cdot)$ denotes the conditional probability of the action sequence given both the current observation encoding and the compressed future observations. The key property of this formulation is that $\boldsymbol{O}^c$ is learned — it is not a fixed feature but the output of a trainable encoder that is optimized end-to-end through the action prediction loss. This means $\boldsymbol{O}^c$ will converge to whatever representation of future observations is most useful for the downstream task of action generation, which is exactly the sufficient condition property the paper aims for.

The factorization that motivates Stage II. Under the assumption of deterministic environmental dynamics (the future is fully determined by the present state and the actions taken, so future observations add no genuinely new information beyond what is already implied by the current observation and the policy itself), the joint distribution of actions and future conditions can be factored as:

P(At:t+T,Ot:t+Tc∣z)=P(At:t+T∣z,Ot:t+Tc)  P(Ot:t+Tc∣z)\boldsymbol{P}(\boldsymbol{A}_{t:t+T}, \boldsymbol{O}^c_{t:t+T} \mid \boldsymbol{z}) = \boldsymbol{P}(\boldsymbol{A}_{t:t+T} \mid \boldsymbol{z}, \boldsymbol{O}^c_{t:t+T}) \; \boldsymbol{P}(\boldsymbol{O}^c_{t:t+T} \mid \boldsymbol{z})

where $\boldsymbol{P}(\boldsymbol{A}_{t:t+T} \mid \boldsymbol{z}, \boldsymbol{O}^c_{t:t+T})$ is the action likelihood conditioned on both current encoding and future conditions (learned in Stage I), and $\boldsymbol{P}(\boldsymbol{O}^c_{t:t+T} \mid \boldsymbol{z})$ is the conditional distribution of future conditions given only the current observation encoding (to be learned in Stage II).

What this factorization accomplishes. At test time, future observations are unavailable, so the model must marginalize over them — that is, it must predict actions from $\boldsymbol{z}$ alone. Under the deterministic dynamics assumption, the optimal way to do this is to have the VLM backbone itself learn to predict $\boldsymbol{O}^c_{t:t+T}$ from $\boldsymbol{z}$, and then condition action prediction on this internally-generated prediction. The factorization tells us that if we can train the VLM backbone to model $\boldsymbol{P}(\boldsymbol{O}^c_{t:t+T} \mid \boldsymbol{z})$ and the action head to model $\boldsymbol{P}(\boldsymbol{A}_{t:t+T} \mid \boldsymbol{z}, \boldsymbol{O}^c_{t:t+T})$, then at test time the model can sample a future condition from its own predictions, feed it into the action head, and produce actions that are optimized under the joint distribution.

Why this form rather than alternatives. An alternative approach would be to train the VLM backbone to predict raw future observations (images, depth maps) and then feed those predictions into the action head. This would violate the compactness requirement because raw visual predictions are high-dimensional and contain substantial task-irrelevant information — the model would waste capacity modeling background textures, lighting variations, and object appearances that don't affect action selection. Another alternative would be to bypass future prediction entirely and just train the model end-to-end on action prediction from current observations (the vanilla VLA approach). This would violate the expressiveness requirement because the model would have no incentive to anticipate dynamics, obstacle trajectories, or contact events — it would be purely reactive. The WoG factorization sits between these extremes: the predictive target $\boldsymbol{O}^c$ is learned to be exactly as rich as action prediction requires (because it is the output of a module trained through the action loss) and no richer (because its dimensionality is constrained to 32 by the Q-Former bottleneck).

The two-stage operationalization. The factorization directly motivates the two-stage training curriculum. Stage I learns the first factor $\boldsymbol{P}(\boldsymbol{A}_{t:t+T} \mid \boldsymbol{z}, \boldsymbol{O}^c_{t:t+T})$ by training the system to generate actions using ground-truth future conditions. This simultaneously trains the Q-Former Future Encoder (which produces $\boldsymbol{O}^c$) and the DiT Action Head (which learns to exploit $\boldsymbol{O}^c$). Stage II learns the second factor $\boldsymbol{P}(\boldsymbol{O}^c_{t:t+T} \mid \boldsymbol{z})$ by freezing the Q-Former to fix the target condition space, then training the VLM backbone to predict these conditions from current observations alone. The action head is simultaneously trained to predict actions from $\boldsymbol{z}$ alone, implementing the marginalization $\boldsymbol{P}(\boldsymbol{A}_{t:t+T} \mid \boldsymbol{z})$ that will be used at test time.

Stage I: World Guidance — Discovering the Condition Space

Purpose. Stage I answers the question: what information from future observations is sufficient and effective for improving action prediction? Rather than hand-designing this information (which would risk either redundancy or insufficiency), the paper lets the action prediction objective itself drive the discovery process. Future observations are fed into the action pipeline, and a learnable encoder is optimized to extract whatever subset of those observations helps the action head produce better actions.

Encoding the current observation and instruction. The current RGB observation $\boldsymbol{O}_t$ and the language instruction $\boldsymbol{l}$ are processed by a Prismatic VLM backbone (the same architecture used in OpenVLA). Following the design of CogACT, the latent representation $\boldsymbol{z}$ is taken as the output feature of the last learnable token from the VLM — this single vector serves as the summary of all visual and linguistic context that the action head receives. The choice to use the last learnable token rather than pooling over all output tokens is significant: in a causal transformer architecture, the final token has attended to all previous tokens and thus aggregates the most comprehensive context. This is the same principle behind using the CLS token in BERT-style models or the last hidden state in autoregressive LMs.

Encoding future observations. Future observations are sampled at a temporal frequency of one-quarter relative to the action sequence: for the default prediction horizon of $T = 16$ action steps, the system uniformly samples 4 frames to serve as the future observation sequence (this is detailed in Appendix 7). These 4 frames are then encoded by a combination of frozen pretrained vision models. By default, the paper uses:

  • DINOv2 for discriminative and semantic features — this is a self-supervised vision transformer that produces feature maps capturing object boundaries, part-level structure, and semantic categories, all useful for understanding what is in the scene and where objects are relative to each other.
  • Wan VAE Encoder for generative and spatiotemporal features — this is the encoder from a video generation model that compresses spatiotemporal information, capturing motion patterns, object dynamics, and how the scene evolves over the sampled frames.

The DINOv2 features are extracted per-frame (spatial semantics for each future timestep). The Wan VAE processes the current observation as the initiating frame and the sampled future frames jointly, encoding temporal and spatial features that capture how the scene changes. Both feature maps have their last two spatial dimensions flattened and are projected into a unified embedding space matching the hidden dimension of the Q-Former, then stacked together. This design is extensible — the paper explicitly notes that other pretrained visual encoders (SigLIP, SAM) can substitute for or augment these choices, and Section 4.3 empirically evaluates alternative configurations.

The Q-Former-Based Future Encoder. The stacked visual features from the frozen foundation models are fed into a learnable Q-Former-based encoder (inspired by BLIP-2's architecture). A Q-Former uses a set of learnable query tokens that cross-attend into the visual feature sequences — rather than processing all visual tokens (which would be computationally expensive and produce a high-dimensional output), the queries selectively extract the information most relevant to the downstream task. The paper instantiates the Q-Former with $N = 16$ learnable query tokens. These 16 queries attend into the combined visual features via cross-attention mechanisms and produce 16 output representations, which are then projected to form the final condition vector:

Oc∈R32\boldsymbol{O}^c \in \mathbb{R}^{32}

where $\boldsymbol{O}^c$ is the compact condition representation with dimensionality 32. This is remarkably low-dimensional — a typical DINOv2 feature map for a single image has thousands of dimensions, and a combined multi-frame feature set is even larger. The Q-Former is thus performing extreme compression, from thousands of visual feature dimensions down to 32 scalar values, and the compression is guided entirely by what helps the downstream action head.

Why the Q-Former rather than direct regression. A simpler alternative would be to directly regress the full DINOv2 or VAE feature maps as predictive targets, without the Q-Former bottleneck (this is essentially what DreamVLA does, and it is tested as an ablation in Section 4.4 as "WoG w/o Future Enc."). The Q-Former serves three functions. First, it filters redundancy: by forcing the encoder to compress through a 32-dimensional bottleneck, it ensures that only the most action-relevant information survives. Information about background textures, lighting, or object appearances that don't affect manipulation will be discarded because modeling them doesn't improve the action loss. Second, it provides a learnable query mechanism: the 16 query tokens can learn to attend to different aspects of the visual features — some might focus on object positions, others on gripper-object relationships, others on motion trajectories — creating a disentangled condition space. Third, it creates a low-dimensional target that is tractable for the VLM backbone to predict in Stage II. Predicting a 32-dimensional vector from a single VLM output token is feasible; predicting a full DINOv2 feature map (tens of thousands of dimensions) would be far harder and would dominate the training objective.

Injection into the action head. The queried condition $\boldsymbol{O}^c$ is injected into each block of the DiT (Diffusion Transformer) action head, where it performs cross-attention with the current latent representation $\boldsymbol{z}$. Specifically, within each DiT block, $\boldsymbol{O}^c$ serves as the key and value in a cross-attention operation, while $\boldsymbol{z}$ (after being processed through the block's self-attention and feed-forward layers) serves as the query. This means that at every layer of the denoising process, the model can consult the future condition to inform its action predictions — the condition modulates the entire depth of the action generation pipeline, not just its input.

Action generation via rectified flow. The action head generates actions using a rectified flow diffusion process rather than directly regressing action values. In diffusion-based action generation, the model learns to denoise a sequence of actions from pure noise to a valid trajectory. Rectified flow is a specific formulation of this idea where the model predicts the velocity field — the direction and magnitude of change needed to transform a noisy sample into a clean one — rather than predicting the clean sample directly.

The loss function for Stage I is:

LI=Eτ,A[∥vθ(Aτ,τ,z,Oc)−v∗∥22]\mathcal{L}_{\text{I}} = \mathbb{E}_{\tau, A} \left[ \| v_\theta(A_\tau, \tau, \boldsymbol{z}, \boldsymbol{O}^c) - v^* \|_2^2 \right]

where $\tau \in [0, 1]$ is the scheduling timestep that controls how much noise has been added to the action sequence, $A_\tau$ is the noisy action sequence at timestep $\tau$ (with $\tau = 0$ being pure noise and $\tau = 1$ being clean actions), $v_\theta(A_\tau, \tau, \boldsymbol{z}, \boldsymbol{O}^c)$ is the model's predicted velocity field (the direction to move the noisy actions toward cleaner actions), and $v^*$ is the ground-truth velocity field (computed from the clean action sequence and the noise schedule). The expectation $\mathbb{E}_{\tau, A}$ is taken over uniformly sampled timesteps and action sequences from the training data.

What this loss computes. For each training sample, a random timestep $\tau$ is drawn, the clean action sequence is corrupted with the corresponding amount of noise to produce $A_\tau$, and the model is asked to predict the velocity that would transform $A_\tau$ back toward the clean actions. The loss is the squared L2 distance between the predicted velocity and the true velocity. Minimizing this loss teaches the model to perform iterative denoising: at test time, the model starts from pure noise and repeatedly applies its predicted velocity updates to produce a valid action trajectory.

Why rectified flow rather than DDPM or direct regression. Direct regression (predicting actions as a deterministic function of observations) would produce a single action trajectory, which is problematic for manipulation tasks where multiple valid trajectories exist (e.g., the robot could avoid an obstacle by going left or right). Diffusion models capture this multimodality by sampling from a learned distribution. Rectified flow is chosen over alternative diffusion formulations (like DDPM) because it provides straighter trajectories through the noise space, requiring fewer denoising steps to produce high-quality samples — an important practical consideration for real-time robotic control where inference speed matters.

Why the condition appears in $v_\theta$ but not in the expectation. The future condition $\boldsymbol{O}^c$ is an input to the model $v_\theta$ but is not part of the expectation — it is a deterministic function of the future observations for each training sample. During training, the model learns to use $\boldsymbol{O}^c$ as additional context that helps it predict better velocities. The Q-Former encoder that produces $\boldsymbol{O}^c$ is trained jointly with $v_\theta$ through this loss: gradients flow from the velocity prediction error back through the DiT blocks, through the cross-attention operations, and into the Q-Former, updating its query tokens to extract whatever visual features are most helpful for reducing the action prediction error.

What Stage I produces. At the end of Stage I, the system has learned two things: (1) a Q-Former Future Encoder that maps future observations into a 32-dimensional condition $\boldsymbol{O}^c$ that is highly effective for guiding action generation, and (2) a DiT Action Head that knows how to exploit these conditions when generating actions. The VLM backbone has not yet been modified — it still produces $\boldsymbol{z}$ from current observations only, and it has no ability to predict future conditions. That capability is built in Stage II.

Stage II: World Inference — Internalizing Future Awareness

Purpose. Stage II answers the question: how can the model anticipate future conditions at test time, when future observations are not available? The strategy is to freeze the Q-Former Future Encoder (making $\boldsymbol{O}^c$ a fixed, stable target) and then train the VLM backbone to predict these conditions from current observations alone, while simultaneously training the action head to predict actions from $\boldsymbol{z}$ alone (without receiving $\boldsymbol{O}^c$ as input).

Why freeze the Future Encoder. If the Q-Former were allowed to continue training in Stage II, the condition space would be a moving target — the VLM backbone would be trying to predict representations that are themselves changing, which is known to cause training instability in distillation-style setups. Freezing the encoder defines a stable target space: the condition vectors $\boldsymbol{O}^c$ for each training sample are computed once (or on-the-fly from the frozen encoder) and do not change throughout Stage II. This is analogous to how knowledge distillation works — the teacher is frozen before the student is trained.

Decoupling future conditions from the action head. In Stage II, the future condition $\boldsymbol{O}^c$ is no longer injected into the DiT Action Head. The action head receives only the VLM output $\boldsymbol{z}$ as input, which means it must learn to generate actions without direct access to future information. This is the configuration that will be used at test time, so it is essential that the action head is trained under these conditions.

The condition prediction mechanism. To teach the VLM backbone to anticipate future conditions, the paper introduces a set of learnable query embeddings that operate on the VLM's output. The mechanism works as follows:

  1. The VLM processes the current observation $\boldsymbol{O}_t$ and language instruction $\boldsymbol{l}$, producing a sequence of hidden states at its final layer. Because the VLM architecture is causal, the later tokens in this sequence have attended to more context and thus carry more comprehensive information.

  2. The paper selects the last 4 tokens from the VLM's output hidden states. This is a design choice motivated by the same principle as using the last learnable token for $\boldsymbol{z}$: in a causal architecture, later tokens aggregate more information. Using 4 tokens rather than 1 provides slightly more capacity for the condition prediction while still being a compact summary.

  3. A set of 16 learnable query embeddings (matching the number of Q-Former query tokens from Stage I) performs cross-attention over these 4 VLM hidden states. The query embeddings attend to the VLM's representations and produce 16 output vectors.

  4. These 16 vectors are projected into the 32-dimensional condition space, producing the predicted condition:

fq(O,l)∈R32f_q(\boldsymbol{O}, \boldsymbol{l}) \in \mathbb{R}^{32}

where $f_q(\boldsymbol{O}, \boldsymbol{l})$ is the function implemented by the learnable queries, cross-attention, and projection layers, mapping from current observation and instruction to the predicted future condition.

The alignment loss. The predicted condition is supervised to match the frozen target condition using cosine similarity:

Alignment Loss=1−S[Oc,fq(O,l)]\text{Alignment Loss} = 1 - \mathcal{S}[\boldsymbol{O}^c, f_q(\boldsymbol{O}, \boldsymbol{l})]

where $\mathcal{S}[\cdot, \cdot]$ denotes the cosine similarity between two vectors, defined as:

S[a,b]=a⋅b∥a∥2∥b∥2\mathcal{S}[\boldsymbol{a}, \boldsymbol{b}] = \frac{\boldsymbol{a} \cdot \boldsymbol{b}}{\|\boldsymbol{a}\|_2 \|\boldsymbol{b}\|_2}

What this loss computes. Cosine similarity measures the angle between two vectors in the 32-dimensional condition space, producing a value in $[-1, 1]$ where 1 means perfect alignment (the vectors point in exactly the same direction), 0 means they are orthogonal, and -1 means they point in opposite directions. The loss $1 - \mathcal{S}[\cdot, \cdot]$ is minimized when the cosine similarity is 1, so it encourages the predicted condition to point in the same direction as the target condition. The loss ranges from 0 (perfect prediction) to 2 (completely wrong prediction).

Why cosine similarity rather than MSE. Mean squared error would penalize both directional errors and magnitude errors — the model would need to predict not just what future conditions exist but also how strongly they are present. For the purpose of encoding action-relevant information, direction is more important than magnitude: the condition space is used to modulate attention patterns in the DiT head, and attention weights depend on direction (via dot products) rather than absolute magnitudes. Cosine similarity focuses the model's capacity on learning the correct pattern of future conditions (which object will move where, what dynamics will unfold) without wasting capacity on predicting exact magnitudes (which may be less stable across different scenes). Additionally, cosine similarity is scale-invariant, which means the model doesn't need to learn the appropriate scaling of the 32-dimensional vectors — it only needs to get the relative values correct.

The action prediction loss. Simultaneously with the condition prediction, the DiT Action Head is trained to predict actions from $\boldsymbol{z}$ alone:

Action Loss=Eτ,A[∥vθ(Aτ,τ,z)−v∗∥22]\text{Action Loss} = \mathbb{E}_{\tau, A} \left[ \| v_\theta(A_\tau, \tau, \boldsymbol{z}) - v^* \|_2^2 \right]

Note the difference from the Stage I loss: the model $v_\theta$ now takes $\boldsymbol{z}$ but not $\boldsymbol{O}^c$ as input. The action head must learn to generate good actions without seeing future conditions, relying instead on whatever future-relevant information the VLM backbone has learned to encode in $\boldsymbol{z}$ through the condition alignment loss.

The combined Stage II loss:

LII=Eτ,A[∥vθ(Aτ,τ,z)−v∗∥22]+1−S[Oc,fq(O,l)]\mathcal{L}_{\text{II}} = \mathbb{E}_{\tau, A} \left[ \| v_\theta(A_\tau, \tau, \boldsymbol{z}) - v^* \|_2^2 \right] + 1 - \mathcal{S}[\boldsymbol{O}^c, f_q(\boldsymbol{O}, \boldsymbol{l})]

where $\mathbb{E}_{\tau, A}$ denotes the expectation over diffusion timesteps and action sequences, $v_\theta$ is the velocity prediction model (now taking only $\boldsymbol{z}$, not $\boldsymbol{O}^c$), $v^*$ is the ground-truth velocity, $\boldsymbol{O}^c$ is the frozen target condition from the Stage I encoder, and $f_q(\boldsymbol{O}, \boldsymbol{l})$ is the predicted condition from the learnable queries.

What this combined loss computes. For each training sample, two errors are computed and summed. The first term is the action prediction error — how far the model's predicted velocity field deviates from the true velocity, given only current observations. The second term is the condition prediction error — how poorly the VLM's internal representations align with the future condition that the Stage I encoder would produce. The losses are simply added; there is no explicit weighting hyperparameter. This equal weighting implies that the paper considers action prediction and condition prediction to be equally important objectives, and that the gradient magnitudes from both losses are naturally balanced (or that the optimal balancing is close to 1:1, which is an empirical claim).

How the gradient flows create the desired behavior. The key mechanism in Stage II is that the VLM backbone receives gradients from both loss terms. The action loss pushes the VLM to encode information in $\boldsymbol{z}$ that is immediately useful for action prediction. The alignment loss pushes the VLM to encode information in $\boldsymbol{z}$ that enables predicting future conditions — which, by the design of Stage I, is precisely the information that future observations carry about action-relevant dynamics. The combination of these two gradient signals teaches the VLM to produce a latent representation $\boldsymbol{z}$ that simultaneously supports accurate action generation and encodes anticipatory knowledge about future states. At convergence, the VLM backbone has internalized the predictive competence that the Future Encoder provided externally in Stage I.

Why this two-stage approach rather than end-to-end training. An alternative would be to train the entire system end-to-end from the start: have the VLM predict future conditions from current observations, use those predicted conditions in the action head, and train everything jointly with some combination of action loss and condition prediction loss. The paper does not provide an explicit ablation of this alternative, but several reasons for the two-stage design can be inferred. First, end-to-end training creates a chicken-and-egg problem: the action head needs good condition predictions to learn to use them, but the VLM can only learn to produce good condition predictions if the condition space is stable and meaningful, which requires the action head to already know what information is useful. The two-stage design breaks this circular dependency by first establishing the condition space using ground-truth future observations (Stage I), and then teaching the VLM to predict into that established space (Stage II). Second, Stage I with ground-truth future conditions provides a stronger learning signal — the model sees actual future observations rather than potentially-noisy predictions — which likely leads to a better condition space than would be discovered through end-to-end training. The condition space discovered in Stage I represents an upper bound on what future information can contribute to action prediction, and Stage II approximates that upper bound from current observations alone.

Post-Stage II training. After the two stages of pretraining, the model is fine-tuned on target task demonstrations. During fine-tuning, WoG mirrors the training protocol of Stage II only — maintaining the co-training objective that jointly optimizes action prediction and future condition forecasting. The Q-Former Future Encoder remains frozen, and the model continues to receive gradients from both the action loss and the alignment loss. This means that even during fine-tuning, the model is incentivized to maintain and refine its internal future prediction capability, not just to memorize the expert actions.

Test-time inference. At test time, only the VLM backbone and the DiT Action Head are used. The VLM encodes the current observation and instruction into $\boldsymbol{z}$. The DiT Action Head takes $\boldsymbol{z}$ and generates actions via iterative denoising. The condition prediction mechanism (learnable queries, alignment loss) is not used — its purpose was to shape the VLM's representations during training. The model's future awareness is now implicit in $\boldsymbol{z}$, which has been trained to carry the same action-relevant information that the Future Encoder extracted from future observations.

Learning from Human Manipulation Videos

Motivation. Robot demonstration data is expensive to collect — it requires teleoperation, careful setup, and produces relatively few trajectories per hour. Human manipulation videos, by contrast, can be collected much more rapidly (the paper cites a rate of approximately 450 trajectories per hour using VR devices, as detailed in Appendix 9.4 and Figure 6). However, most human video data lacks action annotations — the video shows what the human did, but the corresponding robot actions are unknown. The WoG framework is designed to exploit this abundant unannotated data through the condition prediction objective.

Strategy 1: Unannotated human videos only. In this setting, human manipulation videos are used exclusively in Stage II for condition prediction supervision. No action labels are required. The assumption is that the condition space discovered in Stage I (trained on robot data) captures dynamics that are shared across human and robot manipulation — object motion trajectories, contact events, spatial relationships between hands and objects — even though the embodiments differ. During Stage II, the VLM backbone receives condition prediction loss on the human videos (learning to anticipate the same 32-dimensional conditions from human-view observations) while receiving action prediction loss only on robot data (since human action labels are unavailable). The paper validates this strategy in Section 5.4 as the "w. human v." variant.

Strategy 2: Mixed annotated and unannotated human videos. In this more data-rich setting, a small subset of human videos (220 hours out of 1,920 total, approximately 11%) carries action annotations. These annotated videos are used in both Stage I and Stage II for action supervision, while all videos (annotated and unannotated) are used for condition prediction supervision. In Stage I, the annotated human subset joins the robot data for action prediction conditioned on future observations — this expands the condition space to capture human-specific manipulation knowledge that may be absent from robot demonstrations (e.g., more dexterous grasps, different object interaction strategies). In Stage II, the full corpus of 1,920 hours of human video is used for condition prediction, while action supervision is applied only to robot data and the annotated human subset. The paper validates this strategy in Section 5.4 as the "w. human v./a." variant.

Why condition prediction enables human-to-robot transfer. The condition space $\boldsymbol{O}^c$ is designed to be embodiment-agnostic: it captures object dynamics, motion patterns, and spatial relationships rather than robot-specific kinematics or gripper configurations. When a human picks up a cup, the condition vector encodes the trajectory of the cup, the approach direction of the hand, and the timing of the grasp — information that is equally relevant for a robot performing the same task. By training the VLM backbone to predict these conditions from human-egocentric video, the model learns to anticipate object-level dynamics that transfer across embodiments. The action head, which is trained only on robot data (or robot + annotated human data), learns to convert these anticipated dynamics into robot-specific action trajectories.

The UMI data integration pathway. UMI (Universal Manipulation Interface) is a data collection paradigm where robot trajectories are collected from an egocentric camera with a different kinematic configuration than the target robot. The paper collects 120 additional UMI trajectories for real-world pick-and-place and towel-folding tasks. Crucially, this UMI data is introduced only during the fine-tuning stage alongside expert demonstrations — the condition space established during OXE pretraining is not modified. Despite significant domain gaps (egocentric viewpoint, different action representations, different embodiment), the UMI data provides substantial performance gains (Section 5.5): a 42% improvement on pick-and-place (from 60% to 85%) and a 33% improvement on folding (from 60% to 80%). This demonstrates that the condition space learned during pretraining is robust enough to accommodate out-of-distribution observation-action pairs, and that the VLM backbone can learn to predict these conditions from very different visual inputs.

Design Choices Summary

Why 32-dimensional conditions. The condition dimensionality $D = 32$ is a design choice that balances expressiveness against tractability. If the dimension were too high, the condition would reintroduce the redundancy problem (and be harder for the VLM to predict). If too low, it would lose fine-grained information. The paper does not ablate this dimension, but the strong empirical results and the effective use of only 16 Q-Former query tokens suggest that 32 dimensions provide sufficient capacity for the manipulation tasks studied.

Why 16 Q-Former queries. The number of query tokens determines how many independent aspects of future observations the encoder can attend to. With 16 tokens producing 32 total dimensions after projection, each token contributes roughly 2 dimensions of the final condition space. This is a relatively small number, suggesting that the condition space captures a small set of manipulation-relevant factors (object positions, motion directions, contact states, grasp timing) rather than a distributed code.

Why the last 4 VLM tokens for condition prediction. In a causal transformer, earlier tokens have limited context while later tokens have attended to the full sequence. Using the last 4 tokens provides a richer basis for the cross-attention queries to work with than using a single token, while still being computationally lightweight. The 16 learnable query embeddings cross-attend into these 4 tokens, effectively pooling information from the VLM's final representations.

Why the default vision encoder combination (DINOv2 + Wan VAE). DINOv2 provides spatial semantics (what objects are where), while Wan VAE provides spatiotemporal dynamics (how objects are moving). Together they cover both the static and dynamic aspects of future scenes. The paper validates this choice in Section 4.3, showing that DINOv2 alone underperforms the combination, and that different complementary encoders (SigLIP instead of Wan VAE) provide different strengths — SigLIP aids spatial precision while Wan VAE aids trajectory planning.

Why two-stage training rather than joint training. As discussed above, the two-stage design decouples the discovery of the condition space (Stage I, using ground-truth future observations) from learning to predict that space (Stage II, from current observations). This provides a cleaner optimization landscape than joint training, where the condition space and the VLM's predictions of it would co-evolve.

4. Key Insights and Innovations

Innovation 1: The Condition-Space Formulation Reframes World Modeling from "What Can We Predict?" to "What Does Action Generation Need?"

The dominant intellectual framing in world-action modeling has been to ask: given that we can extract rich visual features from future observations (via DINOv2, video generation models, depth estimators), which of these features should we predict to improve action generation? This framing starts from the supply side—from what the vision community has made available—and then attempts to select or compress from that supply to reduce redundancy. World action models (VPP, GeACT, DreamVLA) predict task-agnostic visual representations and struggle with redundancy; latent action models (LAPA, UniVLA) compress via reconstruction objectives and struggle with coarseness. Both inherit their predictive targets from properties of the input data rather than from the demands of the downstream task.

WoG reverses this logic entirely. The paper's key conceptual move is to ask not "what information does the future contain?" but rather "what information does the action head treat as causally relevant?"—and then to let the action prediction objective itself answer that question. The condition space Oc\boldsymbol{O}^c is not designed, selected, or hand-engineered; it is discovered by injecting future observations directly into the action inference pipeline and letting gradient descent determine what subset of those observations reduces the action prediction loss. This is a demand-side rather than supply-side approach to representation learning for world models.

This reframing matters because it changes what counts as a good predictive target from "comprehensive" to "sufficient." The condition space satisfies what the paper calls the sufficient condition criterion: its information content serves as a sufficient and effective condition for action generation—by construction, because it is the output of a module trained through exactly that objective. Any information in the future observations that does not help predict actions will be discarded by the Q-Former bottleneck; any information that does help will be retained. This is a more principled definition of what a world-action model should predict than ad-hoc selection of vision features or reconstruction-driven compression.

The significance extends beyond the specific architecture. The condition-space formulation provides a general recipe: if you want to improve a predictive model by giving it auxiliary future information during training, define the auxiliary target as the internal representation that the model itself learns when that future information is available. This is a form of self-distillation where the "teacher" is not a larger model but a different information source—in this case, actual future observations. The principle generalizes beyond robotics: any sequential decision-making problem where future states contain information relevant to current decisions could benefit from discovering a condition space rather than predicting raw future observations.

The empirical anchoring for this reframing is the comparison between WoG and DreamVLA (Tables 1, 2, 7). DreamVLA also uses DINOv2 as a predictive target but regresses the full, uncompressed feature maps. WoG consistently outperforms it across SIMPLER and real-world tasks because it predicts only the action-relevant subset of those features—the condition space—rather than the full representation. The ablation in Section 4.4 (Tables 5, 6) provides direct evidence: variants that predict uncompressed visual features ("WoG w/o Future Enc.") underperform the full WoG framework that predicts the queried, compressed conditions. This confirms that the discovery mechanism itself—not just having some future predictive target—is what drives the gains.

Innovation 2: The Two-Stage Curriculum as a Decoupling of Discovery from Internalization

Prior work that uses future information for action generation typically does so in one of two ways: (1) it keeps future information as an external input that is available at both training and test time (which is unrealistic for deployment), or (2) it trains the model end-to-end to predict some future representation and use that prediction for action generation simultaneously. The first approach is a non-starter for real-world deployment; the second creates a challenging joint optimization problem where the quality of the action head's use of predicted futures depends on the quality of the predictions, which in turn depends on the action head knowing what information is useful—a circular dependency.

WoG's two-stage design breaks this circularity with a clean conceptual decoupling. Stage I (World Guidance) answers the question "what should the model know about the future?" by giving it access to actual future observations and letting it discover what matters. This establishes an upper bound on what future information can contribute—the model learns to use ground-truth future conditions, so the discovered condition space represents the best possible use of future information for action generation. Stage II (World Inference) then answers a different question: "how can the model approximate that upper bound from current observations alone?" By freezing the Stage I encoder, Stage II creates a stable target space and trains the VLM backbone to predict into it. This is fundamentally a distillation setup, but the "teacher" is not a larger model—it is the same model augmented with privileged future information.

This decoupling is intellectually significant because it separates two problems that prior work conflated: the problem of finding a good predictive target (solved in Stage I with privileged information) and the problem of learning to predict that target (solved in Stage II with standard supervision). Joint training would require the predictive target and the prediction of it to co-evolve, creating a moving-target problem that is known to cause instability in representation learning. The two-stage design avoids this by establishing the target first and then learning to predict it—a pattern that appears in other areas of machine learning (e.g., training a teacher model on a larger dataset, then distilling it into a student) but that had not been applied to the world-modeling-for-action-generation problem.

The ablation in Table 8 ("WoG w/o cotrain") provides empirical evidence for why this decoupling matters. Removing the condition prediction supervision in Stage II—that is, training with the Stage I objective followed by standard action-only supervision—produces performance "comparable to the vanilla VLA" and "notably behind the full WoG." This demonstrates that the condition prediction objective in Stage II is not merely a regularization term; it is the mechanism by which future-awareness is transferred into the VLM backbone. The VLM does not spontaneously learn to anticipate future conditions just because the action head was once exposed to them; it needs explicit supervision to internalize that anticipatory capability.

The two-stage design also enables a cleaner analysis of what future information contributes. Because Stage I uses ground-truth future observations, its performance represents a ceiling: the best the model can do if it had perfect future knowledge. The gap between Stage I performance (with future conditions) and Stage II performance (with predicted conditions) quantifies how much information is lost due to imperfect prediction. This gap is a diagnostic tool that single-stage methods cannot provide—they have no way to separate the quality of the predictive target from the quality of the prediction.

Innovation 3: Future Condition Prediction as an Embodiment-Agnostic Bridge for Cross-Morphology Data Scaling

The field's approach to scaling robot learning has largely bifurcated along two paths. One path focuses on scaling up robot data across diverse embodiments (the Open X-Embodiment paradigm), using techniques like latent action models to compress heterogeneous action spaces into shared representations. The other path explores learning from human video data, but typically requires either action annotations (which are scarce) or task-specific alignment mechanisms (which don't scale). These paths have been largely separate because human videos lack the action labels that most robot learning methods require, and because the embodiment gap—differences in kinematics, viewpoints, and control interfaces—makes direct transfer challenging.

WoG's condition prediction objective creates a bridge between these two scaling paths through a mechanism that requires no action labels and is embodiment-agnostic by design. The key insight is that the condition space captures object-level dynamics rather than embodiment-specific kinematics. When a human picks up a cup, the condition vector Oc\boldsymbol{O}^c encodes the trajectory of the cup, the approach direction, and the timing of the grasp—dynamics that are equally relevant for a robot performing the same task, even though the human's joint angles and the robot's joint angles have no meaningful correspondence. Because the condition space was discovered through the action prediction objective on robot data, it automatically encodes the aspects of object motion that matter for manipulation, while being invariant to the specific embodiment that produced those motions.

This is a fundamental contribution to the data-scaling problem in robotics because it provides a prediction target that is simultaneously learnable from video alone, transferable across embodiments, and directly useful for action generation—a combination that neither world action models (which require expensive video prediction and are brittle to visual domain shifts) nor latent action models (which require action labels) achieve. The paper demonstrates two instantiations of this capability:

First, unannotated human videos (the "w. human v." variant in Table 9) improve performance and OOD generalization on rigid-body tasks solely through condition prediction supervision in Stage II—no action labels are used. This is possible because the condition space, once established in Stage I from robot data, provides a target that human videos can help the VLM backbone learn to predict better, and those improved predictions translate to better actions because the condition space is action-relevant.

Second, UMI data from a completely different embodiment and viewpoint (Figure 4, Section 5.5) provides a 42% improvement on pick-and-place and a 33% improvement on folding when added during fine-tuning—even though the condition space was pretrained exclusively on standard robot data. This demonstrates that the condition space generalizes across substantial domain gaps (egocentric viewpoint, different kinematics, different action representations) without any modification, because it encodes object-level rather than robot-level dynamics.

The significance of this innovation extends beyond the empirical gains. It suggests a path toward scaling robot learning that is not bottlenecked by the availability of robot-specific demonstration data. If future condition prediction can absorb knowledge from large-scale human video corpora (and the paper demonstrates this with 1,920 hours of human manipulation video), then the data efficiency of robot learning could be transformed—the model learns object dynamics from abundant human video and only needs robot data to learn the embodiment-specific mapping from anticipated dynamics to joint-level actions. This is analogous to how large language models benefit from pretraining on unlabeled text corpora before fine-tuning on task-specific labeled data, but adapted to the embodied setting where the "unlabeled" data source is human video and the "task" is robot action generation.

Innovation 4: Diagnostic Evidence That the Condition Space Amplifies Encoder Strengths Without Compensating for Their Deficiencies

Most method papers present their approach as uniformly beneficial and leave limitations implicit. WoG does something more intellectually honest and analytically valuable: it provides concrete evidence that the condition space magnifies the inherent strengths and weaknesses of the underlying vision encoders rather than compensating for their deficiencies. This is a diagnostic finding rather than a performance claim, and it has significant implications for how future work should approach encoder selection in world-action models.

The evidence comes from the encoder configuration experiments in Section 4.3 (Tables 3, 4). When DINOv2 is paired with Wan VAE (the "dino-vae" configuration), the model excels at trajectory planning tasks (Pick Coke, Move Near) because the VAE's spatiotemporal compression capability aids in modeling object dynamics. When DINOv2 is paired with SigLIP (the "dino-siglip" configuration), the model excels at spatial precision tasks (Stack Green on Yellow: 33.0% vs. 29.2% for dino-vae) because SigLIP's high-level semantic alignment aids in fine-grained spatial reasoning. The condition space does not create these capabilities; it selectively extracts and amplifies what each encoder already provides. The paper is explicit about this limitation: "WoG effectively magnifies the inherent strengths of foundation models for action generation, [but] it cannot inject the additional semantic information required to compensate for their intrinsic deficiencies in specific capabilities" (Section 4.4).

This finding has two important consequences. First, it reframes encoder selection as a first-class design decision in world-action models. Prior work using future prediction (DreamVLA, VPP) treated the choice of visual encoder as an implementation detail—use whatever provides rich features. WoG's analysis shows that the choice of encoder directly determines which manipulation capabilities the model will excel at, and that no single encoder configuration is uniformly optimal. This implies that future systems should either use ensemble encoders (as WoG does with DINOv2 + Wan VAE/SigLIP) or develop encoder selection strategies based on the anticipated task distribution.

Second, it clarifies the division of labor between world modeling and spatial reasoning. The paper's results suggest that future prediction—even when done in an optimized condition space—helps primarily with dynamic reasoning (trajectory planning, obstacle avoidance, grasp timing) but does not substantially improve static spatial reasoning (precise relative positioning, alignment, stacking). This is because future observations contain information about how objects move but limited information about sub-centimeter spatial relationships that require high-resolution geometric understanding. The paper's acknowledgment that "modeling fine-grained spatial constraints remains a persistent challenge independent of the future encoder" and "requires dedicated spatial mechanisms or historical observation modeling, which lies beyond the scope of this work" (Section 4.3) is a precise and honest boundary statement that prevents overclaiming and guides future research toward the complementary capabilities that world modeling cannot provide.

This diagnostic framing—showing what the method amplifies versus what it cannot compensate for—is methodologically significant. It transforms a performance table into an analytical tool and provides actionable guidance for practitioners: if your task is dominated by trajectory planning and obstacle avoidance, prioritize world modeling with spatiotemporal encoders; if your task requires precise spatial alignment, invest in dedicated geometric reasoning modules rather than expecting world models to solve it.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. Simulation experiments use the SIMPLER benchmark, which serves as a real-to-sim evaluation proxy for the Bridge and Fractal datasets from the Open X-Embodiment (OXE) collection. Real-world experiments use a self-collected dataset of expert demonstrations: 100 trajectories for Pick and Place, 100 for Close the Microwave, and 200 for Fold the Towel. Pretraining for all experiments uses the OXE dataset with sampling ratios specified in Table 10 (Appendix 8.1). For human data experiments, a self-collected corpus of 1,920 hours of human manipulation video (650k trajectories) is used, with a 220-hour subset carrying action annotations.

  • Base model(s). All experiments use a Prismatic VLM backbone adopted from OpenVLA's architecture, paired with a DiT (Diffusion Transformer) action head. The Prismatic VLM encodes the current RGB observation and language instruction; the DiT head generates action sequences of length T = 16 via rectified flow denoising. The choice of this architecture is motivated by its established use in OpenVLA, making it a representative VLA baseline. The model's pass@1 on MATH is not directly comparable (this is a manipulation benchmark), but the model's performance on SIMPLER and real-world tasks establishes it as a capable but not saturating base.

  • Metrics. The primary metric across all experiments is task success rate (%) — the fraction of evaluation trials where the robot successfully completes the specified task. For SIMPLER, evaluation follows the protocols defined by the benchmark: Visual Matching (minimizing sim-to-real visual discrepancy) and Variant Aggregation (introducing distribution shifts in backgrounds, lighting, distractors, and table textures). For real-world experiments, success is defined per task: for Pick and Place, successfully placing the cup in the plate; for Close the Microwave, fully closing the door; for Fold the Towel, bringing the two bottom corners within 5 cm of each other. Each real-world method is evaluated over 20 trials per task with closely matched randomized initial scene configurations.

  • Baselines. The paper includes a broad set of baselines spanning multiple paradigms. For simulation (Tables 1, 2): (i) Conventional VLAs — π₀ [pi0], π₀-FAST [fast], OpenVLA [openvla], and GR00T-N1 [gr00t]; (ii) Latent Action Models — Moto [moto] and UniVLA [univla]; (iii) World Action Models with video prediction — DeFI [defi]; (iv) Methods combining latent action modeling with future video generation — VITA [vita] and ViPRA [vipra]. For real-world experiments (Table 7): UniVLA [univla] (as a representative Latent Action Model) and VPP [vpp] (as a representative method leveraging future video prediction), both of which have been widely validated in real-world robotic settings. For ablation experiments (Table 8): a vanilla VLA (same VLM backbone and DiT head, supervised on actions from current observations only) and WoG without co-training (Stage I training followed by Stage II with action supervision only, no condition prediction loss). All methods use OXE pretrained checkpoints where available and are fine-tuned on the same expert demonstrations.

  • Generation budget / compute accounting. Compute is measured in training steps (not generations, since this is not a sampling-based method). For simulation, pretraining runs for 100k steps in Stage I and 50k steps in Stage II, both with a global batch size of 1024. For real-world experiments, pretraining uses 100k steps (Stage I) and 50k steps (Stage II) on OXE, followed by 30 epochs of fine-tuning on expert demonstrations with batch size 512. All baselines are aligned to the same total training budget (e.g., the vanilla VLA is trained for 150k total pretraining steps, matching the combined 100k + 50k of WoG). During fine-tuning, WoG mirrors only the Stage II protocol with the co-training objective. This is a fair comparison framework: all models receive equivalent total optimization steps on equivalent data.

  • Cross-validation / statistical protocol. No explicit cross-validation is reported. For real-world experiments, 20 trials are performed per method per task with closely matched randomized initial scene configurations, providing a reasonable estimate of performance variance. For SIMPLER, the benchmark defines standardized evaluation protocols (Visual Matching and Variant Aggregation) that control for environmental variation. The paper does not report confidence intervals or statistical significance tests. The small number of real-world trials (20 per task per method) means observed success rates could have substantial binomial variance (e.g., a rate of 60% from 20 trials has a 95% confidence interval of approximately ±21 percentage points).

Main Quantitative Results

Simulation: Comparison Against Baselines

The headline result is that WoG achieves strong and consistent improvements over all baselines across the majority of SIMPLER tasks. On Google Robot tasks (Table 1), WoG achieves the highest success rate on 4 of 6 tasks, with particularly notable gains on tasks requiring trajectory planning under dynamic interference. For example, on Move Near — a task where the robot must navigate near interfering objects — WoG achieves 78.0% success, substantially outperforming OpenVLA (52.0%), Moto (40.0%), and VITA (60.0%). On Pick Coke Can, a canonical pick-and-place requiring accurate grasp prediction, WoG reaches 86.0% compared to OpenVLA's 70.0%. The only tasks where WoG does not lead are Open/Close Drawer (where π₀ achieves 94.0% vs. WoG's 82.0%) and Stack Green on Yellow (where ViPRA reaches 48.0% vs. WoG's 34.0%). The paper attributes the smaller gains on stacking and drawer tasks to "the limited spatial resolution of the current backbone and the inherent difficulty of modeling fine-grained geometry through current dynamic prediction alone" (Section 4.2).

On WidowX Robot tasks (Table 2), WoG achieves the highest success rate on all 4 tasks. On Put Eggplant in Pot, WoG reaches 64.0% vs. UniVLA's 56.0%. On Pick Coke Can, WoG reaches 80.0% vs. GR00T-N1's 24.0% — a 56 percentage point gap. Notably, WoG outperforms methods that jointly model video prediction and latent actions (VITA, ViPRA) across all 4 tasks, with gaps ranging from 2 percentage points (Put Eggplant: VITA 62.0% vs. WoG 64.0%) to 30 percentage points (Pick Coke Can: VITA 50.0% vs. WoG 80.0%). This supports the paper's claim that extracting critical semantics into a condition space is more effective than full-scale video prediction, which can propagate visual prediction errors into the action space.

The paper's explanation for why WoG excels on trajectory-dependent tasks (Move Near, Pick Coke) while showing smaller advantages on precision-dependent tasks (Stack, Drawer) is substantiated by the encoder configuration experiments (Tables 3, 4), which show that different visual encoders confer different strengths. This pattern is consistent with the paper's stated limitation: the condition space amplifies encoder capabilities without compensating for intrinsic deficiencies in fine-grained geometry.

Simulation: Encoder Configuration Analysis

Tables 3 and 4 evaluate three encoder combinations: DINOv2 only, DINOv2 + SigLIP ("dino-siglip"), and DINOv2 + Wan VAE ("dino-vae"). The key findings:

On Google Robot tasks (Table 3), the dino-vae configuration achieves the highest overall average success rate (70.9%), driven by strong performance on Pick Coke (88.0%) and Move Near (82.0%). This supports the claim that the VAE encoder's spatiotemporal compression aids trajectory planning and collision avoidance.

On WidowX Robot tasks (Table 4), the dino-siglip configuration achieves the highest overall average (64.0%), led by Put Eggplant (76.0%) and Stack Green on Yellow (33.0% vs. dino-vae's 29.2%). The paper interprets this as evidence that SigLIP's explicit high-level semantic alignment is critical for tasks with spatial constraints and precise positioning requirements.

The DINOv2-only baseline underperforms both augmented configurations across nearly all tasks, confirming that enriching the condition space with complementary high-level semantics or temporal dynamics significantly enhances policy robustness. The paper selects dino-siglip as the default for simulation (noting it "partially mitgates spatial precision deficits and ensures robust performance across the SIMPLER benchmark") while retaining the VAE-incorporated framework for real-world experiments to exploit its superior trajectory planning capacity.

Simulation: Future Encoder Ablation

Tables 5 and 6 test the necessity of the Q-Former Future Encoder by comparing three variants: (1) "WoG w/o Future Enc." — no Q-Former at all; learnable embeddings align directly to full uncompressed feature maps; (2) "WoG w/o Future Enc. in Stage-II" — Q-Former used in Stage I but discarded in Stage II; VLM aligns to uncompressed features; (3) "WoG w. Future Enc." — the standard WoG with Q-Former used in both stages and VLM aligning to compressed conditions.

On Google Robot tasks (Table 5), the standard WoG achieves the highest overall average (70.9%) and leads on 4 of 6 tasks. The "WoG w/o Future Enc. in Stage-II" variant achieves 66.7% average, and the "WoG w/o Future Enc." variant achieves 63.5% — a 7.4 percentage point gap from full WoG. The advantage is particularly pronounced in grasping and picking phases: on Pick Coke, standard WoG reaches 88.0% vs. 78.0% for "WoG w/o Future Enc." vs. 74.0% for "WoG w/o Future Enc. in Stage-II." On WidowX tasks (Table 6), standard WoG averages 58.0% vs. 47.3% for "WoG w/o Future Enc." — a 10.7 percentage point gap.

The paper interprets these results as evidence that "the condition extraction mechanism effectively leverages the potential of the underlying vision foundation models and keeps generalizable during various setups, by amplifying the trajectory planning capabilities inherent to the dino-vae configuration during action execution." However, the method does not yield a distinct advantage on spatially sensitive placement tasks like Stack, which the paper acknowledges as "a limitation of the current paradigm."

Real-World: Baseline Comparison

Table 7 reports success rates for WoG, UniVLA, and VPP across 3 tasks under in-distribution (ID) and out-of-distribution (OOD) conditions (Background Change, Light Change, Novel Object).

Under ID settings, WoG achieves 90% success on P&P (vs. UniVLA 55%, VPP 65%), 100% on Close the Microwave (vs. UniVLA 70%, VPP 80%), and 85% on Fold the Towel (vs. UniVLA 55%, VPP 65%). The gaps are substantial: on the deformable manipulation task (Fold), WoG's 85% represents a 30 percentage point improvement over UniVLA and a 20 percentage point advantage over VPP. The paper attributes this to the condition space "effectively distilling manipulation-relevant dynamics (e.g., cloth deformation) while discarding the redundant perceptual signals inherent in video generation."

Under OOD settings, WoG shows minimal degradation. On P&P: 90% (ID) → 80% (Background), 70% (Light), 80% (Novel Object). On Fold: 85% → 75% (Background), 75% (Light), 75% (Novel Object). On Close: 100% → 90% (Background), 80% (Light). In contrast, UniVLA drops from 55% to 25–30% on P&P OOD, and VPP drops from 65% to 35–50% on P&P OOD. The most challenging shift is Light Change, where UniVLA achieves only 25% on P&P (a 30-point drop) while WoG achieves 70% (a 20-point drop). The paper attributes WoG's robustness to "the design of the condition space: by querying and compressing informative features from frozen, pretrained visual encoders, WoG constructs conditions that are highly distinctive for manipulation yet invariant to visual nuisances."

Real-World: Ablation of Training Stages

Table 8 ablates the two-stage training design by comparing vanilla VLA (action supervision only, no future information), WoG without co-training (Stage I future-guided training followed by Stage II with action loss only, no condition prediction loss), and full WoG.

Under ID settings, full WoG substantially outperforms both variants: on P&P, 90% vs. 65% (vanilla) vs. 60% (w/o cotrain); on Fold, 85% vs. 65% (vanilla) vs. 65% (w/o cotrain); on Close, 100% vs. 80% (vanilla) vs. 80% (w/o cotrain). The vanilla VLA and WoG w/o cotrain perform comparably, suggesting that "introducing future conditions does not significantly degrade the action generation capability of the VLM backbone" but that without the condition prediction loss, the future-awareness from Stage I does not transfer to the VLM.

Under OOD settings, full WoG maintains its advantage: on P&P under Light Change, 70% vs. 40% (vanilla) vs. 35% (w/o cotrain). The paper concludes that "explicitly supervising the alignment between future conditions and the VLM backbone is crucial for distilling future action-relevant knowledge into the VLA model."

Real-World: Human Data Integration

Table 9 reports results for two human data strategies. "w. human v." uses unannotated human videos only in Stage II for condition prediction supervision. "w. human v./a." additionally incorporates 220 hours of action-annotated human videos for action supervision in both stages.

With unannotated human videos only ("w. human v."), the model shows mixed results: on P&P, ID success improves from 90% (robot-only) to 95%, and OOD drops are slightly smaller (Background: 85% vs. 80%; Light: 75% vs. 70%). However, on Fold, performance degrades from 85% to 80% ID, with larger OOD drops (Background: 70% vs. 75%; Novel Object: 70% vs. 75%). The paper attributes this to "task-dependent similarity between human and robotic manipulation": for P&P, human and robot behaviors are similar; for deformable manipulation, the mismatch in the condition space limits transferability.

With annotated human videos ("w. human v./a."), the model consistently outperforms robot-only training across all settings: P&P ID improves from 90% to 100%; Fold ID from 85% to 95%; Close from 100% to 100% (no change, already saturated); OOD performance improves across all shifts. The paper concludes that "even with only 220h of action-annotated human data introduced in the first training stage, WoG is able to rapidly acquire human-aligned conditioning representations and effectively transfer them to robotic manipulation."

Real-World: UMI Data Integration

Figure 4 reports success rates when 120 additional UMI trajectories are introduced during fine-tuning. On P&P, success increases from 60% (robot-only) to 85% (with UMI) — a 42% relative improvement. On Fold, success increases from 60% to 80% — a 33% relative improvement. The paper notes that this UMI data was introduced exclusively during fine-tuning alongside expert demonstrations, with the condition space established during OXE pretraining remaining unmodified. Despite significant domain gaps (egocentric observations, different action representations, different embodiment), the model achieves substantial gains, which the paper attributes to "the model's proficiency in capturing embodiment-agnostic dynamics, such as intrinsic object motion."

Ablation Studies and Robustness Checks

  • Pretrained Encoder Configuration (Tables 3, 4): Three configurations are compared: DINOv2 only, DINOv2 + SigLIP, and DINOv2 + Wan VAE. Both augmented configurations consistently outperform DINOv2 alone, confirming that enriching the condition space with complementary features improves robustness. The VAE configuration excels on trajectory planning (Google Robot average: 70.9%); the SigLIP configuration excels on spatial precision (WidowX Stack: 33.0% vs. 29.2%). This ablation reveals that encoder choice directly determines which manipulation capabilities the condition space amplifies — a finding with practical implications for task-specific encoder selection.

  • Future Encoder Necessity (Tables 5, 6): Comparing standard WoG against variants that predict uncompressed visual features (with or without Q-Former in Stage II). The full WoG with Q-Former compression achieves 70.9% average on Google Robot tasks vs. 63.5% for the variant with no Future Encoder at all — a 7.4 percentage point gap. This directly validates the paper's central claim that compressing through the Q-Former bottleneck produces more effective conditions than directly predicting full feature maps. The intermediate variant (Q-Former in Stage I, removed in Stage II) underperforms (66.7%), confirming that both stages benefit from the compression.

  • Training Stage Ablation (Table 8): Removing the condition prediction co-training objective in Stage II (WoG w/o cotrain) reduces performance to levels comparable with the vanilla VLA that never uses future information at all. On P&P ID, vanilla VLA achieves 65%, WoG w/o cotrain achieves 60%, full WoG achieves 90%. This is a critical finding: exposure to future conditions in Stage I alone does not confer future-awareness; the explicit condition prediction objective in Stage II is the mechanism by which that awareness is internalized into the VLM backbone.

  • Human Data: Annotation Requirements (Table 9): The comparison between "w. human v." (no action annotations) and "w. human v./a." (11% annotated) reveals that unannotated human videos alone provide limited and task-dependent benefits (improvement on P&P, degradation on Fold), while even a small amount of annotated human data enables consistent and substantial gains across all tasks and OOD conditions. This is a nuanced finding: the condition prediction bridge works best when the condition space has been at least partially calibrated to human manipulation kinematics through action supervision.

  • Human Data: Task-Dependent Transfer (Table 9): The degradation on Fold the Towel when using unannotated human videos (85% → 80% ID, with larger OOD drops) is a notable negative result. The paper interprets it as evidence that deformable manipulation has a larger human-robot embodiment gap in the condition space, such that forcing the VLM to predict human-derived conditions on such tasks introduces noise rather than signal. This is an honest empirical boundary on the method's human-to-robot transfer capability.

  • UMI Data: Embodiment-Agnostic Transfer (Figure 4): The 42% improvement on P&P and 33% on Fold when adding UMI data during fine-tuning — despite the condition space being pretrained exclusively on standard robot data — serves as a robustness check that the condition space genuinely captures embodiment-agnostic dynamics. If the condition space were overfit to the pretraining embodiment, UMI data would not provide gains (and might hurt). The observed improvements validate the design choice to target object-level rather than robot-level dynamics.

Critical Assessment

Claim 1: "WoG achieves strong and consistent performance improvements over all baselines across the majority of tasks."

This claim is supported for the specific baselines tested, but the framing requires qualification. On simulation (Tables 1, 2), WoG leads on 8 of 10 tasks but does not dominate uniformly: on Google Robot's Open/Close Drawer, π₀ achieves 94.0% vs. WoG's 82.0%, and on Stack Green on Yellow, ViPRA reaches 48.0% vs. WoG's 34.0%. The paper honestly acknowledges these as tasks requiring fine-grained spatial geometry that the condition space does not address. The "consistent performance improvements" claim thus holds specifically for tasks where trajectory planning and dynamics modeling dominate over static spatial precision — a boundary condition that the paper identifies but does not prominently feature.

On real-world experiments (Table 7), the baselines are narrower: only UniVLA and VPP are compared. Missing are comparisons against DreamVLA (the closest prior method, predicting DINOv2 features), OpenVLA fine-tuned on the same expert data, or stronger world-action models like GeACT. The paper's simulation experiments (Table 1) show that DreamVLA and OpenVLA achieve competitive or better performance on some tasks (e.g., OpenVLA 70.0% on Pick Coke vs. WoG's 86.0%, but OpenVLA 52.0% on Move Near vs. WoG's 78.0%). The absence of these baselines from the real-world evaluation weakens the completeness of the comparison. The claim that WoG "significantly outperforms existing methods based on future prediction" (Abstract) is demonstrated against VPP (which uses video generation features) and DreamVLA (in simulation), but a direct real-world comparison against DreamVLA would have been more convincing.

Claim 2: "The two-stage training curriculum effectively transfers future-awareness into the VLM backbone."

The ablation in Table 8 provides the key evidence: removing the Stage II condition prediction loss collapses performance to vanilla VLA levels. This demonstrates that the condition prediction objective is necessary, but does not fully establish sufficiency of the specific two-stage design. A missing ablation is the end-to-end alternative: train everything jointly from the start with the VLM predicting conditions from current observations, using those predicted conditions in the action head, and supervising both. The paper argues (implicitly) that the two-stage design avoids a circular dependency, but no experiment demonstrates that joint training would underperform. This is a genuine gap: the paper shows that the two-stage design works and that removing Stage II's condition loss breaks it, but does not show that the two-stage design is better than single-stage alternatives.

Additionally, Table 8 shows that "WoG w/o cotrain" (Stage I with future guidance, Stage II with action-only supervision) performs comparably to the vanilla VLA — meaning that Stage I alone, without the condition prediction objective in Stage II, does not confer any lasting benefit. This is a strong negative result that the paper interprets as evidence that "explicitly supervising the alignment between future conditions and the VLM backbone is crucial." However, it also raises a question: does Stage I provide any benefit whatsoever if not followed by the co-training objective? The fact that WoG w/o cotrain is comparable to (and on P&P slightly worse than) the vanilla VLA suggests that Stage I's effect is entirely mediated through Stage II. This implies that Stage I's sole function is to define the condition space — it does not independently improve the VLM's representations. This is an honest but under-emphasized finding: the "guidance" in Stage I is not itself beneficial unless the model is subsequently taught to predict that guidance.

Claim 3: "Modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities."

The generalization evidence (Tables 7, 9) is the strongest part of the paper. WoG shows substantially smaller drops under OOD conditions than baselines. On P&P under Light Change, WoG drops from 90% to 70% (a 20-point gap), while UniVLA drops from 55% to 25% (a 30-point gap) and VPP drops from 65% to 35% (a 30-point gap). This pattern holds across tasks and OOD types.

However, there are important caveats. First, the OOD evaluation uses 20 trials per condition, producing success rates with wide confidence intervals. A 70% rate from 20 trials has a 95% binomial confidence interval of approximately 46–88%, meaning the apparent 20-point advantage over VPP (70% vs. 35%) could in reality be substantially smaller or larger. The paper does not report confidence intervals or statistical tests, making it difficult to assess whether the OOD generalization advantages are statistically reliable at this sample size.

Second, the OOD conditions tested (background change, light change, novel object) represent visual distribution shifts — precisely the type of variation that frozen pretrained visual encoders (DINOv2, Wan VAE) are designed to be robust to. It is therefore not surprising that a method that queries and compresses frozen encoder features would generalize better than methods that fine-tune visual representations (potentially overfitting to training appearances). The generalization advantage may be more attributable to the use of frozen encoders than to the condition-space formulation per se. An ablation comparing WoG against a variant that uses frozen encoders but predicts full feature maps (like DreamVLA) under OOD conditions would disentangle these factors, but the paper does not report this.

Claim 4: "WoG can be effectively improved by learning to model and predict future conditions from large-scale human videos."

The evidence for this claim is mixed and task-dependent (Table 9). Unannotated human videos provide a modest benefit on P&P (90% → 95% ID) but degrade performance on Fold (85% → 80% ID). The paper's explanation — that deformable manipulation has a larger human-robot embodiment gap in the condition space — is plausible but post-hoc. The experiment demonstrates that unannotated human video integration is not uniformly beneficial and can hurt performance on certain task types, which is an important boundary condition that the paper's Abstract does not convey.

Annotated human videos (220 hours) do provide consistent gains (P&P ID: 90% → 100%; Fold ID: 85% → 95%; improved OOD across the board), demonstrating that the framework can absorb human data effectively when action labels are available. However, this setting requires action-annotated human data, which undermines the scalability argument — collecting 220 hours of annotated human manipulation is itself a substantial data collection effort, even if faster than robot teleoperation.

The UMI results (Figure 4) provide stronger evidence: 120 additional UMI trajectories produce a 42% improvement on P&P and 33% on Fold. This demonstrates cross-embodiment transfer from egocentric data, which is genuinely impressive. However, UMI data is still robot data (collected via a specific interface), not human video. The claim about human video scalability is therefore supported for annotated human data and for robot data from different embodiments, but the claim about unannotated human video — the more scalable data source — is only partially supported and comes with documented failure modes.

Claim 5: "Our method significantly outperforms existing methods based on future prediction."

This claim holds for the specific baselines compared. On simulation, WoG outperforms VITA and ViPRA (methods that combine latent actions with video prediction) on all WidowX tasks (Table 2) and on the majority of Google Robot tasks (Table 1). Against DreamVLA (a method predicting DINOv2 features), WoG also consistently outperforms. On real-world tasks, WoG outperforms VPP (which uses video generation features).

However, the phrase "existing methods based on future prediction" encompasses a broader category than tested. The paper does not compare against methods that predict explicit future modalities like depth or optical flow (FlowVLA, DreamVLA variants with depth), nor against methods that use world models for planning through imagined rollouts (UniPi-style approaches). The claim is accurate for the specific set of baselines evaluated but overstates the comprehensiveness of the comparison.

Overall Assessment

The experimental design has genuine strengths: multi-benchmark evaluation (SIMPLER + real-world), multi-task coverage (rigid, articulated, deformable objects), OOD generalization testing, and careful ablation of both the encoder configuration and the training stages. The two-stage training ablation (Table 8) is particularly well-designed, isolating the contribution of the condition prediction objective.

The most significant weaknesses are: (1) the small number of real-world trials (20 per condition) without reported confidence intervals, making it difficult to assess whether reported differences are statistically reliable; (2) the narrow set of real-world baselines (UniVLA and VPP only, missing DreamVLA and other world-action models); (3) the absence of an end-to-end joint training baseline to validate the two-stage design against single-stage alternatives; (4) the task-dependent and sometimes negative results for unannotated human video integration, which the Abstract and Introduction present more optimistically than the data supports; and (5) the reliance on a single VLM backbone (Prismatic/OpenVLA) without testing whether the condition-space formulation generalizes across backbone architectures.

Experiments that would have strengthened the paper include: (1) an end-to-end joint training ablation; (2) real-world comparison against DreamVLA and OpenVLA fine-tuned on the same expert data; (3) evaluation on additional robot platforms beyond UR5 to test embodiment generalization claims; (4) ablating the condition dimension (32) to understand the expressiveness-efficiency trade-off; and (5) scaling experiments varying the amount of OXE pretraining data to characterize data efficiency gains from the condition prediction objective.

6. Limitations and Trade-offs

The Condition Space Amplifies Encoder Strengths Without Compensating for Their Deficiencies

The assumption or constraint. The paper explicitly acknowledges that the condition space does not create new capabilities but selectively extracts and amplifies what the underlying frozen vision encoders already capture. Section 4.4 states: "WoG effectively magnifies the inherent strengths of foundation models for action generation, [but] it cannot inject the additional semantic information required to compensate for their intrinsic deficiencies in specific capabilities." The corollary, stated in Section 4.3, is that "modeling fine-grained spatial constraints remains a persistent challenge independent of the future encoder. Solving this requires dedicated spatial mechanisms or historical observation modeling, which lies beyond the scope of this work."

The consequence. This means WoG offers no path to improvement on tasks dominated by precise spatial reasoning — exact relative positioning, sub-centimeter alignment, fine-grained geometric understanding. The empirical evidence bears this out: on Stack Green on Yellow, WoG achieves 34.0% (Google Robot, Table 1) and 33.0% (WidowX, Table 4), substantially underperforming ViPRA's 48.0% on the same Google Robot task. On Open/Close Drawer, WoG's 82.0% trails π₀'s 94.0% (Table 1). The encoder configuration ablation (Tables 3, 4) shows that switching from Wan VAE to SigLIP improves spatial precision (Stack: 29.2% → 33.0%) but worsens trajectory planning (Pick Coke: 88.0% → 84.0%), confirming that the condition space inherits a trade-off between dynamic and spatial reasoning that no encoder configuration fully resolves. A practitioner deploying WoG for a task requiring precise assembly, insertion, or alignment should expect diminishing returns — the method will help with trajectory planning but cannot overcome the VLM backbone's fundamental spatial resolution limits.

Mitigation status. Not addressed. The paper flags this as future work requiring "dedicated spatial mechanisms or historical observation modeling" but provides no concrete path. A practitioner would need to augment WoG with an orthogonal spatial reasoning module — the paper does not offer one.

The Two-Stage Design Lacks a Direct Comparison Against End-to-End Joint Training

The assumption or constraint. The paper's core architectural contribution is the two-stage curriculum: Stage I discovers the condition space using ground-truth future observations, then Stage II freezes that space and teaches the VLM to predict it. The paper argues implicitly that joint end-to-end training (where the VLM predicts conditions from current observations, feeds them into the action head, and everything is trained jointly) would suffer from a circular dependency — the action head needs good conditions to learn to use them, but the VLM can only produce good conditions if the condition space is meaningful. However, this argument is never tested empirically.

The consequence. Without this ablation, the paper cannot distinguish between two competing claims: (a) the two-stage design is necessary for the observed gains, or (b) the condition-space formulation is what matters and the two-stage design is merely one way to implement it. If claim (b) is true, then end-to-end joint training might achieve comparable performance with a simpler training pipeline, removing the need to coordinate two sequential training phases. This matters practically because two-stage training is operationally more complex: it requires checkpoint management between stages, careful scheduling of when to freeze the Q-Former, and twice the hyperparameter tuning surface. If a single-stage variant works equally well, practitioners could adopt the condition-space idea without the two-stage overhead.

The paper does provide evidence that Stage II's condition prediction loss is essential (Table 8: removing it collapses performance to vanilla VLA levels). But this only shows that predicting conditions helps — it does not show that predicting them after they've been defined by a separate stage is better than predicting them while they're being defined jointly. A joint training baseline would have tested whether the condition space discovered through the action prediction loss alone (without privileged future access) is already sufficient.

Mitigation status. Not addressed. The paper provides no joint training ablation, and the architectural description in Section 3 gives no indication that a single-stage variant was ever tested. This is a genuine gap in the experimental design.

Unannotated Human Video Integration Is Task-Dependent and Can Degrade Performance

The assumption or constraint. A central claim of the paper is that WoG can leverage large-scale human manipulation videos without action annotations by using the condition prediction objective as an embodiment-agnostic bridge (Section 3.4). The implicit assumption is that the condition space, discovered in Stage I on robot data, captures object-level dynamics shared across human and robot manipulation. The paper acknowledges a boundary condition in Section 5.4: "the task-dependent similarity between human and robotic manipulation."

The consequence. The empirical results (Table 9, "w. human v." variant) reveal that this assumption holds only partially. On Pick and Place, unannotated human videos improve ID success from 90% to 95% and modestly reduce OOD degradation. However, on Fold the Towel — a deformable manipulation task — unannotated human videos degrade performance from 85% to 80% ID, with larger OOD drops. The paper attributes this to a larger human-robot embodiment gap in deformable manipulation: human towel-folding involves flexible hand postures, variable contact points, and adaptive force application that may induce condition-space representations poorly aligned with the robot's rigid gripper dynamics. When the VLM backbone is forced to predict these human-derived conditions during Stage II, it learns condition patterns that do not correspond to successful robot actions, introducing noise into the joint training objective.

This is consequential because the scalability promise of the framework rests heavily on unannotated video — 1,920 hours were collected, but only 11% (220 hours) carry action annotations. If unannotated video helps on some tasks and hurts on others, practitioners cannot simply dump large video corpora into training and expect uniform improvement. They would need per-task validation of whether human-robot condition alignment holds, which defeats the purpose of scalable, unsupervised pretraining.

What evidence exists. Table 9 directly documents the degradation. On Fold: 85% → 80% ID, 75% → 70% Background, 75% → 70% Novel Object. The paper explicitly attributes this to task-dependent embodiment similarity (Section 5.4), making this a self-acknowledged limitation.

Mitigation status. Partially addressed through the annotated human data variant ("w. human v./a."), which adds 220 hours of action-annotated human videos and resolves the degradation (Fold: 85% → 95% ID, OOD improves across the board). However, annotated human data is substantially more expensive to collect and represents only 11% of the total corpus. The paper does not address whether there exists a middle ground — e.g., using a small amount of annotated human data specifically for deformable tasks to calibrate the condition space, while using unannotated video for rigid-body tasks. This calibration strategy is not explored.

Real-World Evaluation Sample Sizes Are Insufficient for Reliable Comparison Against Baselines, and Key Baselines Are Missing

The assumption or constraint. The real-world evaluation uses 20 trials per method per task (Section 5.1, "Protocols"). The paper does not report confidence intervals or statistical significance tests. For a binomial success rate estimated from 20 trials, the 95% Clopper-Pearson confidence interval is approximately ±22 percentage points at rates near 50% and approximately ±13 percentage points at rates near 90%.

The consequence. Several of the paper's headline comparisons fall within overlapping confidence intervals. Consider the OOD evaluation on P&P under Light Change (Table 7): WoG achieves 70%, VPP achieves 35%. With 20 trials each, the difference appears large (35 percentage points), but the confidence intervals are approximately [46%, 88%] for WoG and [15%, 59%] for VPP — they overlap. The gap is directionally clear but the magnitude is uncertain. On Close the Microwave under Light Change, WoG achieves 80%, VPP achieves 55% — intervals of approximately [56%, 94%] and [32%, 77%] again overlap. This does not mean the claims are wrong, but it means the evidence is weaker than the reported point estimates suggest. A practitioner deciding between WoG and VPP based on this data would be making a decision with considerable statistical uncertainty.

Additionally, the real-world baseline set (Table 7) is narrow: only UniVLA (a latent action model) and VPP (a video prediction method). Missing are DreamVLA (the closest prior method, predicting DINOv2 features, compared in simulation but not real-world), OpenVLA (the VLA backbone WoG is built on, compared in simulation), and stronger world-action models like GeACT. The absence of DreamVLA is particularly notable because it shares WoG's use of DINOv2 as a predictive target but predicts the full feature map rather than a queried condition — a direct comparison would isolate the contribution of the Q-Former compression. The simulation results (Table 1) show DreamVLA performing competitively on some tasks (e.g., Open/Close Drawer: 88.0% DreamVLA vs. 82.0% WoG), so its real-world performance is not predictable from simulation alone.

Mitigation status. Not addressed. The paper does not discuss statistical power, does not report confidence intervals, and does not justify the choice of 20 trials or the baseline selection. Simulation experiments use larger evaluation sets (SIMPLER's standardized protocols) and provide more robust comparisons, but the real-world evidence — which carries greater weight for practical adoption — rests on a weaker statistical foundation.

The Difficulty Estimation Cost for Stage I Is Not Accounted for in the Training Budget Comparison

The assumption or constraint. Stage I requires access to ground-truth future observations — the system must have the actual RGB frames from the next T time steps to extract DINOv2 and Wan VAE features and compute the target conditions O^c. This is available during offline training from recorded trajectories but imposes a data requirement: every training sample must include a multi-frame future observation window. The paper does not discuss whether this data is always available, how the temporal sampling (4 frames over 16 action steps) interacts with varying control frequencies across datasets, or whether the need for future frames constrains which OXE subsets can be used.

The consequence. This is not a test-time limitation (Stage II removes the need for future observations), but it is a training-time constraint that affects data compatibility. Many robot datasets in OXE were collected at different control frequencies, with varying camera setups, and some may not provide temporally aligned observation sequences of sufficient length for WoG's 4-frame sampling requirement. The paper does not report what fraction of OXE data was usable under this constraint, whether any datasets were excluded, or whether temporal downsampling strategies were needed to match the 4-frames-over-16-steps sampling pattern. A practitioner attempting to replicate WoG on a custom dataset would need to ensure sufficient temporal coverage in their demonstrations — a requirement that adds data collection burden.

Furthermore, the computational cost of Stage I is not isolated from the total training budget. Stage I requires forward passes through DINOv2, Wan VAE, the Q-Former, the VLM backbone, and the DiT head for every training sample, compared to Stage II which drops the DINOv2 and Wan VAE forward passes (they are used to precompute frozen targets but not backpropagated through). The paper aligns training steps across methods (100k + 50k for WoG, 150k for vanilla VLA) but does not compare wall-clock time or total FLOPs, which would be higher for Stage I due to the additional vision encoder forward passes.

Mitigation status. Not addressed. The paper does not discuss data compatibility constraints, temporal alignment requirements, or the FLOPs overhead of Stage I relative to Stage II or baseline methods. The training step alignment (Section 9.1) provides a coarse fairness guarantee, but a FLOPs-matched comparison would be more informative.

The Condition Space Is Defined by a Single Model Family and Architecture

The assumption or constraint. All experiments use a Prismatic VLM backbone adopted from OpenVLA, paired with a DiT action head (Section 3.2). The Q-Former Future Encoder uses 16 query tokens producing a 32-dimensional condition space. The paper does not test alternative VLA architectures (e.g., RT-2, Octo, π₀'s own backbone), alternative action head designs (e.g., autoregressive action chunking rather than diffusion), or alternative compression mechanisms (e.g., Perceiver-style architectures rather than Q-Former). The condition space dimensionality (32) and query token count (16) are fixed across all experiments with no ablation.

The consequence. The paper's core claim — that predicting a condition space discovered through action-relevant compression is superior to predicting full visual features or latent actions — is demonstrated for one specific instantiation of that idea with one specific architecture. It is unclear how sensitive the results are to these architectural choices. Would a 64-dimensional condition space improve performance on spatial precision tasks (where the current 32 dimensions may be the bottleneck) at the cost of harder condition prediction? Would a different VLA backbone with stronger spatial reasoning capabilities (e.g., one using higher-resolution inputs) close the gap on Stack and Drawer tasks where WoG currently underperforms? Would an autoregressive action head benefit differently from future conditions than the diffusion head? The paper's encoder configuration experiments (Section 4.3) demonstrate sensitivity to the choice of vision encoder, suggesting that architectural choices in the action pipeline may also matter — but this is not tested.

The fixed dimensionality (D=32) is particularly notable because the paper presents it as a key design choice (compactness to avoid redundancy) but never validates that 32 dimensions is optimal. If the condition space could be expanded to, say, 128 dimensions without making condition prediction intractable, it might capture the fine-grained spatial information that is currently missing. The paper's own analysis shows that the method fails on spatial precision tasks — an ablation over condition dimensionality would reveal whether this failure is due to insufficient capacity in the condition space or a more fundamental limitation of the future-observation information source.

Mitigation status. Not addressed. The paper treats the Prismatic VLM + DiT head + 16-query Q-Former as a fixed substrate and investigates variations only in the vision encoder configuration and the training strategy. The architectural sensitivity of the condition-space idea remains unexplored. The paper does not claim universality across architectures, but the practical consequence is that a practitioner using a different VLA stack (e.g., RT-2, Octo) cannot assume the gains will transfer without revalidation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new optimization criterion for world-action models — the sufficient condition criterion — that reframes the question from "what visual information can we predict?" to "what information does the action head treat as causally relevant for generating actions?" This is not a new architecture or a new loss function bolted onto existing VLA training; it is a different way of thinking about what a world-action model should optimize for. The practical instantiation — injecting future observations, letting the action loss discover what matters, then predicting only that discovered subspace — is the empirical mechanism that realizes this criterion.

The magnitude of the shift is methodological rather than paradigmatic. The paper does not overturn the VLA pretraining paradigm (OXE pretraining followed by task-specific fine-tuning remains the dominant pipeline), nor does it displace the core architectural components (VLM backbones, diffusion action heads). Rather, it changes the predictive target that world-action models are trained against — from task-agnostic visual features (DINOv2, video frames, depth maps) or reconstruction-compressed latent codes to an action-relevant condition space discovered through the action prediction objective itself. This is a reframing of the objective function rather than the architecture, and it operates within the existing infrastructure of VLA training.

The work resolves a contradiction in prior findings that had been implicit but unarticulated. World action models (VPP, DreamVLA) showed that predicting future visual features could improve action generation, but gains were inconsistent across tasks and often came with substantial computational overhead and brittleness to visual shifts. Latent action models (LAPA, UniVLA) showed that compressed future representations could enable learning from heterogeneous data, but their representations were too coarse for fine-grained control. These findings appeared to pull in opposite directions — richness vs. compactness — and the field lacked a framework for understanding when each approach would succeed or fail. The paper's condition-space formulation provides a unifying explanation: both approaches fail because they define their predictive targets based on properties of the input data (visual feature spaces, reconstruction objectives) rather than on what the action head needs. The sufficient condition criterion explains why world action models are redundant (they predict information the action head ignores) and why latent action models are coarse (they compress along axes of maximum visual variance rather than maximum action relevance). The condition space is neither rich nor compact in absolute terms — it is sufficient, defined by the action prediction objective itself.

This reframing makes several research directions more attractive. First, it suggests that the design of predictive targets in world-action models should be treated as a first-class optimization problem rather than an ad-hoc selection from available vision features. The paper provides one discovery mechanism (the two-stage curriculum with a Q-Former bottleneck), but the principle generalizes: any method that lets the action prediction objective define what future information is worth predicting should outperform methods that hand-design the predictive target. Second, it elevates the role of the embodiment-agnostic condition space as a bridge for cross-morphology data scaling — a direction that the paper demonstrates with human video and UMI data but that could extend to other heterogeneous data sources (simulation rollouts, third-person video, multi-robot datasets). Third, it shifts attention away from developing ever-more-expressive visual encoders for world-action models (an arms race that the encoder configuration experiments in Section 4.3 suggest has diminishing returns) and toward developing better discovery mechanisms for action-relevant representations.

Conversely, the work makes some directions less attractive. Naïve scaling of video prediction fidelity (higher resolution, longer horizons, more visual detail) as a route to better world-action models now appears misguided — the paper's encoder ablation shows that richer features help only when they align with the specific capabilities needed for the task, and the Future Encoder ablation (Section 4.4) shows that predicting full uncompressed feature maps underperforms predicting compressed conditions by a substantial margin (7.4 percentage points on Google Robot tasks, Table 5). This suggests that the field should invest in better selection and compression mechanisms for future information rather than better generation of that information.

The paper also implicitly shifts the burden of proof for new world-action model methods. A method claiming to improve action generation through future prediction should now demonstrate not just that its predictive target contains useful information (which is almost always true for rich visual features) but that the predictive target is optimally compact — that predicting a more compressed or a different version would not work better. The Future Encoder ablation provides a template for such a demonstration.

Follow-Up Research This Work Enables

Joint training vs. two-stage curriculum: a direct comparison on equivalent FLOPs. The paper argues that the two-stage design decouples condition-space discovery from condition prediction to avoid a circular dependency, but it never tests whether end-to-end joint training — where the VLM predicts conditions from current observations, feeds those predicted conditions into the action head, and everything is trained jointly with the combined loss from Equation 3 — achieves comparable performance. A strong follow-up would implement this joint training variant, match the total optimization steps and batch size to the two-stage WoG, and compare on the SIMPLER benchmark and a subset of the real-world tasks. If joint training matches or exceeds two-stage performance, the two-stage complexity is unnecessary and the condition-space idea becomes simpler to adopt. If joint training underperforms, the follow-up should characterize why — e.g., does condition prediction collapse to a trivial solution without the stable target provided by a frozen Stage I encoder? Does the action head learn to ignore early poor condition predictions, preventing the VLM from receiving useful gradient signal? The paper's "WoG w/o cotrain" ablation (Table 8) already shows that Stage I alone is insufficient, but a joint training baseline would test whether the Stage I → Stage II sequential dependency is necessary or merely sufficient.

Ablation of condition space dimensionality to characterize the expressiveness-efficiency frontier. The paper fixes the condition space at D = 32 dimensions across all experiments, with 16 Q-Former query tokens. This dimensionality is presented as a design choice that balances compactness against expressiveness, but no sweep is conducted to determine whether 32 is optimal or even near-optimal. A follow-up study should train WoG variants with condition dimensions ranging from 4 to 256 (keeping the Q-Former query token count proportional or fixed) and evaluate on SIMPLER tasks that span the trajectory-planning-to-spatial-precision spectrum. The hypothesis based on the paper's own analysis: smaller dimensions (4–16) will perform adequately on trajectory-focused tasks (Move Near, Pick Coke) but collapse on spatial precision tasks (Stack, Drawer); larger dimensions (64–256) will improve spatial precision but make condition prediction harder for the VLM backbone, potentially degrading overall performance due to increased alignment loss. The study would produce a task-dependent Pareto frontier of condition dimensionality vs. success rate, providing practitioners with concrete guidance on how to configure WoG for their task distribution. It would also test whether the paper's failure on spatial precision tasks is a fundamental limitation of future-observation information or an artifact of insufficient condition capacity — if performance on Stack improves monotonically with condition dimension, the bottleneck is capacity; if it saturates, the bottleneck is the information content of the future observations themselves.

Real-world comparison against DreamVLA with frozen encoder features to isolate the contribution of the Q-Former compression. The paper compares against DreamVLA in simulation (Tables 1, 2) but not in real-world experiments (Table 7). DreamVLA is the closest prior method because it also uses DINOv2 features as a predictive target, but it predicts the full uncompressed feature maps. A direct real-world comparison on the paper's three tasks (Pick and Place, Close Microwave, Fold Towel) under both ID and OOD conditions, with both methods using the same Prismatic VLM backbone, DiT action head, and frozen DINOv2 encoder, would isolate the specific contribution of the Q-Former compression and the condition-space formulation. If WoG outperforms DreamVLA by margins similar to those in simulation (e.g., ~5–15 percentage points on trajectory-dependent tasks), the case for condition-space compression is strong. If the gap is smaller in real-world settings — where visual diversity is lower and the redundancy of full DINOv2 features may be less penalized — then the practical advantage of the more complex WoG pipeline over the simpler DreamVLA approach would be proportionally smaller. This comparison is important for practitioner adoption: DreamVLA is architecturally simpler (no Q-Former, no two-stage training) and may be preferred if the performance gap is modest.

Scaling laws for condition prediction: how much unannotated human video is needed before benefits saturate? The paper demonstrates that unannotated human videos can improve performance on rigid-body tasks (P&P: 90% → 95%, Table 9) but degrade performance on deformable tasks (Fold: 85% → 80%). It uses a fixed corpus of 1,920 hours with a specific ratio of human-to-robot data. A follow-up scaling study should vary the amount of unannotated human video in Stage II from 0 to the full 1,920 hours (at log-spaced intervals) and measure success rates on both P&P and Fold under ID and OOD conditions. The study would produce scaling curves that answer: (1) At what volume of unannotated video do benefits on P&P saturate? (2) At what volume does degradation on Fold begin? (3) Is there a "sweet spot" where P&P improves without Fold degradation? (4) Does the degradation on Fold eventually reverse at very large video volumes as the model learns a more robust condition space? The answers would determine whether the negative result on deformable manipulation is a fundamental limitation (the human-robot condition gap cannot be bridged without action annotations) or a data-scale limitation (more video would eventually help). If the degradation persists at all scales for deformable tasks, practitioners would know to restrict unannotated video augmentation to rigid-body tasks only.

Condition-space interpretability: what do the 32 dimensions encode, and are they consistent across tasks? The paper treats the condition space as a learned black box — it is discovered by the action prediction objective and used as a predictive target, but its contents are never analyzed. A follow-up study should attempt to interpret the 32-dimensional condition vectors. Concrete approaches: (1) Train linear probes to predict task-relevant quantities (object position, gripper-to-object distance, contact state) from the condition vectors, measuring how much of the condition space is devoted to semantically meaningful vs. arbitrary compression artifacts. (2) Compare condition vectors across different tasks (P&P vs. Fold vs. Close) — do they use the same dimensions for the same purposes, or does each task induce a different condition-space geometry? If the space is consistent across tasks, the VLM backbone learns a general-purpose dynamics predictor; if it is task-specific, the condition space is best understood as a task-conditioned representation. (3) Ablate individual dimensions of the condition vector (set to zero) at test time and measure the impact on specific failure modes — does disabling dimension 7 cause the robot to miss grasps, while disabling dimension 12 causes collisions? This would provide mechanistic insight into how the condition space guides action generation and could inform future designs (e.g., structured condition spaces with disentangled dynamics and spatial components).

Extension to tasks where future observations carry different information types — navigation, assembly, deformable manipulation with diverse materials. The paper evaluates on three manipulation tasks (rigid, articulated, deformable) but all within a tabletop setting with relatively simple dynamics. The condition-space formulation should theoretically benefit any task where anticipating future states improves action selection. A follow-up should test WoG on a broader task taxonomy: (1) Mobile manipulation where the robot must navigate around obstacles to reach a target — this stresses the trajectory planning capability that Wan VAE-augmented WoG excels at. (2) Precise assembly tasks (peg insertion, gear meshing) where spatial precision is paramount — this would stress-test the paper's acknowledged limitation and reveal whether combining SigLIP for spatial precision with a higher-dimensional condition space can close the gap. (3) Deformable manipulation with materials of varying stiffness (cloth, rope, dough) — the Fold results already show task-dependent human-to-robot transfer, and diverse materials would characterize whether the condition space captures material-specific dynamics or only gross motion patterns. (4) Bimanual manipulation where the condition space must capture coordinated two-arm dynamics — the Q-Former's 16 query tokens may need to be increased to handle the larger state space. Results from this broader evaluation would define the boundary conditions of the condition-space approach more precisely than the current three-task real-world suite.

Practical Applications and Downstream Use Cases

Data-efficient fine-tuning for small-batch robot deployments. Organizations deploying robots for specific tasks (warehouse pick-and-place, lab automation, food handling) typically collect 50–200 expert demonstrations and fine-tune a pretrained VLA. The paper shows that WoG's two-stage pretraining plus fine-tuning achieves 90% success on Pick and Place from only 100 demonstrations (Table 7), compared to 65% for a vanilla VLA with identical data — a 25 percentage point improvement without additional data collection. For a deployment team, this means reaching operational success rates (~90%) with potentially half the demonstration collection effort compared to standard VLA fine-tuning. The condition prediction objective effectively augments the limited demonstration data with anticipatory knowledge learned during pretraining, extracting more value from each expert trajectory.

Cross-embodiment data reuse for robot fleets with heterogeneous hardware. The UMI data integration result (Figure 4) demonstrates that WoG can absorb trajectories from a completely different robot embodiment (egocentric viewpoint, different kinematics, different action space) during fine-tuning and achieve a 42% improvement on Pick and Place (60% → 85%). For organizations operating multiple robot types (e.g., a UR5 arm for heavy payloads, a Franka arm for precise assembly, a mobile manipulator for logistics), this means that demonstration data collected on one platform can improve performance on another without any embodiment-specific alignment engineering. The condition space, which captures object-level rather than robot-level dynamics, serves as the shared representation. A fleet operator could collect a large demonstration corpus on their easiest-to-teleoperate robot and use it to boost performance across all platforms during fine-tuning — a direct practical consequence of the embodiment-agnostic condition space design.

Human video-augmented pretraining for manipulation skills that are rare in robot datasets. The human data integration results (Table 9, "w. human v./a.") show that adding 220 hours of action-annotated human manipulation video — covering diverse object interactions that may not appear in robot datasets — during WoG pretraining improves real-world task performance from 90% to 100% on Pick and Place and from 85% to 95% on Fold the Towel, with consistent OOD generalization improvements. For organizations building generalist robot policies, this suggests a concrete data strategy: invest in collecting and annotating a modest corpus of human manipulation video (at the reported rate of ~450 trajectories per hour, 220 hours is approximately 100,000 trajectories — a feasible collection effort) and use it during WoG pretraining to expand the condition space with manipulation knowledge absent from robot demonstrations. The annotated human data requirement (11% of the total corpus) is the bottleneck, but the paper's results show that even this modest annotation investment yields substantial downstream gains. This is actionable today for any group with access to VR-based human data collection hardware.

When to Prefer This Method

The paper positions WoG against both world action models (which predict rich but redundant future features) and latent action models (which predict compact but coarse future representations). The following decision rule emerges from the empirical results and stated limitations:

Prefer WoG over standard VLA fine-tuning (vanilla VLA) when:

  • Task success depends on anticipating object dynamics, motion trajectories, or contact events (Pick and Place with obstacles, deformable manipulation, articulated object interaction) — the gap is 25 percentage points on P&P and 20 points on Fold (Table 8).
  • You have access to multi-frame future observations in your training data (the Stage I requirement) and can accommodate the two-stage training pipeline.
  • Your deployment involves visual distribution shifts (backgrounds, lighting, novel objects) where the frozen encoder-based condition space provides robustness — WoG shows 20-point OOD drops vs. 30-point drops for baselines (Table 7).

Prefer WoG over world action models (DreamVLA, VPP) when:

  • Training efficiency matters — WoG predicts a 32-dimensional condition vector rather than full DINOv2 feature maps (thousands of dimensions) or video frames, reducing the predictive burden on the VLM backbone.
  • You want to incorporate unannotated human video or cross-embodiment robot data — the condition space provides an embodiment-agnostic prediction target that world action models predicting raw visual features cannot exploit.
  • Your task mix is dominated by trajectory planning and dynamics reasoning rather than static spatial precision — WoG's 7.4 percentage point advantage over uncompressed feature prediction on Google Robot tasks (Table 5) is concentrated on trajectory-dependent tasks.

Prefer dedicated spatial reasoning modules over WoG (or augment WoG with them) when:

  • Your primary task requires precise relative positioning, sub-centimeter alignment, or fine-grained geometric reasoning (stacking, peg insertion, drawer alignment) — the condition space does not compensate for the spatial resolution limits of the underlying VLM backbone and vision encoders (Stack: WoG 34.0% vs. ViPRA 48.0%, Table 1; the paper explicitly states this requires "dedicated spatial mechanisms" beyond WoG's scope).

Prefer WoG with annotated human data over WoG with unannotated human data when:

  • Your task involves deformable manipulation — unannotated human video degrades Fold performance (85% → 80%, Table 9), while even 11% annotated human data improves it (85% → 95%).
  • You can afford to annotate a modest subset of your human video corpus — the paper's 220-hour annotated subset (11% of 1,920 hours) was sufficient to confer consistent gains across all tasks and OOD conditions.