Exploring Conditions for Diffusion models in Robotic Control

ArXiv: 2510.15510

🎯 Pitch

Text prompts, the driving force behind diffusion model breakthroughs in vision tasks, can actually degrade performance when used for robotic control, because the model’s cross-attention fails to align words with relevant image regions in simulation. To fix this, ORCA learns implicit task prompts and blends them with frame-specific visual prompts, enabling a frozen Stable Diffusion model to deliver state-of-the-art control representations across a wide suite of benchmarks.

1. Executive Summary

This paper studies how to condition pre-trained text-to-image diffusion models to produce task-adaptive visual representations for robotic control, evaluating on DeepMind Control, MetaWorld, and Adroit benchmarks using Stable Diffusion v1.5. The core contribution is ORCA, a framework that replaces conventional text prompts with learnable task prompts (implicit descriptions learned end-to-end during policy training that ground to task-relevant image regions) and visual prompts (frame-specific dense features from a vision encoder that capture fine-grained dynamic details), achieving state-of-the-art performance across all 12 tasks—a 74.3 mean normalized score on DMC versus 68.3 for the strongest diffusion baseline (SCR) and 65.3% mean success on Adroit versus 58.0% for SCR. The paper's central finding is that text conditions, while effective in visual perception tasks, fail in control environments because the diffusion model cannot reliably associate words with image regions in simulated domains (cross-attention maps for tasks like Cheetah-run are noisy and degrade performance below the null-condition baseline), establishing that effective conditioning for control requires task-grounded, frame-sensitive prompts that incorporate visual information rather than relying on language descriptions alone.

2. Context and Motivation

The Core Problem: Frozen Visual Representations Are Task-Agnostic

The fundamental problem this paper tackles is that the current standard approach to using pre-trained visual representations for robotic control—deploying a frozen, general-purpose vision encoder and never updating it during policy learning—produces representations that are task-agnostic. They capture general visual features useful across many scenarios, but they are not specifically optimized for the particular manipulation or locomotion task the robot is currently learning.

To understand why this matters, consider how imitation learning works: a policy network observes images of the environment (e.g., a robotic arm approaching a button, a cheetah-shaped robot running across a floor), encodes them into a visual representation via a pre-trained encoder, and then maps that representation to motor commands. If the encoder is frozen—as it is in methods like CLIP for control, VC-1, or even earlier diffusion-based approaches like SCR—then it produces the same representation regardless of whether the downstream task is pressing a button, running, or assembling parts. There is no mechanism for the representation to emphasize task-relevant features (e.g., the button's position) over task-irrelevant features (e.g., the background texture).

The paper illustrates this in Figure 2: in a task-agnostic approach (Figure 2a), the same observation of a robotic hand and sphere produces an identical representation whether the task is "grasp the sphere" (Task A) or "relocate the hand" (Task B). A task-adaptive approach (Figure 2b) would produce different representations for each task, emphasizing the sphere in one case and the hand in the other.

This is not merely a theoretical concern. The suitability of a particular frozen encoder for a given task is unknown beforehand, requiring manual, task-by-task inspection to determine which pre-trained model performs best. As the paper notes (Section 1):

"Since the suitability of a representation for a specific task is unknown beforehand, determining which representation performs best often requires manual, task-by-task inspection, which becomes cumbersome given the vast variety of control tasks."

The practical consequence is that practitioners must either settle for suboptimal performance from a single frozen encoder or engage in extensive trial-and-error across different pre-trained backbones for each new task—a process that does not scale.

Why Frozen Representations Are the Default—and Why That's Unsatisfactory

There is a good reason frozen encoders have become standard practice. The alternative—fine-tuning the vision encoder on downstream control data—has proven surprisingly harmful in imitation learning settings. As the authors cite from prior work (Hansen et al., 2022; Majumdar et al., 2023; Parisi et al., 2022):

"While a straightforward solution might be to fine-tune the vision encoder, this often results in poor results as the model loses generalization capabilities by overfitting to specific scenes in imitation learning."

The core issue is data scale asymmetry: imitation learning datasets are typically small (the paper uses 2, 5, and 5 demonstrations for Adroit, DMC, and MetaWorld respectively), while the pre-trained vision encoders have millions of parameters trained on internet-scale data. Full fine-tuning on a handful of demonstrations causes the encoder to overfit catastrophically to the specific visual background, lighting conditions, and object positions in the training trajectories, destroying the generalizable features that made pre-training valuable in the first place. Table 3 of the paper quantifies this directly: fine-tuning SCR (a diffusion-based encoder) on Adroit causes success rates to collapse from 58.0% to 9.3%, while fine-tuning VC-1 drops from 47.3% to 31.3%.

This creates a frustrating tradeoff: frozen encoders are safe but task-agnostic, while fine-tuned encoders are task-specific but overfit. The field has largely accepted the frozen option as the lesser evil, but the paper argues this is an unsatisfactory equilibrium. What is needed is a method that adapts the visual representation to the task without modifying the encoder's learned weights—preserving its generalization capability while still injecting task-specific guidance.

The Promise of Diffusion Models with Conditions

This is where text-to-image diffusion models enter the picture. Recent work in computer vision—particularly VPD (Zhao et al., 2023) and TADP (Kondapaneni et al., 2024)—has demonstrated that pre-trained diffusion models like Stable Diffusion can serve as powerful visual backbones for downstream tasks, and that text prompts can serve as a conditioning mechanism to make these representations task-adaptive without fine-tuning.

The mechanism works as follows: Stable Diffusion is a conditional generative model trained to denoise images given a text description. Internally, it contains cross-attention layers that allow text tokens to modulate the visual features at every stage of processing. When used as a feature extractor (rather than a generator), the text prompt can be changed to steer which aspects of the image the model attends to—without updating any model parameters. For semantic segmentation, prompting with "a photo of a cat" causes the diffusion model's internal representations to emphasize cat-relevant regions. For the same image, prompting with "a photo of a dog" produces different internal features. This is task-adaptivity through conditioning, not through weight updates.

VPD (Zhao et al., 2023) showed this explicitly:

"VPD demonstrated that downstream performance can be enhanced by with text conditions, such as the names of objects present in an image, in tasks such as semantic segmentation."

TADP (Kondapaneni et al., 2024) further extended this:

"TADP demonstrated that text descriptions generated from vision-language models can serve as strong conditions, and could be further enhanced with style modifiers learned from Textual Inversion."

The key appeal for robotic control is that this mechanism offers a third path: adapt the representation through conditioning, not through weight updates. By changing the text prompt per task (or even per frame), one could theoretically steer a single frozen diffusion backbone to produce specialized representations for each control scenario, bypassing both the task-agnostic limitation of frozen encoders and the overfitting hazard of fine-tuning.

The Gap: Text Conditions Fail in Robotic Control Environments

The paper's central empirical observation—and the gap it identifies—is that this conditioning strategy, which works well for standard vision tasks, largely fails when applied to robotic control environments. This is shown concretely in Figure 3(a), where using text captions (generated by Gemini 2.5, a state-of-the-art vision-language model) as conditions produces inconsistent results: some tasks improve (Button-press gains 5.3 percentage points), but others degrade significantly (Cheetah-run loses 5.9 points compared to the null condition with an empty prompt).

This is a critical finding because it reveals that the effectiveness of text conditioning is domain-dependent in ways the field had not previously characterized. The paper attributes this to a domain gap between the diffusion model's training data and robotic control environments:

"We attribute this discrepancy to the nature of the diffusion model being trained on web-collected images, which suits visual tasks that involve real-world images and common objects, such as semantic segmentation. However, control environments, featuring specialized robotic agents performing specific tasks, would require a more careful and deliberate approach to devising effective conditions for downstream policy learning."

The cross-attention visualizations in Figures 3(b) and 3(c) make this failure mode concrete. For Button-press—where text conditions help—the attention maps for words like "button" and "press" are well-localized to the relevant image regions (the red button, the yellow control box). For Cheetah-run—where text conditions hurt—the attention maps for words like "cheetah" and "run" are noisy and diffuse, failing to focus on the agent's body. Critically, the null condition's <eos> token is already roughly grounded to the salient object (the agent), which explains why a poorly-grounded text condition can degrade performance below the unconditional baseline: injecting noisy cross-attention signals corrupts an otherwise functional representation.

This domain gap is not surprising in retrospect. Stable Diffusion was trained on LAION-5B, a dataset of web images with natural language captions. These images overwhelmingly depict real-world scenes: photographs of animals, landscapes, objects, and people. The model learned rich associations between language and visual concepts in that domain. When presented with a simulated cheetah-shaped robot on a reflective floor—an image that arguably falls within the training distribution—the model can produce reasonable features. But the word "cheetah" activates associations learned from thousands of photographs of actual cheetahs in savannas, not orange minimalist robots in MuJoCo simulations. The text-image grounding that works for natural images transfers poorly to controlled, synthetic environments where the visual appearance of agents and objects diverges from natural language priors.

Beyond Static: The Dynamic Nature of Control Demands Frame-Sensitive Conditions

The paper identifies a second, equally important gap: even if text conditions could be made to ground properly in control environments, they would still be fundamentally mismatched to the dynamic nature of robotic control tasks. Standard vision tasks like semantic segmentation or depth estimation operate on static images; a single text prompt per image captures the relevant semantics. But robotic control operates on video streams, where the visual state evolves continuously and the action required at each frame depends on fine-grained details of the current state.

The paper makes this argument explicitly in Section 4.2:

"A primary distinction is that control tasks operate on video streams rather than static images. Consequently, a logical approach would be to generate a unique condition for each frame, allowing the representation to adapt to the changing visual state of the environment. For instance, instructing an agent to walk requires a sequence of distinct commands (e.g., move the left foot, then the right). Similarly, an effective condition should vary across frames to guide such dynamic behaviors."

However, generating frame-by-frame text descriptions is impractical. Even if a vision-language model could produce reasonable captions for each frame of a control trajectory, the fundamental grounding problem would persist. And critically, text descriptions lack the spatial granularity needed for control: knowing that "the robot hand is grasping the blue ball" is qualitatively different from knowing the precise position and orientation of the hand relative to the ball, which is the information a policy needs to generate appropriate motor commands.

The paper frames this as a requirement for conditions that incorporate visual information directly:

"We hypothesize that to account for this dynamic adaptability, conditions should incorporate visual information from each frame."

This is the key design insight that motivates ORCA's architecture. Rather than trying to translate visual state into language and back into visual features (text → text encoder → cross-attention → features), the paper argues for a more direct path: use visual features from the current frame as conditions, bypassing the language bottleneck entirely.

Where Existing Approaches Fall Short

The paper positions itself against a specific lineage of methods, each of which addresses part of the problem but falls short in critical ways:

Task-agnostic frozen encoders (CLIP, VC-1, SCR). These methods (Table 1 and Table 2 baselines) use pre-trained vision models as fixed feature extractors. CLIP (Radford et al., 2021) and VC-1 (Majumdar et al., 2023) use ViT-based architectures trained with contrastive or masked autoencoding objectives. SCR (Gupta et al., 2024)—the most directly relevant baseline—uses Stable Diffusion's U-Net with the null (empty) condition as a frozen feature extractor for control, and is one of the first works to demonstrate that diffusion model representations can be effective for robotic tasks. These approaches sidestep overfitting but produce the same representation regardless of the downstream task. As Table 1 shows, no single frozen encoder dominates: VC-1 outperforms CLIP on Assembly but underperforms on Reacher-easy; SCR is best overall but not universally superior. This variability across tasks is the hallmark of task-agnostic representations.

Text-conditioned diffusion models (Text Simple, Text Caption). The paper constructs these baselines by applying the VPD/TADP strategy directly to control: use the task name or a machine-generated caption as the text prompt to Stable Diffusion. As Figure 3(a) and Table 1 show, this produces mixed results—sometimes helping, sometimes hurting, never reliably outperforming the null-condition SCR baseline. The failure is not due to poor caption quality (the captions are generated by Gemini 2.5, a state-of-the-art VLM) but due to the domain gap in text-image grounding.

Prompt learning from vision-language models (CoOp). CoOp (Zhou et al., 2022) was originally developed for adapting CLIP to downstream image classification by learning continuous prefix tokens prepended to the class name. Applied to control by the paper, CoOp replaces the fixed task name with learned prefix tokens while keeping the task name as a suffix. This improves over pure text conditions (Table 1: 92.5% mean MetaWorld success vs. 91.5% for Text Simple), but the gains are modest. The limitation is that CoOp still operates within the text-conditioning paradigm—it learns better text prompts but doesn't incorporate visual information or address the frame-by-frame dynamic requirement.

Text + global visual style (TADP). TADP (Kondapaneni et al., 2024) extends text captions by learning a single special token $S^*$ that encodes the visual style of a reference image through Textual Inversion. The token is optimized to reconstruct a specific image when used as a prompt, and then appended to task captions during downstream use. The paper includes TADP as the strongest text-based baseline (Table 1: 93.1% MetaWorld, 70.7 DMC), and it does outperform simpler text methods. However, TADP's limitation is that the visual information it injects is global and static: a single $S^*$ token captures the overall visual style of one reference frame, not the per-frame fine-grained state information needed for control. The paper explicitly distinguishes ORCA from TADP on this basis:

"TADP, however, tend to embed the global representation into the condition or require additional optimization steps to acquire specialized tokens. Since our goal is to enable the recognition of fine-grained regions within each frame, we consider that adopting global representations and extra optimization steps should be avoided to facilitate effective frame-wise conditioning."

Fine-tuning approaches (full fine-tuning, parameter-efficient adaptation). Table 3 provides the direct comparison. Full fine-tuning of either VC-1 or SCR on control data causes severe degradation (SCR drops from 58.0% to 9.3% on Adroit). Parameter-efficient methods—RoboAdapter for ViT-based VC-1 and LoRA for diffusion-based SCR—recover some performance (59.3% and 60.0% respectively) but still fall well below ORCA (65.3%) while actually requiring more learnable parameters (18.0M for RoboAdapter, 4.6M for LoRA, vs. ORCA's total 10.6M learnable parameters, most of which come from the vision encoder projection layers). This validates the paper's premise that conditioning is a more effective and efficient path to task-adaptivity than adaptation of encoder weights.

How ORCA Positions Itself

The paper frames ORCA not as an incremental improvement to text conditioning, but as a fundamentally different approach to conditioning diffusion models for control that addresses both failures simultaneously: the domain-gap-induced grounding failure of text conditions, and the static nature of text prompts in dynamic environments.

The architecture replaces text prompts entirely with two learned components (Section 5 and Figure 4):

• Task prompts ($p_t$): learnable parameter vectors that are shared across all observations within a given task. They are not initialized from any language prior and have no predetermined meaning; they are optimized end-to-end through the behavior cloning loss to capture whatever visual concepts are relevant to the specific task. The cross-attention visualizations in Figure 6 show that these prompts learn to attend to task-relevant regions (e.g., the robot hand and target sphere in Relocate) without any textual supervision.

• Visual prompts ($p_v$): dense spatial features extracted from a frozen vision encoder (DINOv2) applied to the current observation frame, projected through a small convolutional layer. These are frame-specific by construction—each observation produces different visual prompts—and they provide the fine-grained spatial information that text cannot capture. Figure 6 shows that different tokens within the visual prompt learn to track different aspects of the scene dynamics (e.g., one token attends to the hand throughout, another tracks the table surface and shifts attention as the hand moves).

Both prompt types pass through Stable Diffusion's frozen text encoder $\tau_\theta$ and are injected into the U-Net via cross-attention, exactly as text tokens would be. This means ORCA leverages the existing conditioning machinery of the diffusion model—the cross-attention layers, the U-Net architecture—but supplies it with task-grounded, frame-sensitive conditioning signals rather than language-derived ones.

The paper explicitly pitches this as a synthesis of insights from multiple lines of prior work: the use of diffusion models as feature extractors (from SCR), the concept of learned prompts for task adaptation (from CoOp), and the incorporation of visual information into conditioning (from TADP)—but combined in a way that specifically targets the requirements of robotic control (frame-level granularity, dynamic adaptation, grounding in simulated environments) that none of the prior methods addressed.

3. Technical Approach

3.1 Reader Orientation

ORCA is a system that conditions a frozen Stable Diffusion model to produce task-specific visual representations for robotic control without fine-tuning any of the diffusion model's weights. The problem it solves is that frozen visual encoders produce task-agnostic representations (same features regardless of the control task), while fine-tuning them on limited imitation learning data causes catastrophic overfitting. The solution takes the shape of learning two types of prompts—task prompts and visual prompts—that are fed into Stable Diffusion's text encoder and cross-attention layers during inference, steering the diffusion model's internal representations toward task-relevant features while preserving its pre-trained generalization capability.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a feedforward pipeline:

1. Observation Image — an RGB frame from the robot's camera in the simulated environment (e.g., a MuJoCo rendering of a Sawyer arm reaching toward a button).

2. Vision Encoder $\mathcal{E}_V$ — a frozen DINOv2 model that processes the observation image and produces dense spatial feature maps. Its output provides the raw material for the visual prompts.

3. Learnable Prompts — two parameter tensors optimized during policy training: task prompts ($p_t$, shared across all frames of a given task) and visual prompts ($p_v$, derived per-frame from $\mathcal{E}_V$'s output via a small convolutional projection). Both pass through Stable Diffusion's frozen text encoder $\tau_\theta$ to become conditioning tokens.

4. Diffusion Model $\epsilon_\theta$ — the frozen denoising U-Net from Stable Diffusion v1.5. It receives the observation image (encoded to a latent $z_0$, then noised to $z_t$ at a fixed timestep $t=0$) and the conditioning tokens from $\tau_\theta$ via cross-attention. Its intermediate feature maps—specifically from the downsampling blocks and bottleneck—constitute the visual representation.

5. Policy Network $\pi_\phi$ — a learned mapping from the diffusion model's feature maps (after compression through a small convolutional layer) to action commands. Trained end-to-end with behavior cloning.

Information flows as follows: observation image → $\mathcal{E}_V$ extracts dense features → convolutional projection produces visual prompts $p_v$ → $p_v$ is concatenated with learned task prompts $p_t$ → text encoder $\tau_\theta$ encodes the combined prompt sequence → observation image is encoded to latent $z_0$, noised to $z_t$ → U-Net $\epsilon_\theta$ processes $z_t$ while cross-attending to the encoded prompts → intermediate U-Net features are extracted and compressed → policy $\pi_\phi$ predicts the action → behavior cloning loss backpropagates through $\pi_\phi$, the prompt parameters, and the visual prompt projection layer, but not through $\epsilon_\theta$, $\tau_\theta$, or $\mathcal{E}_V$.

3.3 Roadmap for the Deep Dive

• First, the Stable Diffusion architecture as a feature extractor — how an image goes in and intermediate features come out, including the critical design choices of which layers to extract and why timestep $t=0$ is chosen. This establishes the backbone that ORCA builds upon.
• Second, the conditioning mechanism — how text prompts normally modulate the U-Net via cross-attention, and why this mechanism is repurposed for learned prompts rather than language. Understanding cross-attention injection is prerequisite to understanding why task and visual prompts work.
• Third, the task prompts — what they are, how they are parameterized, how they are learned, and what design choices distinguish them from prior prompt learning methods like CoOp.
• Fourth, the visual prompts — the vision encoder $\mathcal{E}_V$, the convolutional projection layer, the spatial resolution of the dense features, and why dense rather than global features are critical for control tasks.
• Fifth, the combined conditioning pipeline — how task and visual prompts are concatenated, passed through $\tau_\theta$, and injected into the U-Net, including the token lengths and the interaction between the two prompt types.
• Sixth, the policy learning objective — the behavior cloning loss, what parameters are updated during training, and why the diffusion model itself remains frozen while the prompts are learned.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems design and empirical analysis paper whose core idea is that text-to-image diffusion models can be adapted for robotic control by replacing language-based conditioning with learned, task-grounded, frame-sensitive prompts that are optimized end-to-end through the behavior cloning objective while keeping the diffusion model frozen. The key insight is that the cross-attention mechanism in Stable Diffusion—designed to inject text semantics into image features—can be repurposed to inject arbitrary conditioning signals, and that signals derived from visual features and learned task parameters are far more effective for control than language descriptions.

Stable Diffusion as a Frozen Feature Extractor

The model. The paper uses Stable Diffusion v1.5 (SD 1.5) as its backbone. SD 1.5 is a Latent Diffusion Model (LDM) with a U-Net denoising architecture. The model was trained on LAION-5B, a large-scale dataset of web-collected image-text pairs, to perform text-conditioned image generation.

Why a diffusion model for representations? Prior work (SCR by Gupta et al., 2024; DIFT by Tang et al., 2023; VPD by Zhao et al., 2023) established that the internal activations of frozen diffusion U-Nets serve as strong visual representations for downstream tasks. The intuition is that the denoising objective forces the U-Net to learn rich, multi-scale features that capture both high-level semantics (what objects are present, driven by text conditioning) and low-level structure (where edges and textures are, driven by the denoising task). These features transfer well to tasks like semantic segmentation and correspondence, and SCR showed they transfer to robotic control when using the null (unconditional) setting.

From image to latent. An observation image $I$ (an RGB frame from the simulation) is first encoded into a compressed latent representation using SD 1.5's frozen VQGAN encoder $E$:

$z_0 = E(I)$

where $I \in \mathbb{R}^{H \times W \times 3}$ is the input image (typically 224×224 or similar for control tasks) and $z_0 \in \mathbb{R}^{h \times w \times c}$ is the latent representation at a reduced spatial resolution (the VQGAN compresses by a factor of 8, so $h = H/8$, $w = W/8$, with $c=4$ channels in SD 1.5's latent space).

What this computes: the VQGAN encoder applies a series of convolutions and downsampling operations to map the raw pixel image into a lower-dimensional latent code. This is the standard first step for any operation with a latent diffusion model—the U-Net never sees raw pixels; it operates entirely in this compressed latent space.

Why this form: the latent space is the native representation that SD 1.5 was trained to denoise. Operating in this space means the U-Net's internal features are computed under the exact distribution it was trained on, preserving the quality of the learned representations. Extracting features from the pixel-space decoder would be possible but would introduce a distribution shift since the decoder's features were not trained for the denoising objective.

The noising step. For a chosen fixed timestep $t$, Gaussian noise is added to the latent following the standard forward diffusion process:

$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1-\bar{\alpha}_t} \epsilon$

where $\epsilon \sim \mathcal{N}(0, I)$ is random Gaussian noise, $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$ is the cumulative product of the noise schedule coefficients (with $\alpha_i \in (0,1)$ controlling how much signal is retained at each step), and $z_t$ is the noised latent at timestep $t$.

What this computes: this equation blends the clean latent $z_0$ with random noise $\epsilon$ according to a ratio determined by $\bar{\alpha}_t$. When $t$ is small, $\bar{\alpha}_t$ is close to 1, so $z_t$ is mostly the clean signal with a small amount of noise added. When $t$ is large (near $T$, the total number of diffusion steps, which is 1000 for SD 1.5), $\bar{\alpha}_t$ approaches 0, and $z_t$ is almost pure noise.

Why this form: the diffusion model was trained to predict the noise component $\epsilon$ from noised latents at all timesteps. The choice of $t$ during feature extraction determines which "level" of the denoising process the features come from. The paper uses $t=0$ throughout, meaning $z_t = z_0$ (no noise added, since $\bar{\alpha}_0 = 1$ and $\sqrt{1-\bar{\alpha}_0} = 0$). This is a critical design choice explored in the ablation in Table D of the appendix: $t=0$ achieved 74.3 mean DMC score, while $t=100$ dropped to 68.5 and $t=200$ to 69.4. The intuition is that $t=0$ features correspond to the "clean" signal path through the U-Net, carrying the richest semantic information, while higher timesteps increasingly emphasize noise-prediction features that are less useful for representation learning.

Feature extraction from the U-Net. The noised latent $z_t$ and a conditioning signal $\mathcal{C}$ are fed into the denoising U-Net $\epsilon_\theta$. Rather than using the U-Net's noise prediction output (which is what it was trained to produce), the paper extracts intermediate feature maps from specific layers within the U-Net. The U-Net architecture consists of:

• Downsampling blocks (down 1, down 2, down 3): progressively reduce spatial resolution while increasing channel dimension, capturing increasingly abstract features.
• Bottleneck block (mid): the lowest spatial resolution, highest-level semantic features.
• Upsampling blocks (up 0, up 1, up 2): progressively restore spatial resolution via skip connections from the corresponding down blocks.

The paper extracts features from the downsampling blocks and the bottleneck only, concatenating them: "d + m" in Table 5 notation (down 1-3 + mid). The upsampling blocks are discarded entirely—a design choice validated by the layer-wise ablation in Table 5.

The layer selection ablation (Table 5). The paper evaluates each U-Net block individually on the five DMC tasks:

• down 1 alone: 68.4 mean
• down 2 alone: 65.1 mean
• down 3 alone: 67.5 mean
• mid alone: 64.6 mean
• up 0 alone: 59.0 mean
• up 1 alone: 60.4 mean
• up 2 alone: 49.7 mean
• d + m (down 1-3 + mid): 74.3 mean

The pattern is clear: earlier (down) layers outperform later (up) layers, and combining all down + mid layers yields the best result. This aligns with findings in other diffusion feature extraction work (e.g., DIFT, VPD): the downsampling path captures features that are more transferable to discriminative tasks, while the upsampling path is specialized for the generative reconstruction task. By discarding the up blocks, the paper removes approximately 500M parameters from the U-Net (as noted in Appendix A.6), making the feature extractor comparable in size to a ViT-L encoder while preserving the most useful representations.

Compression layer. The extracted multi-scale feature maps from the U-Net are 2D spatial feature grids at different resolutions and channel dimensions. To produce a unified representation suitable for a policy network that expects a 1D feature vector, the paper applies a compression layer—a small convolutional network applied independently to each extracted feature map:

nn.Conv2d(hidden_dim, compress_dim, kernel_size=3, padding=1)
nn.BatchNorm2d(hidden_dim)
nn.ReLU(inplace=True)
nn.Flatten()

The compressed features are concatenated to form the final visual representation fed to the policy network. compress_dim is set to 48 for all methods (including baselines), following prior work (SCR, Yadav et al., 2023) which showed this produces better representations than using the U-Net's bottleneck features directly or using special tokens like CLS embeddings.

Design choice summary. The paper's feature extraction pipeline inherits directly from SCR: use SD 1.5, encode to latent, process at $t=0$, extract down + mid features, compress. The innovation is entirely in what conditioning signal $\mathcal{C}$ is supplied—SCR uses the null condition (empty string), while ORCA replaces it with learned task and visual prompts. This means the feature extraction machinery is identical between SCR and ORCA; the performance difference comes entirely from the conditioning.

How Conditioning Works in Stable Diffusion

To understand why task and visual prompts work, one must understand the conditioning mechanism they leverage. Stable Diffusion implements text conditioning through cross-attention layers embedded within the U-Net blocks.

Text encoding. A text prompt (e.g., "a cheetah-shaped robot running on a reflective floor") is first tokenized into a sequence of discrete tokens, then processed by a frozen CLIP text encoder $\tau_\theta$ to produce a sequence of continuous embedding vectors. These embeddings have shape $(L, d_\tau)$ where $L$ is the number of tokens (padded to a maximum length, typically 77 for SD 1.5) and $d_\tau$ is the text encoder's output dimension.

Cross-attention injection. Within each U-Net block that contains cross-attention layers (specifically within the Transformer blocks interspersed in the U-Net architecture), the intermediate image features act as queries ($Q$) while the text embeddings act as keys ($K$) and values ($V$). The standard cross-attention operation is:

$\text{CrossAttn}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

where $Q$ is derived from the U-Net's image features at the current layer, and $K, V$ are linear projections of the text embeddings from $\tau_\theta$.

What this computes: for each spatial position in the U-Net's feature maps, the model computes attention weights over all text tokens, then aggregates the value vectors according to those weights. If the word "button" has high attention weight at a particular spatial location, the value vector for "button" is added to the features at that location, modulating them with button-relevant information. This is how text "guides" the denoising process—and, when used for feature extraction, how it steers the intermediate representations.

Why this mechanism is repurposable: the cross-attention layers are agnostic to what the "text" tokens actually represent. They are trained to attend from image features to a sequence of conditioning vectors. Those vectors happen to come from a text encoder during SD 1.5's training, but at inference time, any sequence of vectors of the same shape can be injected. The U-Net will compute attention weights over them and modulate features accordingly. This is the key insight that ORCA exploits: replace language-derived conditioning vectors with learned, visually-grounded conditioning vectors, and the cross-attention machinery will naturally learn to use them to steer features toward task-relevant aspects of the image.

The null condition. SD 1.5 is trained with classifier-free guidance, meaning some fraction of training examples use an empty conditioning signal (the "null" or "unconditional" setting, typically implemented as an empty string or a special null token). This means the U-Net learns to produce reasonable features even without text conditioning—which is why SCR (using the null condition) works at all. ORCA's improvements come from replacing this null signal with a richer, task-specific conditioning signal.

Task Prompts: Learning What to Attend To

What they are. Task prompts $p_t$ are learnable parameter vectors that replace the text prompt tokens. Specifically, the paper uses $l_t = 4$ task tokens (the length is specified in Section 6.2). Each token is a vector in the embedding space of the CLIP text encoder $\tau_\theta$. These vectors are randomly initialized—they carry no semantic meaning from language pre-training—and are optimized during downstream policy learning.

Parameterization. Concretely, $p_t \in \mathbb{R}^{l_t \times d_\tau}$ where $l_t = 4$ is the number of task tokens and $d_\tau$ is the dimensionality of the text encoder's token embedding space. These 4 vectors are stored as a learnable parameter tensor, one set per task (so training on 12 tasks means 12 separate $p_t$ tensors, though each policy is trained independently per task in the single-task setting).

How they are used. The task prompt vectors are concatenated with the visual prompt vectors (described next) to form a combined sequence of conditioning tokens. This sequence is then passed through the frozen CLIP text encoder $\tau_\theta$, which applies its standard transformer processing (self-attention, feedforward layers) to produce the final conditioning embeddings. These embeddings are then injected into the U-Net via cross-attention, exactly as text embeddings would be.

Why pass through the text encoder? This is a subtle but important design choice. One could imagine directly injecting the learned vectors into the cross-attention layers, bypassing $\tau_\theta$. However, $\tau_\theta$ provides crucial processing: its self-attention layers allow the task and visual tokens to interact and contextualize each other before being used for cross-attention. The text encoder's learned transformations—trained on internet-scale text-image data—provide a powerful inductive bias for producing conditioning signals that the U-Net can effectively use. By passing learned prompts through $\tau_\theta$, ORCA leverages this pre-trained processing while still allowing the prompt content to be task-adapted.

What they learn. The cross-attention visualizations in Figures 3(b,c) and 6 provide qualitative evidence. In the task prompt column of Figure 6, the attention map consistently highlights the robot hand and the green target sphere across all frames of the Relocate task. This is not pre-programmed—the task prompts learn during training that attending to these regions helps the policy predict actions correctly. The learning is driven entirely by the behavior cloning loss; there is no explicit attention supervision.

Relationship to CoOp. CoOp (Zhou et al., 2022) also learns prefix tokens prepended to a class name. The paper includes CoOp as a baseline (Table 1: 70.6 DMC, 92.5 MetaWorld) with 4 learnable prefix tokens and the task name as a suffix, e.g., [V*][V*][V*][V*] button press for Button-press. ORCA's task prompts differ in two critical ways:

1. No language anchoring: CoOp's learned tokens interact with a real text prompt (the task name) via the text encoder, anchoring the learned tokens to language semantics. ORCA's task prompts are standalone—there is no text prompt, no task name, no language grounding whatsoever. This eliminates the domain-gap-induced grounding failures that plague text conditions.

2. Task-only, no visual information: CoOp's tokens are the only conditioning signal beyond the fixed text. In ORCA, the task prompts are combined with visual prompts, which provide the frame-specific information that CoOp lacks.

The ablation in Table 4 quantifies the contribution: using only task prompts (without visual prompts) achieves 69.8 mean DMC, up from 68.3 for the unconditional SCR baseline but substantially below the 74.3 with both prompts. This confirms that task prompts alone provide some task-adaptive benefit, but the major gains come from combining them with visual prompts.

Why learn rather than design? The paper could have attempted to engineer better text prompts for each task (and the Text Caption baseline represents a best-effort attempt at this, using Gemini 2.5 to generate detailed descriptions). However, the domain gap problem means that even perfectly descriptive text may not ground correctly in the diffusion model's feature space. Learning the prompts directly from the control objective sidesteps this: the optimization process discovers whatever conditioning signal produces the best policy performance, regardless of whether it corresponds to human-interpretable language. The cross-attention maps show that the learned prompts do converge to human-interpretable patterns (attending to task-relevant objects), but this is an emergent property, not a design constraint.

Visual Prompts: Injecting Frame-Specific Information

What they are. Visual prompts $p_v$ are frame-specific conditioning vectors derived from a frozen vision encoder applied to the current observation image. Unlike task prompts, which are shared across all frames of a given task, visual prompts are unique to each observation—they change as the visual state changes, providing the dynamic, fine-grained conditioning that the paper argues is essential for control.

The vision encoder $\mathcal{E}_V$. The paper uses DINOv2 (Oquab et al., 2023) as $\mathcal{E}_V$, specifically the ViT-L/14 variant with registers. DINOv2 is a self-supervised vision transformer trained on a curated dataset of 142 million images, producing features that excel at dense prediction tasks like semantic segmentation and depth estimation. The choice is motivated by DINOv2's strong performance on fine-grained visual understanding—exactly the capability needed to capture the detailed state of a robotic agent and its environment.

The encoder $\mathcal{E}_V$ is frozen throughout training. Only the projection layer that converts its output to visual prompts is learned.

Dense vs. global features. The paper explicitly chooses to use dense spatial features from DINOv2 rather than a global representation (like the CLS token). The ViT architecture produces a feature map of shape $(N_{\text{patches}}, d_{\text{vit}})$ where $N_{\text{patches}}$ is the number of image patches (e.g., for a 224×224 image with patch size 14, this is 16×16 = 256 patches) and $d_{\text{vit}}$ is the feature dimension. Each patch token encodes information about a specific spatial region of the image.

The paper justifies this choice in Section 4.2:

"Since our goal is to enable the recognition of fine-grained regions within each frame, we consider that adopting global representations and extra optimization steps should be avoided to facilitate effective frame-wise conditioning."

A global representation (like the CLS token or average-pooled patch features) would capture the overall scene content but lose spatial specificity—it could tell you "there is a button in the scene" but not "the button is at position (x,y) and the robot hand is 3cm away." Dense features preserve this spatial information, which is critical for control tasks where the precise configuration of the agent and objects determines the correct action.

This is a key distinction from TADP, which encodes visual information into a single global style token $S^*$. TADP's $S^*$ captures the overall visual appearance of a reference image but cannot convey per-frame state changes. ORCA's dense visual prompts can.

The projection layer. The dense feature map from $\mathcal{E}_V$ has shape $(N_{\text{patches}}, d_{\text{vit}})$. To convert this into a form suitable for conditioning the diffusion model, the paper applies a small convolutional projection layer. Specifically (from Appendix A.6 and the description in Section 5):

1. The patch tokens are reshaped into a 2D spatial grid (e.g., 16×16 for a 224×224 image with ViT-L/14).
2. A convolutional layer (kernel size 3, padding 1) maps this grid from $d_{\text{vit}}$ channels to $d_\tau$ channels (the dimensionality expected by the text encoder).
3. The output is a spatial grid of conditioning vectors, which are then flattened into a sequence of tokens.

The paper uses $l_v = 16$ visual tokens (specified in Section 6.2), meaning the spatial grid from $\mathcal{E}_V$ is projected and potentially pooled or selected to produce 16 token vectors. These 16 vectors constitute the visual prompts $p_v$ for the current frame.

What the visual prompts learn. The cross-attention visualizations in Figure 6 provide compelling qualitative evidence. Two different visual prompt tokens ($p_v^1$ and $p_v^2$) are visualized across frames of the Relocate task:

• $p_v^1$ consistently attends to the robot hand across all frames, tracking its position as it moves to grasp the ball and carry it to the target.
• $p_v^2$ shows more dynamic behavior: in early frames (Frame 1, Frame 15), it attends to the table surface as the hand descends. In later frames (Frame 30, Frame 45), as the hand lifts off with the ball and moves toward the target, its attention shifts to the hand.

This dynamic attention behavior is exactly what the paper hypothesized: different visual prompt tokens learn to specialize in tracking different task-relevant elements of the scene, and their attention patterns evolve frame-by-frame as the visual state changes. No text prompt could capture this granularity—generating text descriptions that specify "the hand is at coordinates (x,y,z) moving at velocity (vx,vy,vz)" for every frame would be impractical even if the grounding problem were solved.

Ablation on encoder choice (Appendix Table C). The paper tests alternatives to DINOv2 for $\mathcal{E}_V$:

• Without any vision encoder (task prompts only): 54.7% Adroit mean
• With SigLIP encoder: 55.3%
• With CLIP encoder: 63.3%
• With SD-VAE (the VQGAN encoder already in SD 1.5): 61.3%
• With DINOv2 (ORCA default): 65.3%

All vision encoder variants outperform the no-encoder baseline, confirming that the visual prompt mechanism itself—not the specific encoder—is the core contributor. Interestingly, SD-VAE achieves competitive results (61.3%), suggesting that even the diffusion model's own encoder provides useful visual conditioning signals, though DINOv2's stronger dense features yield the best performance.

Combined Conditioning Pipeline

Prompt construction. For each observation frame during training or inference, the conditioning signal $\mathcal{C}^*$ is constructed as follows:

1. Visual prompts: the observation image $I$ is fed through the frozen DINOv2 encoder $\mathcal{E}_V$, producing dense features. These are projected through the learned convolutional layer to produce $l_v = 16$ visual token vectors.

2. Task prompts: the $l_t = 4$ learned task token vectors (shared across all frames of this task) are retrieved from the parameter store.

3. Concatenation: the visual and task token vectors are concatenated into a sequence of length $l_t + l_v = 20$ tokens. The paper does not specify the ordering (task-then-visual or visual-then-task), but the standard approach in prompt learning is to prepend or append learned tokens to form a continuous sequence.

4. Text encoding: the combined token sequence is passed through the frozen CLIP text encoder $\tau_\theta$, producing a sequence of conditioning embeddings $\tau_\theta(p_t; p_v)$ of shape $(l_t + l_v, d_\tau)$. The text encoder's self-attention layers allow the task and visual tokens to interact—task tokens can contextualize the visual information, and visual tokens can ground the task tokens to the current frame.

5. U-Net injection: these conditioning embeddings are injected into the cross-attention layers of the frozen U-Net $\epsilon_\theta$ as the keys and values, while the U-Net's image features serve as queries.

Token lengths. The specific lengths are $l_t = 4$ and $l_v = 16$ (Section 6.2). These are relatively small numbers: SD 1.5's text encoder normally processes up to 77 tokens. The paper does not ablate these lengths, but the choice reflects a tradeoff: more tokens could provide richer conditioning at the cost of more learnable parameters and longer sequences for the text encoder to process. Four task tokens are sufficient to capture task-level semantics (as the visualizations show clear attention patterns), and 16 visual tokens provide enough capacity to represent multiple spatial regions without excessive overhead.

Why concatenate rather than use separate conditioning paths? One could imagine injecting task and visual prompts through separate cross-attention layers, or using task prompts in some U-Net blocks and visual prompts in others. The paper's approach is simpler: concatenate everything into one sequence, let the text encoder's self-attention figure out the interactions, and inject through the standard cross-attention path. This has the advantage of leveraging SD 1.5's existing conditioning infrastructure without architectural modification. The text encoder was trained to process sequences of related tokens (words in a sentence); the concatenated prompt sequence mimics this structure, with task tokens providing "context" and visual tokens providing "content."

Parameter count and efficiency. From Appendix Table F:

• Total parameters (including frozen components): 480.1M for ORCA
• Learnable parameters: 10.6M
• Of these, the bulk comes from the projection layers for visual prompts (DINOv2 is 304M parameters frozen, the U-Net is ~383M frozen, the text encoder is frozen)

Compare to the alternatives in Table 3:

• Fine-tuning VC-1: 302.3M learnable parameters (entire ViT-L)
• RoboAdapter on VC-1: 18.0M learnable parameters (adapter modules)
• LoRA on SCR: 4.6M learnable parameters (low-rank matrices)
• ORCA: 10.6M learnable parameters (mostly in the visual prompt projection)

ORCA achieves better performance than the parameter-efficient fine-tuning methods while using a comparable parameter budget. The key efficiency gain is that ORCA's learnable parameters are in the conditioning pathway, not in the feature extraction backbone—meaning the diffusion model's pre-trained weights are completely preserved, avoiding the overfitting that plagues even parameter-efficient fine-tuning (as shown by SCR+LoRA's 60.0% vs. ORCA's 65.3% on Adroit).

Policy Learning Objective

Behavior cloning setup. The policy is trained on a small set of expert demonstrations: 2 demonstrations for Adroit, 5 for DMC, and 5 for MetaWorld. Each demonstration is a trajectory of observation-action pairs: $\{(I_o^i, a_o^i)\}_{o=0}^{T_o}$ for the $i$-th trajectory, where $I_o^i$ is the RGB observation at timestep $o$ and $a_o^i$ is the corresponding expert action (joint torques for DMC, end-effector commands for MetaWorld and Adroit).

The loss function. The policy network $\pi_\phi$ and the prompt parameters are trained jointly by minimizing the behavior cloning loss:

$\mathcal{L}_{\text{BC}}(\phi, \mathbf{p}) = \sum_{i=1}^N \sum_{o} \|\pi_\phi(\epsilon_\theta(z_t, t; \mathcal{C}^*)) - a_o^i\|$

where $N$ is the number of demonstration trajectories, $\phi$ are the policy network parameters, \mathbf{p} = \{p_t, \text{conv_proj_params}\} are the prompt-related learnable parameters (task prompt vectors and visual prompt projection layer weights), $z_t = \sqrt{\bar{\alpha}_t} E(I_o^i) + \sqrt{1-\bar{\alpha}_t}\epsilon$ is the noised latent (with $\epsilon \sim \mathcal{N}(0,I)$ being fresh noise sampled for each observation independently), $t=0$ is the fixed timestep (making $z_t = z_0$, the clean latent), $\mathcal{C}^* = \tau_\theta(p_t; p_v)$ is the conditioning signal derived from the concatenated task and visual prompts, $\epsilon_\theta(\cdot)$ denotes the extracted intermediate features from the U-Net (not the noise prediction), and $\|\cdot\|$ denotes the L1 or L2 distance between predicted and expert actions (the paper does not specify which norm; standard behavior cloning typically uses MSE).

What this computes: for each observation in each demonstration trajectory, the system extracts visual features from the diffusion U-Net conditioned on the learned prompts, feeds them through the policy network to predict an action, and computes the distance between the predicted action and the expert's action. The loss is summed over all timesteps in all trajectories, and gradients flow back through the policy network, the compression layer, and into the prompt parameters—but not into the U-Net, text encoder, VQGAN, or DINOv2, which are all frozen.

What parameters are updated. Three sets of parameters receive gradients:

1. Policy network $\phi$: the standard behavior cloning update, learning to map visual features to actions.
2. Task prompts $p_t$: the $l_t$ token vectors per task, updated to produce conditioning signals that highlight task-relevant visual features. The gradient path is: loss → policy → compression layer → U-Net features → cross-attention weights → text encoder → task prompt embeddings. The loss signal teaches the task prompts which visual regions to attend to by rewarding attention patterns that lead to correct action predictions.
3. Visual prompt projection layer: the small convolutional network that maps DINOv2 features to visual prompt tokens. The gradient path is: loss → policy → compression layer → U-Net features → cross-attention weights → text encoder → visual prompt tokens → projection layer. The loss signal teaches the projection layer to extract and format the DINOv2 features in a way that the text encoder and U-Net can effectively use for conditioning.

Why this form (end-to-end, single loss): the prompts are not pre-trained or optimized separately—they are learned directly from the downstream task objective. This is a key differentiator from TADP, which requires a separate optimization step (Textual Inversion) to learn the $S^*$ token before it can be used in the downstream task. ORCA's end-to-end approach means the prompts are optimized specifically for control performance, not for image reconstruction or any proxy objective. The cross-attention patterns that emerge (focusing on task-relevant objects, tracking dynamic elements) are a byproduct of optimizing for action prediction accuracy.

Why L1 or L2 loss rather than a more sophisticated objective: standard behavior cloning with regression loss is the established baseline for these benchmarks (following VC-1 and SCR). The paper's contribution is in the representation, not the policy learning algorithm, so keeping the simplest possible policy learning objective isolates the effect of the conditioning method.

Training hyperparameters. The paper trains each agent for 100 epochs with periodic online evaluation every 10 epochs (Section 6.2). The evaluation reports success rates (for MetaWorld and Adroit) or normalized scores (for DMC) from running the learned policy in the simulation environment. Proprioceptive data (joint angles, end-effector positions) is used except for DMC, where the policy operates purely from vision. This follows the standard protocol from VC-1.

Why periodic evaluation rather than validation loss for model selection: in imitation learning, offline validation loss on held-out demonstration data does not reliably predict online performance, because small errors compound over time when the policy is deployed (the "distribution shift" problem). Periodically rolling out the policy in the simulator and measuring task success provides the true signal of policy quality.

Noise sampling and stochasticity. The forward diffusion step $z_t = \sqrt{\bar{\alpha}_t} E(I) + \sqrt{1-\bar{\alpha}_t}\epsilon$ samples fresh Gaussian noise $\epsilon$ for each observation independently at each training step. Even though $t=0$ makes the noise coefficient $\sqrt{1-\bar{\alpha}_0} = 0$, so the noise term vanishes and $z_t = E(I)$ deterministically, the paper includes the noise term in the equation for notational consistency with the standard diffusion formulation. At $t=0$, there is no stochasticity in the feature extraction, which is desirable for policy learning (deterministic features given the same observation).

Design Rationale Summary: Why This Approach Over Alternatives

Why prompts rather than fine-tuning? As demonstrated in Table 3, full fine-tuning of the diffusion U-Net causes catastrophic collapse (SCR drops from 58.0% to 9.3% on Adroit). Even parameter-efficient fine-tuning (LoRA) underperforms ORCA (60.0% vs. 65.3%) while modifying the feature extraction backbone. The prompt-based approach preserves the diffusion model's pre-trained generalization capability—the U-Net weights are untouched, so the rich visual features learned from LAION-5B remain intact. The prompts merely steer which aspects of those features are emphasized for the current task.

Why visual + task prompts rather than one or the other? The ablation in Table 4 shows that each prompt type contributes independently: task prompts alone improve over the unconditional baseline (69.8 vs. 68.3 DMC), visual prompts alone also improve (70.5 vs. 68.3), and combined they produce the best result (74.3). The complementary roles are supported by the cross-attention visualizations: task prompts provide consistent attention to task-relevant objects, while visual prompts provide dynamic, frame-specific attention that tracks motion and state changes.

Why DINOv2 rather than CLIP or SD-VAE for $\mathcal{E}_V$? Table C shows DINOv2 achieves the highest Adroit performance (65.3%), but the gap over CLIP (63.3%) and SD-VAE (61.3%) is modest. DINOv2's advantage likely comes from its training objective (self-supervised learning on curated data) producing features that are particularly strong for dense spatial understanding. However, the fact that even SD-VAE—the VQGAN encoder already in SD 1.5—provides useful visual prompts suggests the mechanism is robust to the choice of vision encoder. The core insight (inject frame-specific visual information) matters more than which specific encoder produces that information.

Why pass through the text encoder rather than directly into cross-attention? The text encoder provides a learned transformation that was trained to produce conditioning signals effective for the U-Net. Bypassing it would require the prompt vectors to directly occupy the cross-attention key/value space, which may not be the optimal format. The text encoder's self-attention layers also enable interaction between task and visual tokens before they're used for cross-attention—the task tokens can "read" the visual information and adjust their representation accordingly, and vice versa.

Why $t=0$ rather than higher timesteps? Table D shows $t=0$ achieves 74.3 mean DMC vs. 68.5 for $t=100$ and 69.4 for $t=200$. The paper's interpretation is that $t=0$ features correspond to the "clean" image processing path and carry the richest semantic information. Higher timesteps emphasize noise-prediction features that are less transferable to discriminative control tasks. This is consistent with findings in the diffusion feature literature (e.g., DIFT found early timesteps better for semantic correspondence, later timesteps better for low-level texture matching).

4. Key Insights and Innovations

Innovation 1: Text Conditions as a Diagnostic Tool That Exposes a Domain Gap, Not a Solution Strategy

The paper's deepest conceptual move is not proposing a new method but reframing text conditioning from a solution to a diagnostic probe. Prior work—VPD (Zhao et al., 2023), TADP (Kondapaneni et al., 2024), and the broader diffusion-for-perception literature—treated text prompts as a mechanism that demonstrably improves performance on semantic segmentation, depth estimation, and correspondence. The unspoken assumption was that this mechanism transfers: if text conditioning helps segmentation, it should help any vision task, including robotic control.

ORCA's analysis in Section 4.1 shows this assumption is false, but more importantly, it shows why it's false in a way that illuminates a previously undiagnosed structural problem. The cross-attention visualizations in Figure 3(b,c) are not just debugging artifacts—they reveal that the diffusion model's text-image grounding, learned from web-collected photographs of natural scenes, fails systematically when applied to simulated robotic environments. Words like "cheetah" activate attention patterns derived from savanna photographs, which distribute noisily over a minimalist orange MuJoCo robot. The null condition's <eos> token, by contrast, grounds reasonably well to salient objects because it has no misleading semantic associations. This explains the counterintuitive result that an empty prompt outperforms a descriptively accurate caption on tasks like Cheetah-run (Figure 3a: text caption degrades performance by 5.9 points relative to null).

This reframing matters because it changes what future work should optimize. If text conditioning were simply weak, the fix would be better captions. But the paper shows the problem is fundamental domain mismatch in cross-modal grounding, not caption quality. The Text Caption baseline uses Gemini 2.5, a state-of-the-art VLM, to generate descriptions—these are near-optimal text prompts, yet they still fail. The implication is that the language pathway itself is the bottleneck, not the specific words chosen. This is a negative result with positive implications: it rules out an entire class of approaches (better text engineering) and redirects effort toward visual conditioning pathways.

This diagnostic function is what separates ORCA from being "just another prompt learning method." The paper could have jumped straight to the proposed method without the text-conditioning analysis. But the analysis establishes why the method takes the form it does—why task prompts must be learned rather than designed, why they must be free of language anchoring, why visual information must be injected directly rather than mediated through text. The failure of text conditions is the paper's intellectual fulcrum, not a mere motivation paragraph.

Innovation 2: Repurposing Cross-Attention as a Task-Adaptive Steering Mechanism for Frozen Representations

Prior work on pre-trained representations for robotic control faced a binary choice: freeze the encoder (task-agnostic, safe, but suboptimal) or fine-tune it (task-specific, but overfits catastrophically on small imitation learning datasets). VC-1 (Majumdar et al., 2023) and SCR (Gupta et al., 2024) chose freezing. The fine-tuning alternative was known to collapse—the paper quantifies this in Table 3, where full fine-tuning of SCR drops Adroit success from 58.0% to 9.3%.

ORCA introduces a third regime: modify the conditioning signal that the frozen encoder receives, not the encoder weights themselves. This is conceptually distinct from both freezing and fine-tuning. The diffusion U-Net's weights are never touched—the pre-trained features are fully preserved, avoiding the overfitting that destroys fine-tuned encoders. But the representations are not static; the cross-attention layers, which were trained to modulate image features based on text, are repurposed to modulate features based on learned task and visual prompts. The U-Net produces different feature maps for the same image depending on what conditioning signal it receives.

This is a fundamental shift in how to think about task adaptation, not merely an incremental improvement over prior prompt learning methods. CoOp (Zhou et al., 2022) learns prefix tokens for CLIP classification, but CLIP's architecture uses a single pooled representation—there is no cross-attention mechanism for the prompts to steer spatial features. TADP learns a style token for diffusion models, but it optimizes a single global token through a separate Textual Inversion procedure, not end-to-end with the task objective. ORCA is the first method to learn conditioning tokens and a visual feature projection jointly with the policy through the behavior cloning loss, with gradients flowing from action prediction errors all the way back through the cross-attention weights to the prompt parameters and the visual projection layer. The cross-attention mechanism, originally designed for text-to-image generation, is being used as a differentiable feature steering module that can be optimized for any objective—in this case, action prediction accuracy.

The significance extends beyond the specific prompts ORCA uses. This repurposing of cross-attention suggests a general paradigm: any pre-trained conditional model with cross-attention can be adapted to new tasks by learning what to inject through its existing conditioning pathway, without weight modification. This is architecturally general—it applies to any U-Net or Transformer with cross-attention layers—and task-agnostic in terms of what the conditioning signals represent. The paper chose task tokens and DINOv2 features, but the framework could accommodate other conditioning sources (language instructions, goal images, force-torque readings) without architectural change.

Innovation 3: Decomposing Conditioning into Task-Level and Frame-Level Components with Complementary Functions

The separation of conditioning into task prompts (shared across all frames, learned per task) and visual prompts (unique per frame, derived from a vision encoder) is not an arbitrary design choice—it reflects a structural insight about what robotic control tasks require from visual representations. The paper argues, and the cross-attention visualizations in Figure 6 demonstrate, that these two components serve complementary functions that neither alone can fulfill.

Task prompts learn to consistently attend to static, task-defining elements—the robot hand and the target sphere in Relocate, the button in Button-press. These are the objects and regions that define what the task is about, and they remain relevant across the entire trajectory. The fact that task prompts can learn this from the behavior cloning loss alone, without any supervision specifying what to attend to, is evidence that the policy learning objective provides sufficient signal to discover task-relevant regions.

Visual prompts learn to attend to dynamic, state-dependent elements—the table surface when the hand is descending to grasp, the hand itself when it is moving toward the target. These attention patterns change frame-by-frame, tracking the fine-grained visual state that determines what action to take next. The fact that different visual tokens ($p_v^1$ and $p_v^2$ in Figure 6) learn to track different aspects of the scene dynamics suggests that the conditioning mechanism has sufficient capacity to represent multiple concurrent motion patterns.

This decomposition matters because it resolves a tension that prior methods either ignored or handled unsatisfactorily. Pure text conditioning (Text Caption) provides only task-level information—a single caption per task—and cannot convey frame-specific state. Pure visual conditioning from a single reference image (TADP's $S^*$ token) provides visual information but captures only the global appearance of one static frame, not the evolving dynamics. CoOp's learned prefixes provide task-level adaptation but no visual information at all. The ablation in Table 4 confirms that neither component alone approaches combined performance: task prompts alone achieve 69.8 DMC (vs. 68.3 unconditional), visual prompts alone achieve 70.5, and combined they reach 74.3. The gain is more than the sum of the individual improvements (1.5 + 2.2 = 3.7 would be additive, but 6.0 is achieved), confirming that the components interact synergistically through the text encoder's self-attention and the U-Net's cross-attention.

This decomposition is not just effective—it is conceptually extensible. One could imagine adding more conditioning components for other sources of information: goal-image prompts for goal-conditioned policies, language-instruction prompts for instruction-following, proprioceptive-state prompts for sensorimotor integration. The framework separates what varies with the task (learned, fixed per task), what varies with time (computed per frame), and what could vary with other modalities (future extensions), providing a clean ontology for thinking about conditioning in control.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on 12 tasks drawn from three standard vision-based robot learning benchmarks: DeepMind Control (DMC) (Tassa et al., 2018) with 5 tasks (Walker-stand, Walker-walk, Reacher-easy, Cheetah-run, Finger-spin), MetaWorld (Yu et al., 2020) with 5 tasks (Assembly, Bin-picking, Button-press-topdown, Drawer-open, Hammer), and Adroit (Rajeswaran et al., 2018) with 2 tasks (Relocate, Pen). These are the same task splits used by VC-1 (Majumdar et al., 2023). The paper uses 2 demonstrations for Adroit, 5 demonstrations for DMC, and 5 demonstrations for MetaWorld, following the VC-1 protocol.

• Base model(s). All diffusion-based methods—including ORCA and the SCR, Text, CoOp, and TADP baselines—use Stable Diffusion v1.5 (Rombach et al., 2022) as the frozen feature extraction backbone. The non-diffusion baselines use CLIP ViT-L/16 (Radford et al., 2021) and VC-1 ViT-L/16 (Majumdar et al., 2023). The choice of SD 1.5 is motivated by its status as a widely-used text-to-image diffusion model with established feature extraction procedures (following SCR, Gupta et al., 2024) and its U-Net architecture with cross-attention layers that support the conditioning mechanism ORCA exploits.

• Metrics. For DeepMind Control, the paper reports normalized scores, where 0 corresponds to a random policy and 100 corresponds to an expert policy, averaged over three random seeds with standard deviation. For MetaWorld and Adroit, the paper reports success rates (%), where a task is considered successful if the agent achieves the specified goal (e.g., pressing the button, assembling the parts, relocating the object) within the episode, also averaged over three seeds with standard deviation. Online evaluation is conducted every 10 epochs during the 100-epoch training run, and the best success rate or normalized score across all evaluations is reported.

• Baselines. The paper constructs eight baselines organized into two categories. Task-agnostic baselines (no conditioning adaptation): CLIP (Radford et al., 2021)—ViT-L/16 features from a vision-language model trained with contrastive learning on internet-scale image-text pairs; VC-1 (Majumdar et al., 2023)—a ViT-L/16 trained with MAE on egocentric videos and robotics data, specifically designed as a foundation model for embodied intelligence; SCR (Gupta et al., 2024)—Stable Diffusion v1.5 used with the null (empty string) condition, representing unconditional diffusion features for control. Task-adaptive baselines (some form of conditioning): Text (Simple)—uses the task name from the benchmark as the text prompt (e.g., "button press" for Button-press); Text (Caption)—uses detailed descriptions generated by Gemini 2.5 Pro (Comanici et al., 2025), a state-of-the-art vision-language model, as the text prompt (full captions provided in Appendix Table G); CoOp (Zhou et al., 2022)—extends Text (Simple) by prepending 4 learnable prefix tokens to the task name, formatted as [V*][V*][V*][V*] task_name; TADP (Kondapaneni et al., 2024)—extends Text (Caption) by appending a special style token S* learned through Textual Inversion on a single reference image from the task, injecting global visual style information into the text condition. All methods use the same compression layer (a small conv-bn-relu-flatten network compressing features to 48 dimensions) for fair comparison.

• Generation budget / compute accounting. The paper does not use a "generation budget" concept in the standard test-time compute sense (there is no iterative sampling or search over candidate solutions). Instead, all methods are compared under equal training conditions: same number of demonstrations (2, 5, or 5), same number of training epochs (100), same evaluation protocol (online every 10 epochs), and same compression layer architecture. The key resource metric is number of learnable parameters and inference latency (reported in Appendix Table F). ORCA uses 10.6M learnable parameters (task prompts + visual prompt projection layers) with inference time of 48ms per frame, compared to 0 learnable parameters and 26ms for SCR (frozen U-Net only) and 0 learnable parameters and 11ms for VC-1. The paper does not normalize for parameter count or latency—the comparison is about whether conditioning improves accuracy, not whether it is pareto-optimal in the accuracy-efficiency tradeoff.

• Cross-validation / statistical protocol. The paper does not use cross-validation for strategy selection (unlike the compute-optimal scaling literature). Results are reported as means and standard deviations over three random seeds for each task. The standard deviations are substantial in some cases—for example, VC-1 on Adroit Pen achieves 65.3 ± 16.7 (Table 2), indicating high variability—but the paper does not perform formal statistical significance testing. All comparisons are based on mean differences with qualitative interpretation. Periodic online evaluation every 10 epochs provides a form of temporal validation, selecting the best-performing checkpoint rather than the final one, which is standard in imitation learning to account for policy degradation with extended training.

Main Quantitative Results

Aggregate Performance Across All Benchmarks

The central quantitative finding is that ORCA establishes new state-of-the-art results across all three benchmarks, outperforming both task-agnostic and task-adaptive baselines. The headline numbers are presented in Tables 1 and 2:

• DeepMind Control (Table 1): ORCA achieves a mean normalized score of 74.3 across 5 tasks, compared to 68.3 for SCR (the strongest diffusion baseline), 70.7 for TADP (the strongest task-adaptive baseline), 59.5 for CLIP, and 53.1 for VC-1. This represents a 6.0 point improvement over SCR (8.8% relative improvement) and a 3.6 point improvement over TADP (5.1% relative improvement).

• MetaWorld (Table 1): ORCA achieves a mean success rate of 95.2% across 5 tasks, compared to 90.4% for SCR, 93.1% for TADP, 81.3% for CLIP, and 84.2% for VC-1. The improvement over SCR is 4.8 percentage points (5.3% relative), and over TADP is 2.1 percentage points. Notably, ORCA achieves 100.0% success on Drawer-open and 98.7% on Assembly and Hammer, approaching ceiling performance on these tasks.

• Adroit (Table 2): ORCA achieves a mean success rate of 65.3% across 2 tasks, compared to 58.0% for both SCR and CoOp, 57.3% for TADP and both Text variants, 51.4% for CLIP, and 47.3% for VC-1. This represents a 7.3 percentage point improvement over SCR (12.6% relative), the largest relative gain among the three benchmarks. The improvement is driven entirely by Pen (86.7% vs. 84.0% for SCR), while Relocate performance is tied with CLIP at 44.0%—but CLIP's Relocate score is anomalous (VC-1 achieves only 29.3%, SCR only 32.0%), suggesting that Relocate may be a task where CLIP's representations happen to be particularly well-suited through chance rather than systematic advantage.

Cross-benchmark pattern. A clear hierarchy emerges across the baselines. Among task-agnostic methods, SCR consistently outperforms CLIP and VC-1 (except on Adroit Relocate where CLIP anomalously leads), confirming that diffusion model representations—even when used unconditionally—are stronger for control than ViT-based representations from contrastive or MAE pre-training. Among task-adaptive text-based methods, TADP is the strongest, followed by CoOp, with Text (Caption) and Text (Simple) roughly tied. However, the gap between the best text-conditioned method (TADP) and the unconditional baseline (SCR) is modest: 70.7 vs. 68.3 on DMC, 93.1 vs. 90.4 on MetaWorld, and actually negative on Adroit (57.3 vs. 58.0). This confirms the paper's central claim that text conditioning provides inconsistent and limited gains in control environments. ORCA's improvements over TADP are consistent across all three benchmarks (3.6 DMC, 2.1 MetaWorld, 8.0 Adroit), suggesting the visual prompt mechanism provides benefits that text-based conditioning cannot replicate.

Per-Task Analysis Reveals Divergent Behavior of Conditioning Methods

The aggregate means mask substantial per-task variation that is crucial for understanding where different methods succeed and fail:

Walker-walk (DMC, Table 1): ORCA achieves 76.9, dramatically outperforming all baselines: SCR at 64.3 (+12.6), TADP at 69.9 (+7.0), CoOp at 67.8 (+9.1). This is the largest per-task improvement on DMC. Walker-walk is a locomotion task requiring the bipedal agent to walk forward while maintaining balance—the visual state changes continuously as the agent moves, making frame-sensitive visual prompts particularly valuable.

Cheetah-run (DMC, Table 1): ORCA achieves 50.0, compared to SCR at 43.4 (+6.6), TADP at 41.1 (+8.9), CoOp at 45.0 (+5.0). Importantly, the Text (Caption) baseline degrades performance to 37.5—5.9 points below the unconditional SCR baseline—which is the clearest quantitative evidence for the paper's claim that text conditions can actively harm performance. ORCA not only avoids this degradation but achieves the highest score among all methods.

Reacher-easy (DMC, Table 1): ORCA achieves 87.6, a modest improvement over CoOp (87.1) and TADP (86.6). This is a task where text conditions already work reasonably well (Text Caption achieves 86.2 vs. SCR's 81.8), suggesting the visual grounding problem is less severe for this environment (a simple robotic arm against a checkerboard background may be more similar to natural images than MuJoCo's minimalist agent designs).

Finger-spin (DMC, Table 1): ORCA achieves 68.0, only marginally better than SCR's 66.6 (+1.4) and essentially tied with the other baselines (all in the 65-68 range). This is the DMC task where ORCA provides the smallest benefit, suggesting that the fine-grained visual dynamics of a spinning object may require different types of features than what ORCA's prompts emphasize, or that the policy learning bottleneck lies elsewhere for this task.

Button-press (MetaWorld, Table 1): ORCA achieves 88.0%, a substantial improvement over SCR at 74.7% (+13.3) and TADP at 80.0% (+8.0). This aligns with the cross-attention analysis in Figure 3(b), which showed that text conditions for Button-press (where "button" and "press" ground well) already help (Text Simple achieves 78.7%, Text Caption 80.0%), but ORCA's combined task and visual prompts push performance substantially higher by providing more precise spatial grounding.

Relocate (Adroit, Table 2): ORCA achieves 44.0%, tied with CLIP and 12 points above SCR (32.0%). However, the standard deviations are large (CLIP: 44.0 ± 4.0; ORCA: 44.0 ± 4.0), and CLIP's performance on this task appears to be an outlier among task-agnostic methods (VC-1 achieves only 29.3%). The improvement over SCR is genuine but the absolute performance remains low, indicating that Relocate—which requires a 28-DoF anthropomorphic hand to pick up a ball and move it to a target—is near the ceiling of what the current demonstration scale (2 demos) and visual representation can support.

Pen (Adroit, Table 2): ORCA achieves 86.7%, compared to SCR at 84.0% (+2.7). This is a dexterous manipulation task requiring the hand to reorient a pen to match a target pose. The improvement is modest but consistent across seeds.

Comparison to Fine-Tuning Approaches (Table 3)

Table 3 provides the critical comparison between ORCA's conditioning approach and alternative methods for achieving task-adaptivity through weight modification:

• Full fine-tuning of SCR: Success rates collapse to 9.3% on Adroit (from 58.0% frozen), confirming that updating all 346.7M U-Net parameters on limited imitation learning data destroys the pre-trained representations. Individual task breakdown: Pen drops from 84.0% to 17.3%, Relocate drops from 32.0% to 1.3%. This is a nearly complete failure of policy learning.

• Full fine-tuning of VC-1: Drops from 47.3% to 31.3%, a less severe but still substantial degradation. The ViT architecture may be somewhat more robust to fine-tuning than the U-Net, but the result still validates the paper's premise that weight modification on small control datasets is harmful.

• Parameter-efficient fine-tuning (RoboAdapter on VC-1): Achieves 59.3%, recovering beyond the frozen VC-1 baseline (47.3%) and slightly exceeding frozen SCR (58.0%). However, this requires 18.0M learnable parameters—more than ORCA's 10.6M.

• Parameter-efficient fine-tuning (LoRA on SCR): Achieves 60.0%, matching RoboAdapter performance with only 4.6M learnable parameters. This is the strongest fine-tuning baseline, but still falls 5.3 percentage points below ORCA.

• ORCA: Achieves 65.3% with 10.6M learnable parameters. The comparison to LoRA on SCR (60.0% vs. 65.3%) is the most direct test of the conditioning-vs-fine-tuning hypothesis, since both modify SCR's effective behavior—LoRA by adapting the U-Net weights, ORCA by changing the conditioning input. ORCA's advantage confirms that preserving the frozen weights while steering features through conditioning is more effective than even parameter-efficient weight adaptation.

The paper notes that full fine-tuning requires "approximately 30× more parameters but proves ineffective," while ORCA "significantly boosts performance while maintaining a comparable parameter count, highlighting the superior efficiency and efficacy of our condition-based approach." The 30× figure compares SCR full fine-tuning (346.7M) to ORCA's learnable parameters (10.6M ≈ 32.7× less).

Ablation Studies and Robustness Checks

Component analysis of task and visual prompts (Table 4): The paper ablates each prompt type individually on the five DMC tasks. Starting from the unconditional SCR baseline (68.3 mean): adding only task prompts achieves 69.8 (improvement of +1.5), adding only visual prompts achieves 70.5 (+2.2), and adding both achieves 74.3 (+6.0). The combined gain exceeds the sum of individual gains (1.5 + 2.2 = 3.7 < 6.0), indicating positive interaction between the two prompt types. The per-task pattern reveals that the prompts' relative contributions vary by task: on Walker-walk, task prompts alone underperform the baseline (71.4 + task vs. 64.3 baseline? wait—no, the baseline row shows 64.3 for Walk, and task-only shows 71.4, which is +7.1, a large gain), while visual prompts alone achieve 71.1 (+6.8). This suggests Walker-walk benefits from both types roughly equally. On Cheetah-run, task prompts alone degrade performance (38.9 vs. 43.4 baseline, -4.5), visual prompts alone also degrade slightly (42.0 vs. 43.4, -1.4), but combined they improve to 50.0 (+6.6). This is the most dramatic example of synergy: neither prompt type alone helps Cheetah-run, but together they provide complementary information that enables substantial improvement.

Layer selection for feature extraction (Table 5): Evaluating each U-Net block individually on DMC, the paper finds that downsampling layers consistently outperform upsampling layers: down1 achieves 68.4 mean, down2 65.1, down3 67.5, mid 64.6, up0 59.0, up1 60.4, up2 49.7. The performance monotonicity along the upsampling path (up2 < up1 < up0) suggests that features become increasingly specialized for reconstruction and less transferable for control as they approach the output. The combined "d + m" (down1-3 + mid) configuration achieves 74.3, substantially outperforming any single layer. The paper does not explore which specific combination of layers is optimal (e.g., down1+down3 vs. all down), nor does it ablate the compression layer architecture.

Choice of vision encoder for visual prompts (Appendix Table C): Testing alternatives to DINOv2 on Adroit: without any vision encoder (task prompts only) achieves 54.7% mean; with SigLIP achieves 55.3%; with CLIP achieves 63.3%; with SD-VAE (the VQGAN encoder already in SD 1.5) achieves 61.3%; with DINOv2 achieves 65.3%. All vision encoders improve over the no-encoder baseline, confirming that the visual prompt mechanism is robust to encoder choice. The gap between the best (DINOv2, 65.3%) and the worst (SigLIP, 55.3%) is 10 percentage points, indicating that encoder quality does matter but any reasonable vision encoder provides useful conditioning signals. The SD-VAE result (61.3%) is notable because it means the system could operate without any external vision encoder—the VQGAN is already part of SD 1.5—reducing the parameter and latency overhead of adding DINOv2.

Diffusion timestep selection (Appendix Table D): Testing $t = 0$, $t = 100$, and $t = 200$ on DMC, the default $t=0$ achieves the best mean (74.3), while $t=100$ drops to 68.5 and $t=200$ to 69.4. However, the per-task pattern is not uniform: at $t=200$, Walker-stand improves to 92.2 (vs. 89.1 at $t=0$) and Walker-walk improves to 78.6 (vs. 76.9), but Cheetah-run collapses to 24.7 (vs. 50.0 at $t=0$). This suggests that different tasks may benefit from different timesteps—locomotion tasks (Walker-stand, Walker-walk) may benefit from noisier features that capture motion-relevant structure, while Cheetah-run's dramatic degradation at higher timesteps suggests its features are particularly sensitive to the noise level. The paper selects $t=0$ as the default based on aggregate performance, but this is a design choice that could be optimized per-task in a more sophisticated system.

Different diffusion backbones (Appendix Table A): Testing Stable Video Diffusion (SVD) and Stable Diffusion 3.0 (SD 3.0, which uses a Diffusion Transformer architecture) on Adroit: SVD achieves only 26.0% mean (vs. 58.0% for SCR/SD 1.5), and SD 3.0 achieves 30.0%. The paper attributes SVD's poor performance to input discrepancy (SVD expects 8-frame clips but is fed 3 frames for fair comparison) and lack of text conditioning support in the publicly released version, which precludes the use of learnable prompts. For SD 3.0, the MM-DiT architecture underperforms U-Net-based models, and the paper notes that "empirical studies on DiT-based models remain limited compared to the extensive research on U-Net architectures." This result is important because it constrains the generality of the paper's findings: ORCA's conditioning approach is specifically validated on U-Net-based latent diffusion models, and may not transfer directly to newer DiT-based architectures without adaptation.

Stronger pre-trained visual representations (Appendix Table E): Comparing ORCA to recent state-of-the-art frozen encoders on DMC: DINOv2 (Oquab et al., 2023) achieves 52.5 mean, SigLIP (Zhai et al., 2023) achieves 60.8, Theia (Shang et al., 2024, a distilled vision foundation model built for manipulation) achieves 64.6, and ORCA achieves 74.3. The gap between the best task-agnostic encoder (Theia at 64.6) and ORCA (74.3) is 9.7 points, confirming that task-adaptive conditioning provides benefits beyond simply using a stronger frozen encoder. The per-task breakdown reveals the brittleness of task-agnostic representations: DINOv2 achieves only 22.5 on Reacher-easy (vs. ORCA's 87.6), and SigLIP achieves only 18.6 on Cheetah-run (vs. ORCA's 50.0). These dramatic per-task failures of otherwise strong encoders illustrate exactly the problem ORCA solves.

LIBERO-Long benchmark (Appendix Table B): On language-conditioned long-horizon manipulation tasks: in the single-task setting, ORCA achieves a mean success rate of 65.5% across 3 tasks (KITCHEN-3: 93.3%, LIVING-1: 48.3%, STUDY-1: 55.0%), compared to SCR at 57.2%, VC-1 at 43.3%, and SigLIP at 36.7%. In the multi-task setting (jointly training all 3 tasks with a shared policy), ORCA achieves 46.6%, compared to SCR at 23.3%, VC-1 at 26.7%, and SigLIP at 16.7%. This demonstrates that ORCA's conditioning mechanism scales to more complex, language-conditioned, long-horizon tasks and that the visual and task prompts can be learned jointly from multiple tasks. The multi-task result is particularly important because it shows the prompt learning does not require per-task isolation—the model can learn different conditioning patterns for different tasks within a shared policy.

Efficiency analysis (Appendix Table F): ORCA's total parameter count is 480.1M (including frozen DINOv2 at 304M and frozen U-Net at ~383M, minus discarded up-sampling blocks), with 10.6M learnable parameters and inference latency of 48ms per frame. VC-1 uses 303.3M total parameters (all frozen), 0 learnable, 11ms latency. SCR uses 382.9M total (all frozen), 0 learnable, 26ms latency. The inference time increase from SCR (26ms) to ORCA (48ms) is 22ms (85% increase), driven by the DINOv2 forward pass and the additional conditioning computations. The paper does not frame this as a limitation and does not discuss whether the accuracy gains justify the latency increase for real-time control applications.

Critical Assessment

Does the paper demonstrate that text conditions fail for robotic control?

Yes, with strong evidence, but only for the specific model and environments tested. The quantitative evidence comes from Figure 3(a) and Table 1: on Cheetah-run, Text (Caption) achieves 37.5 compared to SCR's 43.4 (the unconditional baseline), a degradation of 5.9 points. On Finger-spin, Text (Caption) achieves 65.1 vs. SCR's 66.6. On Reacher-easy, the gap is positive but small (Text Caption 86.2 vs. SCR 81.8). On MetaWorld, Text (Caption) slightly outperforms SCR on average (92.5 vs. 90.4), but this masks per-task variation. On Adroit, Text (Caption) matches SCR at 57.3 vs. 58.0.

The paper's interpretation—that text conditions "yield minimal or even negative gains" (abstract)—is accurate for the aggregate picture, but the evidence is more nuanced than the strong claim "text conditions fail." They fail on some tasks and marginally help on others. The paper does not quantify what fraction of tasks see degradation vs. improvement, nor does it establish statistical significance for the per-task differences given the standard deviations (Text Caption on Walker-walk: 68.3 ± 5.9 vs. SCR 64.3 ± 3.5—the confidence intervals overlap).

The cross-attention visualization evidence (Figure 3b,c) provides a mechanistic explanation for the failures, but only two tasks are visualized (Button-press where text helps, Cheetah-run where it hurts). The paper does not show cross-attention maps for the intermediate cases (e.g., Walker-walk, Assembly, Pen), which would strengthen the causal claim that grounding quality drives performance differences.

The paper attributes the failure to a "domain gap between real-world images and simulated control environments" (Section 4.1). This claim is plausible but untested—no experiment systematically varies the visual realism of the environment to test whether text conditioning improves as environments become more photorealistic. The LIBERO results (Appendix Table B) use a different simulator but still show ORCA substantially outperforming frozen baselines; however, the paper does not include a Text (Caption) baseline on LIBERO, so we cannot assess whether text conditions fail there too or whether the domain gap is MuJoCo-specific.

Does ORCA achieve state-of-the-art performance?

Yes, across all 12 tasks tested, but the framing of "state-of-the-art" requires qualification. ORCA outperforms all baselines on all three benchmarks, with mean scores of 74.3 (DMC), 95.2% (MetaWorld), and 65.3% (Adroit). The gains over the strongest baseline (TADP on DMC and MetaWorld, SCR on Adroit) range from 2.1 percentage points (MetaWorld vs. TADP) to 8.0 points (Adroit vs. SCR/CoOp). These improvements are consistent across tasks and seeds.

However, the set of baselines is not exhaustive. The paper compares against CLIP, VC-1, SCR, and text-conditioning variants, but does not compare against:

• Other pre-trained visual representations specifically designed for control beyond VC-1 and Theia. R3M (Nair et al., 2022) and VIP (Ma et al., 2022) are notable omissions.
• Non-diffusion prompt learning methods beyond CoOp. Methods like MaPLe (Khattak et al., 2023) that learn prompts at multiple transformer layers could be adapted to the U-Net architecture.
• Representations from larger/stronger diffusion models (SD 2.1, SDXL). The paper only tests SD 1.5, SD 3.0 (which underperforms), and SVD (which underperforms). SDXL's representations might be stronger and might benefit differently from conditioning.
• End-to-end trained policies (e.g., DrQ-v2, Dreamer) that don't use frozen pre-trained encoders—though the paper's focus is specifically on pre-trained visual representations, so this is a scope limitation rather than an omission.

The paper's claim to "state-of-the-art" is thus best interpreted as "state-of-the-art among methods that use frozen pre-trained visual representations with behavior cloning on these specific benchmarks," not a claim to being the best possible method for these control tasks by any approach.

The standard deviations are substantial for several tasks, raising questions about statistical reliability. On Adroit Relocate, ORCA achieves 44.0 ± 4.0, which overlaps with CLIP's 44.0 ± 4.0. On DMC Reacher-easy, ORCA achieves 87.6 ± 2.9, overlapping with CoOp's 87.1 ± 5.9. The paper does not perform statistical tests, so some of the "improvements" may not be statistically significant at conventional levels. With only three seeds, the power to detect moderate effect sizes is limited.

Does ORCA demonstrate that conditioning through prompts is superior to parameter-efficient fine-tuning?

Partially, with important caveats. Table 3 shows ORCA (65.3%) outperforming LoRA on SCR (60.0%) and RoboAdapter on VC-1 (59.3%) on Adroit. This is the most direct evidence for the conditioning-over-fine-tuning claim. The comparison is fair in terms of task and data, and the gap (5.3 points over LoRA) is meaningful.

However, the comparison has several limitations:

1. The LoRA and RoboAdapter configurations may not be optimally tuned. The paper does not specify what rank was used for LoRA, which layers were adapted, or what learning rate was used—all of which significantly affect LoRA performance. Similarly for RoboAdapter. Without evidence of hyperparameter optimization, it is possible that a better-tuned parameter-efficient fine-tuning approach could close or eliminate the gap.

2. The comparison is only on Adroit (2 tasks). DMC and MetaWorld fine-tuning results are not reported. The claim that conditioning is "superior" would be stronger if demonstrated across all three benchmarks.

3. The comparison does not control for total parameter count or architecture. ORCA adds a DINOv2 encoder (304M frozen parameters) that LoRA on SCR does not have access to. A fairer comparison might give LoRA-on-SCR access to the same visual information, perhaps by concatenating DINOv2 features with the U-Net features before the policy network. The current comparison conflates two factors: (a) conditioning vs. fine-tuning, and (b) access to additional visual features (DINOv2) vs. not.

4. The paper does not test combining ORCA's prompts with LoRA fine-tuning. An interesting ablation would be: does adding LoRA on top of ORCA further improve performance, or does it cause the same overfitting issues as fine-tuning the unconditional model? This would test whether the prompts provide sufficient task adaptation that fine-tuning becomes unnecessary, or whether there are residual gains from also adapting the weights.

Does the paper's two-component decomposition (task + visual prompts) reflect a genuine structural insight or an arbitrary design choice?

The ablation in Table 4 provides support, but the evidence is incomplete. The finding that task prompts alone (69.8) and visual prompts alone (70.5) both improve over the unconditional baseline (68.3), and that combined performance (74.3) exceeds the sum of individual improvements, is consistent with the paper's claim that the two components serve complementary functions. The cross-attention visualizations in Figure 6 provide qualitative support: task prompts consistently attend to static task-relevant objects, while visual prompts dynamically track scene elements.

However, several questions remain unaddressed:

1. Is the "task" vs. "visual" distinction the right decomposition, or would any increase in conditioning capacity produce similar gains? The paper does not compare against a baseline that simply uses more task tokens (e.g., 20 task tokens and no visual prompts) or more visual tokens (e.g., 20 visual tokens and no task prompts). If 20 purely task tokens achieved similar performance to 4 task + 16 visual, the decomposition claim would be weakened—the benefit might simply be from increased conditioning capacity rather than from the task/visual distinction.

2. The paper does not ablate the interaction mechanism. The task and visual tokens are concatenated and passed through the text encoder together, allowing self-attention to mix their information. An ablation where they are concatenated but passed through separate text encoder forward passes (no self-attention between them) would test whether the interaction is important or whether they operate independently.

3. The "synergy" interpretation (combined gain > sum of individual gains) assumes linear additivity of effects. This is not necessarily valid for deep networks with non-linear interactions. The observation that 1.5 + 2.2 < 6.0 could reflect superlinear scaling of conditioning benefit with total prompt length, rather than a specific synergy between task and visual information.

What experiments are missing that would strengthen the paper?

1. Systematic study of text conditioning failure modes. The paper shows that text conditions fail on some tasks but not others, and attributes this to domain gap. A controlled experiment varying environment visual realism (e.g., photorealistic renders vs. minimalist MuJoCo graphics, real robot images vs. simulation) would test this hypothesis directly. If text conditioning improves monotonically with visual realism, the domain gap explanation is supported. If not, the failure has other causes.

2. Scaling behavior with demonstration count. The paper uses 2-5 demonstrations per task, following VC-1 protocol. How does the benefit of ORCA's conditioning change with more demonstrations? With 1 demonstration, does conditioning help more (by providing additional task information) or less (because there's less data to learn good prompts)? With 50 demonstrations, does the gap between ORCA and frozen baselines shrink because the policy can compensate for weaker representations with more data?

3. Ablation of token lengths. The paper fixes $l_t = 4$ and $l_v = 16$ without ablating these choices. How does performance vary with 1, 2, 8, 16 task tokens? With 4, 8, 32, 64 visual tokens? Is there a point of diminishing returns? Does the optimal split between task and visual tokens depend on the task?

4. Visual prompt without the text encoder. The paper passes visual prompts through the CLIP text encoder before cross-attention. What if visual prompts were injected directly into the U-Net's cross-attention layers, bypassing the text encoder? This would test whether the text encoder's processing is beneficial for visual features or whether a more direct injection would work better.

5. Real-robot evaluation. All experiments are in simulation (MuJoCo, MetaWorld, LIBERO). The domain gap argument suggests that text conditioning fails because simulated environments differ from web images. Would text conditioning work better on a real robot with natural images? Would ORCA's advantage over text conditioning shrink in that setting?

6. Direct comparison to other conditioning architectures. The paper compares to CoOp and TADP as prompt-learning baselines, but does not compare to adapter-based conditioning methods (e.g., injecting learned feature offsets directly into U-Net layers rather than through cross-attention) or to feature concatenation methods (concatenating DINOv2 features directly with U-Net features before the policy network without any cross-attention conditioning). These would help isolate whether cross-attention-based conditioning is specifically beneficial or whether the gains come from simply providing additional visual features.

7. Interaction between prompt learning and policy learning dynamics. The task prompts and visual projection layer are learned jointly with the policy. How do the learning dynamics interact? Do the prompts converge early and then the policy learns to use them, or do they co-evolve? Does the behavior cloning loss provide sufficient signal to learn good prompts, or would auxiliary losses (e.g., attention regularization, contrastive objectives) help?

Summary of assessment

The experiments substantiate the paper's primary claims—that text conditions are unreliable for robotic control, that learned task and visual prompts can provide effective task-adaptive conditioning, and that this conditioning approach outperforms both frozen task-agnostic encoders and parameter-efficient fine-tuning—with quantitative evidence across 12+ tasks and three benchmarks. The component ablations (Table 4, Appendix Tables C, D) provide mechanistic support for the design choices, and the cross-attention visualizations offer qualitative insight into what the prompts learn.

The principal weaknesses are: (1) the text-conditioning failure analysis, while compelling for the two visualized tasks, is not systematically tested across all tasks and environments; (2) the comparison to fine-tuning methods, while favorable, may not use optimally-tuned baselines and conflates conditioning with access to additional visual features; (3) several obvious ablations (token counts, interaction mechanisms, demonstration scaling) are missing; (4) all results are simulation-only, leaving open the question of transfer to real-robot settings where the domain gap argument predicts text conditioning might actually work better; (5) the state-of-the-art claim, while technically accurate for the baselines tested, does not encompass several relevant comparison methods from the pre-trained representations literature.

The paper's contribution is solidly empirical: it convincingly demonstrates that a specific conditioning architecture works well on standard benchmarks, outperforming reasonable baselines. The broader claims about why it works (domain gap, complementary task/visual decomposition) are supported by the evidence presented but would benefit from the additional controlled experiments identified above.

6. Limitations and Trade-offs

Limitation 1: The Conditioning Approach Is Validated Only on U-Net-Based Diffusion Models, Not on Modern DiT Architectures

The assumption or constraint. ORCA's entire conditioning mechanism depends on the cross-attention layers in Stable Diffusion's U-Net architecture, which accept text-encoder outputs as keys and values. The paper acknowledges that its findings are architecture-specific. In Appendix A.1, when testing Stable Diffusion 3.0 (which uses a Multimodal Diffusion Transformer, MM-DiT, rather than a U-Net), performance collapses to 30.0% Adroit mean success rate, compared to 58.0% for SCR with SD 1.5 and 65.3% for ORCA with SD 1.5. The paper states:

"we find that the DiT-based model underperforms compared to U-Net-based models, regardless of whether text conditions are provided. Given that empirical studies on DiT-based models remain limited compared to the extensive research on U-Net architectures, we believe there is significant room for improvement with DiT-based models. Nonetheless, we leave the deeper exploration of video diffusion and DiT-based models as a direction for future work."

The paper also tests Stable Video Diffusion (SVD), which uses a U-Net but is designed for 8-frame video inputs, and finds it achieves only 26.0% Adroit mean—though the paper attributes this partly to input discrepancy (feeding 3 frames to a model expecting 8).

The consequence. The diffusion model landscape is shifting rapidly toward DiT-based architectures. SD 3.0, Flux, and other recent models use transformer backbones with different conditioning mechanisms. ORCA's approach—learning prompt tokens and injecting them through a text encoder into cross-attention—may not transfer cleanly to DiT architectures, which use joint attention over image and text tokens rather than separate cross-attention layers. A practitioner using SD 3.0 or a DiT-based backbone cannot directly apply ORCA; the method would need to be re-derived for the new conditioning pathway. This constrains the shelf-life and generality of the approach.

What evidence exists in the paper. Appendix Table A provides the only evidence: SD 3.0 (MM-DiT) achieves 30.0% on Adroit, compared to 58.0% for SD 1.5 (U-Net) under the same base conditioning (null prompt, equivalent to SCR). This is not merely a failure of ORCA's prompts—the base unconditional representation from SD 3.0 is substantially worse than SD 1.5 for control tasks, suggesting the entire feature extraction paradigm (which ORCA inherits) is U-Net-specific.

Mitigation status. The paper does not attempt to adapt ORCA for DiT architectures. It acknowledges the gap and defers to future work. The core contribution—the idea of learning task and visual prompts—may be architecture-agnostic, but the specific implementation depends on U-Net cross-attention and the CLIP text encoder pathway. Adapting to DiT would require identifying the analogous injection point for conditioning signals.

Limitation 2: The Gains Over Text Conditioning Are Attributed to Domain Gap, but This Hypothesis Is Not Directly Tested

The assumption or constraint. The paper's central explanatory claim is that text conditions fail because of a "domain gap between the diffusion model's training data and robotic control environments" (Section 1), and that cross-attention grounding quality is the mechanism. The evidence for this is: (1) cross-attention visualizations for two tasks (Button-press where text helps and Cheetah-run where it hurts) show correlated grounding quality and performance, and (2) text captions from a state-of-the-art VLM (Gemini 2.5) sometimes underperform task-name-only prompts and the null condition. However, the paper does not systematically test the domain gap hypothesis by varying the visual realism of environments and measuring how text conditioning effectiveness changes.

The consequence. Without direct tests of the domain gap mechanism, it is unclear whether text conditioning fails because of: (a) the specific visual appearance of MuJoCo/ MetaWorld environments (the domain gap explanation), (b) the nature of control tasks requiring frame-level detail that language fundamentally cannot provide (the dynamic granularity explanation), or (c) poor captioning by the VLM despite its general capabilities. The paper argues for both (a) and (b), but the experiments cannot distinguish them. If the failure is primarily due to domain gap, then text conditioning should work better on photorealistic simulators or real robot images. If it is primarily due to the dynamic granularity problem, then even perfect text descriptions would fail, and the visual prompt solution is necessary regardless of visual realism. This distinction has practical implications for whether text conditioning has any role in robotic control on more realistic platforms.

What evidence exists in the paper. The paper provides only two cross-attention visualizations (Figure 3b,c). It does not report cross-attention maps for tasks where text conditioning provides intermediate benefits (e.g., Walker-walk, Assembly), nor does it quantify the correlation between attention localization quality and performance improvement across all 12 tasks. The paper does not test text conditioning on photorealistic renderings of the same tasks or on real-world robot images. The LIBERO results (Appendix Table B) use a different simulator but do not include a Text (Caption) baseline, so the question of whether text conditioning works better in that visual domain is unanswered.

Mitigation status. The paper does not address this limitation. The domain gap explanation is presented as the primary diagnosis in Section 4.1 and in the abstract ("We attribute this to the domain gap..."), but the paper treats this diagnosis as established rather than as a hypothesis requiring direct experimental validation. Future work could test this by systematically varying environment visual fidelity and measuring text conditioning effectiveness, or by evaluating on real-robot setups where the visual domain is closer to the diffusion model's training distribution.

Limitation 3: The Inference Latency Penalty and Additional Parameter Cost Are Not Discussed as Trade-Offs

The assumption or constraint. ORCA adds substantial computational overhead compared to the frozen baselines it outperforms. From Appendix Table F: ORCA requires 48ms inference latency per frame, compared to 26ms for SCR (85% increase) and 11ms for VC-1 (336% increase). Total parameter count is 480.1M for ORCA, compared to 382.9M for SCR and 303.3M for VC-1. The learnable parameter count is 10.6M, which is modest but non-zero (compared to 0 for SCR, VC-1, and CLIP). The paper does not discuss whether these overheads are acceptable for real-time control applications, nor does it normalize performance by inference cost.

The consequence. For real-time robotic control, inference latency is a hard constraint—policies typically need to run at 10-50 Hz, meaning per-frame processing budgets of 20-100ms. At 48ms per frame, ORCA operates at approximately 21 Hz, which may be acceptable for many manipulation tasks but represents a meaningful reduction in control bandwidth compared to SCR (38 Hz) or VC-1 (91 Hz). For high-frequency control tasks (e.g., dynamic locomotion, dexterous manipulation requiring tight feedback loops), the additional latency could degrade closed-loop performance in ways not captured by the offline behavior cloning evaluation. The paper's simulation evaluation does not model control frequency effects—actions are computed and applied at the simulation's native timestep without latency constraints—so the reported success rates may overstate real-time deployability. Additionally, the 480.1M total parameters means roughly 1-2 GB of GPU memory for the vision pipeline alone, which may be prohibitive for on-device or embedded deployment.

The comparison to fine-tuning methods in Table 3 highlights this tradeoff indirectly: ORCA achieves 65.3% with 10.6M learnable parameters and 48ms latency, while LoRA on SCR achieves 60.0% with 4.6M learnable parameters and presumably similar latency to SCR (26ms + LoRA overhead). The paper frames ORCA as superior, but a practitioner optimizing for latency rather than accuracy might prefer LoRA-on-SCR's 60.0% at ~26-30ms over ORCA's 65.3% at 48ms. The paper provides no accuracy-latency pareto analysis.

What evidence exists in the paper. Appendix Table F provides the parameter counts and latency numbers. The paper does not analyze these numbers in the main text or discuss their implications. The 85% latency increase over SCR is presented as a factual measurement without interpretation.

Mitigation status. The paper does not address this limitation. There is no ablation of the latency sources (how much comes from DINOv2 vs. the text encoder forward pass vs. the U-Net forward pass), no experiment with smaller/faster vision encoders to trade off accuracy for speed, and no discussion of whether the visual prompt mechanism could be optimized (e.g., by running DINOv2 at a lower resolution, or by using the SD-VAE encoder which achieved 61.3% Adroit in Appendix Table C without an external vision model). The only partial mitigation is the Appendix C result showing that SD-VAE (already part of SD 1.5) can serve as the vision encoder, which would eliminate the DINOv2 overhead—but this configuration is not ORCA's default and its performance (61.3%) erases most of ORCA's advantage over baselines on Adroit.

Limitation 4: The Difficulty Estimation and Strategy Selection Problem Is Not Addressed—ORCA Trains One Prompt Set Per Task

The assumption or constraint. ORCA learns separate task prompts $p_t$ for each task, trained end-to-end with the policy on that specific task's demonstrations. The paper does not address the multi-task or zero-shot setting where a single model must handle multiple tasks or adapt to new tasks without per-task training. While Appendix Table B shows ORCA can learn prompts jointly from multiple LIBERO tasks (achieving 46.6% in the multi-task setting vs. 23.3% for SCR), this still requires training on all tasks simultaneously with separate task prompts per task, and does not address adaptation to novel tasks. The paper does not provide a mechanism for selecting or generating prompts for tasks not seen during training, nor does it explore whether task prompts transfer between related tasks.

The consequence. The practical deployment scenario for robotic control increasingly involves multi-task policies and zero-shot generalization to new tasks (as pursued by Vision-Language-Action models like RT-2 and OpenVLA). ORCA's current formulation requires collecting demonstrations and training a new policy with new prompts for each new task—exactly the trial-and-error, per-task engineering burden that the paper's motivation (Section 1) criticizes about task-agnostic frozen encoders. The paper argues that frozen encoders require "manual, task-by-task inspection" to find the best representation; ORCA replaces this with per-task prompt training, which may be similarly burdensome. A practitioner wanting to deploy one vision system across 50 manipulation skills would need to train 50 separate prompt sets and policies, or train a multi-task system with 50 distinct task prompts, with unclear scaling behavior.

What evidence exists in the paper. The main experiments (Tables 1, 2) train one policy per task with task-specific prompts. The multi-task LIBERO result (Appendix Table B) shows ORCA can be extended to multiple tasks (3 tasks jointly), but the paper does not explore: how many tasks can be jointly trained before prompt interference degrades performance; whether task prompts for different tasks learn orthogonal or overlapping attention patterns; whether prompts from one task can initialize prompts for a related task to reduce training time; or whether language descriptions could be used to select among pre-trained task prompts at inference time. The paper explicitly notes in Appendix A.8 that "we focus on efficient, vision-based specialist agents for robotic control tasks rather than generalist approach," positioning the limitation as a scope choice rather than an oversight.

Mitigation status. The paper acknowledges the specialist-vs-generalist tradeoff in Appendix A.8 but does not propose solutions for scaling to many tasks or for zero-shot adaptation. The discussion of VLAs (Vision-Language-Action models) in the same section acknowledges that adapting billion-scale models to new robot embodiments requires expensive fine-tuning, implicitly justifying the specialist approach—but this does not address the limitation within ORCA's own framework. Future work could explore whether task prompts can be conditioned on language instructions (e.g., learning a mapping from task descriptions to prompt parameters) or whether meta-learning across tasks can produce prompts that generalize to new tasks with few-shot adaptation.

Limitation 5: No Demonstration That Visual Prompts Capture Causal Task Dynamics Rather Than Correlation

The assumption or constraint. ORCA's visual prompts are derived from a frozen DINOv2 encoder applied to the current observation frame, projected through a learned convolutional layer, and optimized end-to-end via the behavior cloning loss. The cross-attention visualizations in Figure 6 show that different visual prompt tokens attend to different scene elements and that their attention patterns change over time—for example, $p_v^2$ attends to the table when the hand descends to grasp the ball, then shifts to the hand when it lifts off. The paper interprets this as evidence that visual prompts "capture task-relevant movements" (Section 6.5). However, these visualizations show correlation between attention maps and task phases, not causation—they do not demonstrate that the visual prompt tokens are actually driving the policy's action selection, nor that the specific attention patterns observed are necessary for performance.

The consequence. Without causal analysis, it is unclear whether ORCA's performance gains come from the mechanism the paper claims (visual prompts providing fine-grained, frame-specific state information that guides action selection) or from simpler effects: (a) the visual prompts add conditioning capacity (more tokens) that improves feature extraction regardless of their content; (b) the visual prompts inject DINOv2 features into the representation, and it is the DINOv2 features themselves (not the cross-attention conditioning mechanism) that provide the benefit; or (c) the visual prompts act as a regularizer on the task prompts during training, preventing them from overfitting to spurious correlations in the small demonstration dataset. The paper's missing ablation—comparing ORCA to a baseline that simply concatenates DINOv2 features with U-Net features before the policy without any cross-attention conditioning—would help distinguish these alternatives.

What evidence exists in the paper. The evidence is correlational: cross-attention visualizations show attention patterns that look interpretable, and ablations show that removing visual prompts hurts performance (Table 4: 74.3 → 69.8 DMC mean). But the ablation removes the entire visual prompt pathway, including the DINOv2 features, the projection layer, and the conditioning tokens. It does not isolate whether the performance drop is because frame-specific visual information is missing, or because the total conditioning capacity is reduced, or because the training dynamics change. The paper also does not report whether the attention patterns in Figure 6 are consistent across random seeds or whether different training runs learn qualitatively different attention strategies that achieve similar performance.

Mitigation status. The paper does not address the causation-vs-correlation distinction. The interpretation of Figure 6 as showing that visual prompts "capture task-relevant movements" assumes that attention patterns correspond to information use, an assumption that has been questioned in the attention interpretability literature (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). A causal test—e.g., ablating specific visual prompt tokens and measuring the effect on action predictions, or intervening on attention maps during inference—would strengthen the mechanistic claims but is not performed. The paper frames the visualizations as supporting evidence for its design rationale rather than as proof of mechanism, which is appropriate, but the limitation remains that the proposed mechanism (frame-specific visual conditioning capturing dynamic state) is not rigorously validated.

Limitation 6: All Evaluations Are in Simulation; No Evidence for Real-World Transfer

The assumption or constraint. All 12 primary tasks and the 3 LIBERO tasks are evaluated in simulation environments: MuJoCo (DMC, Adroit), MetaWorld, and LIBERO. The paper does not evaluate ORCA on a physical robot, nor does it test whether the learned task and visual prompts transfer when the visual domain changes (e.g., different lighting, camera positions, background textures). This is particularly significant given the paper's central claim about domain gap: if text conditioning fails because of a domain gap between web images and simulated environments, it is unclear how ORCA's learned prompts—trained on a specific simulator's rendering—would transfer to the real world, where the visual appearance of the same task could be substantially different.

The consequence. The paper's argument that text conditioning fails due to domain gap actually cuts both ways: if the gap between web images and simulation is large enough to break text grounding, then the gap between simulation and reality may be large enough to break ORCA's learned visual-grounding patterns. The task prompts are optimized to attend to task-relevant regions in the specific simulator renders they were trained on; if deployed on a real robot with different visual characteristics (realistic textures, lighting variations, camera noise, motion blur), there is no guarantee the learned attention patterns would generalize. The DINOv2 encoder provides some robustness (it was trained on real-world images and transfers across visual domains), but the learned projection layer and the interaction with the text encoder and U-Net cross-attention are optimized end-to-end on simulation data and may overfit to simulator-specific visual features.

What evidence exists in the paper. None. The paper does not include any real-world experiments, nor does it test sim-to-sim transfer (e.g., training on one simulator's rendering of a task and evaluating on a different simulator's rendering of the same task with different visuals). The LIBERO experiments (Appendix Table B) use a different simulator (LIBERO's environment) but still evaluate in simulation and train on simulation data. The paper does not discuss domain randomization, data augmentation, or other techniques that might improve transfer.

Mitigation status. The paper does not address the sim-to-real gap. While it is common for methods papers in robot learning to evaluate only in simulation (VC-1, SCR, and most baselines the paper compares against also report simulation-only results on these benchmarks), the paper's specific contribution—conditioning mechanisms for task-adaptive representations—raises questions about transfer that go beyond standard benchmarking concerns. The task prompts, in particular, are learned parameters that may encapsulate simulator-specific visual priors. The authors acknowledge a related limitation in Appendix C, noting that "our findings may not directly apply to diffusion models with different architectures," but do not discuss the simulation-to-reality gap. Future work would need to evaluate ORCA on real-robot platforms, test whether prompts trained in simulation transfer to reality, and explore whether techniques like domain randomization during prompt training could improve sim-to-real robustness.

7. Implications and Future Directions

How This Work Changes the Landscape

ORCA does not introduce a new architecture or learning algorithm; its contribution is diagnostic and methodological rather than technical. The paper reframes how the field should think about conditioning pre-trained models for robotic control—away from language as the default modality and toward learned, visually-grounded, frame-sensitive signals. This is a reframing with practical consequences, not a paradigm shift. The fundamental building blocks (Stable Diffusion feature extraction, behavior cloning, prompt learning) all exist in prior work. ORCA's value is in showing which combination works, why the obvious alternative (text conditioning) fails, and what design principles emerge from that failure.

The most impactful conceptual shift is the demotion of text conditioning from a portable solution to a domain-dependent heuristic. Prior to this work, VPD (Zhao et al., 2023) and TADP (Kondapaneni et al., 2024) had established a narrative: text prompts make diffusion representations task-adaptive, and this works across vision tasks. ORCA demonstrates that this narrative breaks at the boundary of robotic control environments—and provides a mechanistic diagnosis (cross-attention grounding failure from domain mismatch) rather than just an empirical observation. This changes the default approach for practitioners: starting with text conditioning and iterating on prompt engineering is likely wasted effort for control tasks; starting with learned, visually-grounded conditioning is the better bet.

This reframing also reconciles a latent tension in the pre-trained representations literature. Frozen encoders (VC-1, SCR, CLIP) are safe but task-agnostic, producing variable per-task performance that requires manual inspection to select the best backbone (Majumdar et al., 2023). Fine-tuning is task-specific but overfits catastrophically on small imitation learning datasets (SCR dropping from 58.0% to 9.3% on Adroit, quantified in Table 3). ORCA demonstrates a third regime—modifying the conditioning signal while preserving frozen weights—that achieves task-adaptivity without the overfitting penalty. The concrete demonstration that this regime outperforms even parameter-efficient fine-tuning (LoRA on SCR: 60.0% vs. ORCA: 65.3% on Adroit, Table 3) provides an existence proof that conditioning can be more effective than weight adaptation for this problem setting, a result that was not obvious before this work.

The paper also shifts attention toward cross-attention as a general-purpose task-adaptation interface. The U-Net's cross-attention layers were designed for text-to-image generation, but ORCA demonstrates they can be repurposed to inject arbitrary conditioning signals optimized for entirely different objectives (action prediction). This suggests a broader principle: any pre-trained conditional model with cross-attention can potentially be adapted to new tasks by learning what to inject through its existing conditioning pathway, without architectural modification. This is a design pattern, not a theorem—it requires the conditioning pathway to have sufficient capacity and the pre-trained features to be sufficiently general—but it opens a research direction that extends beyond diffusion models and robotic control.

Which research directions become more attractive? Methods that learn conditioning signals end-to-end with task objectives, particularly for domains where language grounding is unreliable (simulation, specialized visual environments, domains with large appearance gaps from web images). The success of visual prompts—which inject frame-specific dense features through cross-attention—suggests that feature injection through pre-trained conditioning pathways is a broadly applicable technique. Research on better text prompting for control becomes less attractive, since the paper shows that even state-of-the-art captions (Gemini 2.5) fail to reliably improve over the null condition.

Which directions become less attractive? Incremental improvements to text-based conditioning for simulated control environments. The paper's evidence that text conditions produce inconsistent gains—sometimes helping (Button-press: +5.3 points), sometimes hurting (Cheetah-run: −5.9 points, Figure 3a)—and that the best text-based method (TADP) provides only modest aggregate improvements over the unconditional baseline (SCR: 68.3 → TADP: 70.7 on DMC, a +2.4 point gain vs. ORCA's +6.0) suggests that text conditioning faces fundamental limitations in these domains. Future effort is better spent on visual conditioning mechanisms or on hybrid approaches that use language only where grounding is reliable.

Follow-Up Research This Work Enables

Causally testing whether cross-attention grounding is the mechanism, not just a correlate. The paper's central diagnostic claim is that text conditions fail because the diffusion model cannot reliably associate words with image regions in simulated environments, and that ORCA succeeds because learned task and visual prompts achieve better grounding. The evidence is cross-attention visualizations (Figure 3b,c, Figure 6) showing qualitatively different attention patterns between successful and failing conditions. However, these are correlational. A direct causal experiment would: (1) train ORCA normally, (2) at inference time, intervene on the cross-attention maps by masking or perturbing specific prompt tokens' attention weights, and (3) measure the effect on action predictions and task success. If ablating the attention from visual prompt tokens that track dynamic elements (like $p_v^2$ in Figure 6, which follows the table and hand) disproportionately degrades performance on frames where those elements are task-relevant, that would establish causation. Conversely, if randomizing attention weights produces similar degradation regardless of which tokens are perturbed, the attention patterns may be epiphenomenal—the prompts might work through some other mechanism (e.g., adding conditioning capacity, regularizing the feature extraction) that correlates with but is not caused by specific attention patterns. This experiment matters because it determines whether future work should focus on improving attention grounding specifically (e.g., with attention regularization losses) or on the broader conditioning architecture.

Testing the domain gap hypothesis through controlled variation of visual realism. The paper attributes text conditioning failures to domain gap—the diffusion model's text-image grounding, trained on web photographs, transfers poorly to simulated environments. This hypothesis is plausible but untested. A strong follow-up would: take one or two control tasks where text conditioning hurts (e.g., Cheetah-run, Finger-spin), render them at multiple levels of visual fidelity (minimalist MuJoCo graphics, textures enabled, photorealistic rendering via a path tracer, and real-robot video if available), and measure text conditioning effectiveness at each level. If the text-conditioning performance gap (Text Caption minus SCR) narrows monotonically as visual realism increases, the domain gap hypothesis is supported. If text conditioning still underperforms the null condition even with photorealistic rendering, the failure may be due to the dynamic granularity problem (language fundamentally cannot capture frame-level action-relevant detail) rather than visual domain mismatch. This experiment would also inform whether ORCA's visual prompt advantage shrinks in photorealistic settings—if the domain gap is the primary problem, then better text grounding in realistic environments might make text conditioning competitive with visual prompts, changing the practical recommendation.

Disentangling conditioning capacity from conditioning content. ORCA concatenates 4 task tokens + 16 visual tokens for a total of 20 conditioning tokens. The paper does not ablate whether the performance gain comes from having 20 tokens' worth of conditioning capacity (regardless of content) or specifically from the task/visual decomposition. A clean ablation would: compare ORCA (4 task + 16 visual) against (A) 20 purely learned task tokens (no visual encoder), (B) 20 purely visual tokens (no learned task tokens, all tokens derived from DINOv2 features projected through separate learned layers), and (C) 20 randomly initialized, learnable tokens with neither task semantics nor visual grounding (a pure capacity baseline). If (A) and (B) both significantly outperform (C), then the content matters. If (A) and (B) perform similarly to the 4+16 split, the decomposition is not special—any reasonable allocation of conditioning capacity works. If (A) underperforms on tasks requiring dynamic tracking and (B) underperforms on tasks requiring consistent object attention, the decomposition is functionally meaningful. The outcome determines whether future work should focus on carefully designing what information goes into conditioning tokens or can simply scale up conditioning capacity with less structured approaches.

Combining ORCA-style prompts with parameter-efficient fine-tuning. The paper shows ORCA (65.3% Adroit) outperforms LoRA on SCR (60.0%) and that full fine-tuning of SCR is catastrophic (9.3%). But these comparisons treat conditioning and fine-tuning as alternatives. An obvious follow-up tests whether they are complementary: apply LoRA to the U-Net in addition to ORCA's learned prompts, training both the low-rank weight updates and the prompts jointly on the behavior cloning objective. If LoRA+ORCA achieves, say, 68% on Adroit, then conditioning and fine-tuning provide non-overlapping benefits and the optimal approach combines them. If LoRA+ORCA performs the same as ORCA alone (or degrades, as full fine-tuning does), then ORCA's prompts already extract the maximum useful task-adaptivity from the frozen backbone, and further weight modification is harmful. This experiment is particularly important because it tests whether the paper's "conditioning beats fine-tuning" finding is fundamental (the representation should not be modified at all) or contingent (the specific fine-tuning methods tested were suboptimal, but better ones could help). It also has practical implications: many practitioners default to LoRA for domain adaptation; knowing whether to combine it with prompt learning or to replace it entirely would guide engineering decisions.

Scaling behavior with demonstration count and task diversity. The paper uses 2-5 demonstrations per task, following the VC-1 protocol. This low-data regime is where frozen representations are most valuable (fine-tuning overfits) and where conditioning provides the largest relative benefit (since the policy has limited data to learn task-relevant features from scratch). An important open question is how ORCA's advantage scales: with 1 demonstration, does conditioning help even more (by providing task structure when behavioral data is extremely scarce) or less (because the prompts themselves cannot be learned well from a single trajectory)? With 50 demonstrations per task, does the gap between ORCA and frozen baselines shrink because the policy can compensate for weaker representations with more behavioral data? Similarly, how does training on many tasks jointly (10, 20, 50) affect prompt quality? The multi-task LIBERO result (Appendix Table B: 46.6% on 3 tasks) is promising but limited in scale. A systematic scaling study across demonstration counts and task counts would establish the regime where ORCA's conditioning provides the highest return on investment, guiding practitioners on when to adopt the approach vs. when simpler frozen encoders suffice.

Extending the conditioning framework to instruction-following and goal-conditioned settings. ORCA learns fixed task prompts per task, making it suitable for specialist policies but not for settings where a single policy must handle varied language instructions or goal specifications. A natural extension would replace the learned task prompt with a prompt generated from a language instruction or goal image. For language-conditioned control: encode the instruction with a language model, project it into the text encoder's embedding space, and use it as the task prompt for that episode—while keeping the visual prompt mechanism for frame-level conditioning. For goal-image-conditioned control: encode the goal image with DINOv2 and use its features as the task prompt. This would combine the dynamic, frame-sensitive benefits of visual prompts with the flexibility of instruction-following, potentially making ORCA's conditioning framework applicable to the generalist policy setting. The multi-task LIBERO result (46.6% with jointly trained prompts) suggests the infrastructure can handle multiple tasks, but the paper does not explore whether prompts can be generated rather than retrieved. A strong experiment would: train a multi-task policy with language-conditioned task prompts on LIBERO-100, compare to a retrieval-based baseline (looking up the closest training task prompt), and measure zero-shot generalization to held-out task instructions.

Practical Applications and Downstream Use Cases

Efficient specialist policies for simulation-to-real transfer with limited per-task data. In industrial robotics settings where a manipulator needs to learn a new task from a handful of human demonstrations, ORCA offers a concrete recipe: use a pre-trained Stable Diffusion backbone, learn task and visual prompts jointly with a behavior cloning policy on the few available demos, and deploy without fine-tuning. The paper's results on MetaWorld (95.2% mean success from 5 demonstrations per task) and Adroit (65.3% from 2 demonstrations) suggest this is feasible for manipulation tasks of moderate complexity. The key practical advantage over the current standard (pick a frozen encoder, hope it works, try another if it doesn't) is that ORCA adapts the representation to the specific task automatically through the prompt learning, eliminating the manual trial-and-error of encoder selection that the paper identifies as a pain point (Section 1: "determining which representation performs best often requires manual, task-by-task inspection"). The 10.6M learnable parameters train with standard behavior cloning; no specialized RL infrastructure is needed.

Cost-efficient visual perception for simulated robotics research at scale. For academic labs and industry research teams running large-scale simulation experiments (e.g., training populations of agents across many environments, benchmarking RL algorithms, conducting ablations over environment parameters), the computational cost of the vision pipeline is a meaningful line item. ORCA's 48ms inference latency (Appendix Table F) is higher than the frozen baselines (26ms for SCR, 11ms for VC-1), but the accuracy gains are substantial. A research team running 10,000 evaluation episodes across 5 DMC tasks might find that ORCA's 74.3 mean score reduces the number of experimental reruns needed to achieve statistical significance, offsetting the per-frame latency cost. More importantly, the consistency of ORCA's gains across tasks (unlike frozen encoders, which have per-task weaknesses) means researchers can use a single vision pipeline for all tasks rather than switching encoders per task, simplifying experimental infrastructure and reducing the risk of confounds from encoder choice.

Bootstrapping vision systems for low-cost robot platforms with limited onboard compute. The paper's finding that SD-VAE—the VQGAN encoder already inside SD 1.5—can serve as the visual prompt encoder with competitive results (61.3% Adroit mean vs. 65.3% with DINOv2, Appendix Table C) is practically significant. It means ORCA's conditioning framework can be deployed without an external vision encoder, using only the components already in the diffusion model. A low-cost robot with a Jetson-class GPU could run the SD 1.5 VQGAN encoder, the diffusion U-Net feature extractor (with upsampling blocks discarded to save 500M parameters, as noted in Appendix A.6), and the learned prompts, all within a unified model. While the 48ms latency reported in the paper includes DINOv2, removing it would bring latency closer to SCR's 26ms while preserving most of the conditioning benefit. This enables task-adaptive visual representations on hardware that cannot afford a separate ViT-L encoder, expanding the reach of pre-trained vision for robotics to lower-cost platforms.

When to Prefer This Method

The paper does not present an explicit, quantitative decision rule for choosing ORCA over named alternatives, and the experimental design does not systematically vary conditions (demonstration count, task complexity, visual realism) to map out the boundaries of ORCA's advantage. The comparisons to fine-tuning (Table 3) and to frozen encoders (Tables 1, 2) are point evaluations at fixed data scales and task suites. The paper's contribution is demonstrating that ORCA can achieve state-of-the-art results on standard benchmarks, not characterizing when it will outperform specific alternatives. A forced "prefer X when..." matrix would require speculation beyond what the paper's experiments support. The paper's practical contribution is better understood as: on the specific benchmarks and data scales tested (2-5 demonstrations, MuJoCo/MetaWorld/Adroit tasks, SD 1.5 backbone), ORCA achieves the best reported performance among methods that use frozen pre-trained visual representations with behavior cloning. For practitioners working in similar regimes, ORCA is the strongest available option based on published comparisons. For regimes outside this envelope (many demonstrations, real-world deployment, DiT backbones, instruction-following policies), the applicability is untested.