ArXiv: 2602.03973
🎯 Pitch
Pretrained imitation-learning policies fail not from missing motor skills, but from an inability to compose them under shifted spatial constraints—like placing a cup near a table edge instead of its center. VLS solves this at inference time without any fine-tuning by using vision–language models to synthesize rewards that steer frozen diffusion or flow-matching policies, achieving a 31% absolute gain on CALVIN long-horizon tasks.
1. Executive Summary
This paper proposes Vision–Language Steering (VLS), a training-free framework that adapts frozen diffusion or flow-matching robot policies to out-of-distribution (OOD) observation–language inputs at inference time without modifying policy parameters. VLS treats adaptation as an inference-time control problem, grounding OOD inputs into geometric keypoints via SAM and DINOv2, then using vision–language models to synthesize stage-wise differentiable programmatic reward functions that steer the denoising process toward constraint-satisfying trajectories, combining gradient-based refinement with Feynman–Kac resampling (gradient-free particle resampling based on reward-weighted potentials) and RBF repulsion (pairwise-distance-based diversity forcing between action proposals). Across simulation benchmarks CALVIN and LIBERO-PRO, VLS achieves a 31% absolute improvement on CALVIN long-horizon tasks over prior steering methods and a 13% gain on LIBERO-PRO over frozen VLA policies including OpenVLA and π₀, while real-world Franka deployment demonstrates robust adaptation under appearance, position, and object shifts—establishing that inference-time steering can substitute for retraining only when the required motor behaviors already exist in the base policy's training distribution and the challenge is selectively composing them under altered spatial structure.
2. Context and Motivation
The Core Problem: Policies Fail Under Mild Test-Time Shifts They Should Handle
The paper addresses a specific and practically important failure mode in modern robot learning: pretrained imitation-learning policies fail catastrophically when the test-time observation or language instruction deviates from the training distribution, even when the required motor skills are already present in the policy's repertoire. The authors motivate this through a human analogy (Section I):
"Once a child learns to place a cup at the center of a table, they have not merely mastered a single task. The same motor skill generalizes to placing the cup near an edge, atop a stack of books, or inside a crowded cabinet."
For robots, however, state-of-the-art diffusion and flow-matching policies can succeed at "place the red cube at the center of the table" but fail when asked to "place the red cube near the table's edge"—despite having executed reaching, grasping, and placing motions thousands of times during training. The motor primitives exist, but the policy cannot selectively compose them under the new spatial constraint.
This matters because real-world deployment inevitably involves distribution shift. A robot trained in a specific table configuration will encounter different object layouts, changed backgrounds, or slightly reworded instructions. The standard responses—retraining with more data or fine-tuning on the new scenario—are characterized as costly and conceptually misaligned:
"Addressing such failures through retraining or fine-tuning is costly and conceptually misaligned, as it attempts to relearn behaviors rather than control their execution."
The core intellectual framing is that these failures represent an inference-time control problem, not a skill-learning problem. The policy already knows how to reach, grasp, and place—it just needs a mechanism to modulate how those primitives are instantiated under the test-time spatial constraints.
Why This Problem Is Important
The significance operates on three levels:
Practical deployment. If every mild environmental shift requires retraining, scaling robot deployment becomes intractable. A policy trained in one lab might fail on a different table in the same building—not because the task is fundamentally different, but because the spatial configuration is unfamiliar. VLS offers a path to bridge this gap without data collection or parameter updates.
Architectural separation of concerns. The paper argues for a design principle where skill execution is decoupled from task specification. The base policy provides reusable motor primitives, and an orthogonal steering mechanism enforces test-time constraints. This is important beyond any single method: if successful, it means robot policies can be trained once on broad motor skill data and then adapted to specific task requirements at inference time through lightweight, training-free intervention.
Theoretical framing of imitation learning brittleness. The paper identifies OOD brittleness not as a lack of capability but as an artifact of how imitation learning couples action generation to training-specific spatial correlations (Section III-A):
"Since the base policy tends to overfit on the spatial and semantic correlations present in the training manifold, it exhibits severe brittleness when faced with such OOD scenarios."
This reframes the problem from "the policy cannot do X" to "the policy cannot translate do X under Y into appropriate motor commands when Y was not seen during training." The distinction is important because it implies that test-time intervention—not additional training—is the correct conceptual tool.
Where Prior Approaches Fall Short
The paper identifies five categories of prior work, each with specific limitations:
1. Pure imitation-trained policies (no adaptation). Large-scale VLA models like OpenVLA, , and achieve strong in-distribution performance but exhibit sharp degradation under OOD conditions (Section II-A). The paper's experiments confirm this on LIBERO-PRO (Table I): these pretrained VLAs, despite using VLM backbones for perception and language understanding, struggle when tested under position perturbations (objects moved) or task perturbations (instructions changed). The authors attribute this to post-training entanglement:
"post-training on robot data entangles spatial reasoning with specific training contexts, effectively degrading the VLM's generalization ability when the execution environment deviates from the training manifold."
This is a critical insight: even models with strong visual and language generalization capabilities lose that flexibility after being fine-tuned on specific robot demonstration data, because the fine-tuning process inadvertently couples spatial understanding to the specific configurations seen during training.
2. VLM-based scene understanding with re-optimization. Methods like VoxPoser and ReKep use VLMs to generate scene representations that improve spatial understanding, then re-optimize actions online via planning, search, or iterative refinement (Section II-B). These can handle unseen observations but:
"they typically require rollouts, repeated evaluation, or online optimization loops, which are computationally heavy and often incompatible with real-time control."
Moreover, they shift the burden of generalization entirely to optimization at deployment. VLS instead retains the pretrained policy as the skill prior and applies lightweight steering—a fundamentally different allocation of responsibility between training and inference.
3. Value/critic-guided steering. V-GPS re-ranks actions using an offline-learned value function; VGD injects gradients from a learned value/Q model into denoising (Section II-C). The key limitation: these methods learn an auxiliary objective separate from the base policy, which can effectively reshape the policy toward the critic's preferences. The authors explicitly reject this:
"we view this as undesirable as the base policy should remain the invariant, and only the test-time constraints should modulate execution."
The base policy represents the distilled motor expertise; steering should work with it, not overwrite it.
4. Dynamics/world-model guided steering. DynaGuide uses an external dynamics model to guide denoising; Latent Policy Barrier learns a dynamics model to predict and optimize future latent states (Section II-C). These increase dependence on predictive modeling and rollout-style evaluation:
"and can become sensitive to model error and inference cost as it pushes adaptation burden into heavier test-time optimization."
The learned dynamics model becomes a single point of failure—if it is inaccurate, guidance degrades.
5. Human/VLM-in-the-loop steering and verification. ITPS steers generative sampling through human interaction signals; FOREWARN and Do What You Say use VLMs as open-vocabulary verifiers to select among candidate plans (Section II-C). The unifying limitation:
"their supervision is typically discrete and sparse, which forces adaptation to occur through selection/rejection over candidates rather than through continuous, differentiable steering within generation, making them sample-inefficient when the desired behavior requires fine-grained constraint satisfaction."
This is a key distinction VLS exploits. Selection-based methods ("generate N candidates, pick the best") work when the desired behavior is well-represented in the candidate set. But under fine-grained spatial constraints (e.g., "place the cube 5cm from the edge"), the probability of randomly sampling a constraint-satisfying trajectory may be low, requiring exponentially many candidates. Gradient-based steering can push intermediate denoising steps toward constraint satisfaction, making it more sample-efficient.
How This Paper Positions Itself
VLS occupies a specific, underexplored intersection in this landscape (Section I):
"Our approach is inspired by inference-time steering techniques developed for large language models and image generation models, where a pretrained model's output distribution is reshaped to elicit desired behaviors without additional training. VLS extends this paradigm to robotics by treating action generation as a controllable denoising process."
The key positioning moves:
Training-free, parameter-free. Unlike value-guided methods that require learning auxiliary functions, or decorator methods that add residual policy networks, VLS modifies nothing. The base policy remains frozen. This makes it applicable to any diffusion or flow-matching policy without access to training data or the ability to fine-tune.
Dense, differentiable guidance. Unlike selection-based methods that provide discrete feedback, VLS provides trajectory-level gradients through the denoising process. This is what enables sample-efficient constraint satisfaction—the gradient signal propagates through the denoising chain, pushing all steps toward reward alignment rather than simply accepting or rejecting complete trajectories.
VLM-synthesized rewards, not learned rewards. Unlike critic-guided methods, VLS does not learn a value function. Instead, it uses VLMs to interpret the OOD input once per task stage and synthesize programmatic reward functions as differentiable PyTorch code. The VLM operates "off-graph"—it is queried to produce the reward code, but gradients only flow through the instantiated reward function, not through the VLM itself. This decouples spatial reasoning (which VLMs already do well) from gradient computation.
Grounded in geometric structure. The grounding step—using SAM for segmentation, DINOv2 for semantic features, and depth-based reprojection into 3D keypoints—converts the high-dimensional OOD input into a compact geometric scaffold . This is essential because it provides the variables over which the VLM can write programmatic rewards: distances, angles, containment relations, etc. Without this grounding, the VLM would need to reason about raw pixels to produce differentiable scores, which is both computationally expensive and unlikely to produce reliable gradients.
Combining gradient-based and gradient-free guidance. The paper explicitly distinguishes itself from purely gradient-based methods by incorporating Feynman–Kac resampling (gradient-free) and RBF repulsion for diversity. This combination addresses a subtle challenge: reward landscapes for spatial constraints (e.g., "close to the edge but not past it") can be multi-modal and non-convex. Pure gradient descent can get trapped in local optima; the resampling mechanism provides global exploration, while the gradients provide local refinement.
The Gap VLS Fills
The paper's central claim about the gap it fills can be stated concisely: existing methods either require learning (critics, dynamics models, residual policies), provide only discrete feedback (selection/verification), or are too expensive for real-time control (online optimization). VLS sits at a specific, underexplored intersection: training-free, dense guidance, synthesized programmatically from VLM reasoning, grounded in geometric structure, and deployed within the denoising loop itself.
The paper positions this as an extension of classifier guidance from image generation (Dhariwal and Nichol, 2021) to robotics, where the "classifier" is replaced by a VLM-synthesized reward function that scores how well an action trajectory satisfies spatial constraints expressed in the OOD input. The key innovation is not the steering mechanism itself but rather the pipeline that makes it work for robotics: how to go from an RGB-D image and a language instruction to a differentiable function over action space without any task-specific training.
3. Technical Approach
3.1 Reader Orientation
VLS is a pipeline that sits on top of an existing frozen robot policy (diffusion or flow-matching) and—at inference time only—takes an RGB-D image and language instruction that are out-of-distribution, interprets them to understand what spatial constraints they imply, and then intervenes inside the policy's denoising loop to push the generated action trajectory toward satisfying those constraints. The problem it solves is that pretrained imitation-learning policies fail when the test-time scene geometry or task wording differs from training, even though the required motor skills (reaching, grasping, placing) already exist in the policy; the solution "shape" is to treat the policy as an uncontrollable motor-primitive generator and add an external, training-free controller that biases the generation process using gradients from VLM-synthesized reward functions grounded in 3D geometric keypoints.
3.2 Big-Picture Architecture (Diagram in Words)
The system has five major stages, executed per environment timestep:
-
Input Grounding (Section IV-A1): Given the OOD RGB-D observation
$o_{OOD}$and language instruction$l_{OOD}$, a VLM identifies task-relevant objects; SAM segments them to produce masks; DINOv2 extracts semantic features on those masks; depth-based reprojection lifts masked pixels to 3D point clouds; clustering compresses these into a small set of 3D keypoints$\mathcal{P} = \{p_i\}_{i=1}^n$that anchor the spatial constraints. -
Reward Generation (Section IV-A2): The VLM is queried with the observation, instruction, and keypoint set to decompose the task into
$S$sequential stages and, for each stage$s$, produce a differentiable programmatic reward function$\mathcal{R}_s(\mathbf{a}_{t:t+T}^k, \mathcal{P})$implemented as PyTorch code that scores how well a proposed action trajectory satisfies that stage's spatial constraints. -
Diverse Proposal Initialization (Section IV-B1): At the start of denoising,
$B$action proposals are sampled from$\mathcal{N}(\mathbf{0}, \mathbf{I})$; a repulsive force based on pairwise inverse distances ($\nabla_{\mathbf{a}} \sum_{j \neq i} 1/(\|\mathbf{a}^k[i] - \mathbf{a}^k[j]\|_2 + \epsilon)$) is injected during early denoising steps to prevent premature mode collapse and maintain broad coverage of the action manifold. -
Guided Denoising Loop (Section IV-B2-3): For each denoising step
$k$(from$K$down to$0$), the current stage's reward gradient$g_s = \nabla_{\mathbf{a}} \mathcal{R}_s$is added to the noise/velocity prediction of the base policy (via classifier guidance, with multiple inner MCMC refinement steps), and a Feynman–Kac resampling step periodically reweights and resamples the particle population according to exponentiated reward potentials$G_i^k = \exp(\mathcal{R}_s(\mathbf{a}^k[i]))$. -
Closed-Loop Stage Control (Section IV-C): After each action chunk is executed in the environment, the achieved reward
$\mathcal{R}_s^t$is compared against baseline reward$\mathcal{R}_s^{base}$to adapt the guidance strength$\lambda_t$, and a Schmitt-trigger hysteresis mechanism determines whether to advance to the next stage, maintain, or reinforce the current stage based on reward thresholds$R_{high}$and$R_{low}$—enabling multi-stage task coordination under physical execution uncertainty.
Information flows as follows: $(o_{OOD}, l_{OOD})$ → grounding pipeline → keypoints $\mathcal{P}$ → VLM → reward functions $\{\mathcal{R}_s\}$ → gradient $g_s$ injected into denoising updates → $\nabla_{\mathbf{a}} \mathcal{R}_s$ biases the noise/velocity prediction → higher-reward action trajectories emerge → executed in environment → reward feedback loops back to update $\lambda_t$ and potentially trigger stage switching → next action chunk generated with updated stage and guidance strength.
3.3 Roadmap for the Deep Dive
- First, the formal problem setup (Section III): what OOD means mathematically, how diffusion and flow-matching policies work at the denoising level, and exactly how classifier guidance couples an external gradient to the denoising process—since the entire method rests on injecting
$\nabla_{\mathbf{a}} \log p((o,l)_{OOD} \mid \mathbf{a}^k)$into the base policy's updates, and this needs to be understood before the reward functions that approximate it can make sense. - Second, the grounding pipeline (Section IV-A1): how an RGB-D image and language instruction become a set of 3D keypoints via SAM, DINOv2, depth reprojection, and clustering—because the reward functions operate over these keypoints, so understanding what they represent is prerequisite to understanding the rewards.
- Third, reward function synthesis (Section IV-A2): how the VLM produces stage-wise, differentiable, programmatic reward functions from the keypoints, and why "programmatic" (PyTorch code) rather than learned or VLM-embedded matters for differentiability.
- Fourth, the denoising guidance mechanism (Section IV-B): the three interacting sub-mechanisms—RBF repulsion for diversity, gradient-based refinement via classifier guidance, and Feynman–Kac resampling for gradient-free global exploration—and how they combine to steer the particle population toward high-reward regions.
- Fifth, closed-loop execution control (Section IV-C): how adaptive guidance strength and Schmitt-trigger stage switching create a robust outer loop that handles physical uncertainty across multi-stage tasks.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a systems and methods paper whose core contribution is a training-free inference-time pipeline that grounds OOD robot-task specifications into differentiable reward functions and injects their gradients into the denoising process of frozen generative policies. The paper does not propose a new policy architecture or a new learning algorithm; it proposes a control framework that sits outside an existing policy and modulates its output distribution without touching its parameters.
The Formal Problem Setup: OOD Adaptation as Classifier Guidance
The paper formalizes the problem in Section III. Let the base policy $\pi^\star$ be a pretrained diffusion or flow-matching model with frozen parameters $\theta$, trained on an expert demonstration dataset $\mathcal{D}_{expert} = \{(o_i, \mathbf{a}_i), l_i\}_{i=1}^N$ to maximize the likelihood of action chunks $\mathbf{a}_{t:t+T}$ (horizon $T$ timesteps of robot actions—typically joint positions or end-effector poses) conditioned on observation $o$ (RGB images and proprioception) and language instruction $l$:
where the expectation is over the expert dataset, and the sum is over the $T$ action timesteps in the chunk. The optimization maximizes the log-probability the policy assigns to expert action chunks given the observation and instruction—this is standard behavioral cloning.
The failure arises at deployment when the policy encounters an $(o, l)_{OOD}$ pair not in the training distribution. The paper distinguishes two axes of OOD perturbation (Section V introduction): observation perturbations (adding unseen objects as distractors, changing object attributes, changing positions or orientations of task-relevant objects and support surfaces) and language perturbations (changing target objects or goal behaviors in the instruction). In either case, the spatial and semantic correlations that the policy learned during training no longer hold, causing action generation to produce trajectories that are inappropriate for the new configuration.
Classifier Guidance for Diffusion Policies
The paper adopts the classifier guidance framework from Dhariwal and Nichol (2021), originally developed for conditional image generation. The core mathematical operation is to modify the denoising process so that the generated sample not only follows the data distribution but also satisfies an auxiliary condition.
For a diffusion policy, the standard denoising step (Equation 2 in the paper) is:
where $\mathbf{a}_{t:t+T}^k$ is the action chunk at denoising step $k$ (with $k = K$ being pure noise and $k = 0$ being the clean action), $\alpha_k$ and $\bar{\alpha}_k$ are noise schedule coefficients (standard DDPM notation: $\bar{\alpha}_k = \prod_{i=1}^k \alpha_i$ is the cumulative product), $\epsilon(\cdot)$ is the noise prediction network, and $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ is injected noise.
With classifier guidance, the noise prediction is modified to incorporate a gradient from an external score function. The paper defines the guidance signal $g$ as:
where $g$ is the gradient with respect to the action proposal $\mathbf{a}_{t:t+T}^k$ of the log-likelihood that the OOD condition $(o, l)_{OOD}$ is satisfied given that action proposal.
What it computes: the direction in action space that most rapidly increases the probability (under some model) that the proposed action trajectory is consistent with the OOD observation-language pair. If the current action proposal would place the gripper far from the target object, the gradient points in the direction that would move the gripper closer. This is the core mathematical object that VLS must approximate.
For diffusion models, the modified noise prediction becomes (Equation 4):
where $\hat{\epsilon}$ is the steered noise prediction, $\epsilon(\cdot)$ is the original noise prediction from the frozen base policy, $\lambda$ is the guidance strength hyperparameter controlling how aggressively the steering is applied, $\sqrt{1-\bar{\alpha}_k}$ is a scaling factor derived from the noise schedule (ensuring the guidance magnitude is properly normalized relative to the denoising step's variance), and $g$ is the guidance gradient.
What it computes operationally: the original noise prediction $\epsilon$ estimates the noise that was added to the clean action to produce the current noisy sample $\mathbf{a}_{t:t+T}^k$; subtracting this noise (with appropriate scaling) denoises toward a clean action from the base policy's distribution. The guidance term $-\lambda \cdot \sqrt{1-\bar{\alpha}_k} \cdot g$ adds an additional bias: it pushes the denoising update in the direction of increasing $\log p((o,l)_{OOD} \mid \mathbf{a})$, meaning the resulting clean action will be more likely to satisfy the OOD condition. The negative sign is because $\hat{\epsilon}$ predicts the noise, and pushing the noise prediction in the direction of $+g$ would push the clean sample away from high-likelihood regions; subtracting $g$ from $\epsilon$ pushes the clean sample toward them.
Why this form: the scaling by $\sqrt{1-\bar{\alpha}_k}$ is theoretically motivated by the score-matching interpretation of diffusion models. In the DDPM framework, the noise prediction $\epsilon$ is related to the score function (gradient of the log data density) by $\epsilon(\mathbf{a}^k) = -\sqrt{1-\bar{\alpha}_k} \nabla_{\mathbf{a}^k} \log q(\mathbf{a}^k)$ where $q$ is the noisy data distribution. When adding an external gradient, the $\sqrt{1-\bar{\alpha}_k}$ factor ensures the guidance term is properly scaled with respect to the inherent denoising dynamics—without it, guidance would be too weak at early denoising steps (where $\bar{\alpha}_k$ is close to 0 and noise dominates) and too strong at late steps (where the signal is already clean).
Classifier Guidance for Flow-Matching Policies
For flow-matching policies (which model a continuous ODE rather than discrete denoising steps), the guidance takes a simpler form (Equation 5):
where $\hat{v}$ is the steered velocity field prediction, $v(\cdot)$ is the original velocity prediction from the frozen base policy, and $k$ now represents a continuous time $t \in [0,1]$ (with $k=1$ being noise and $k=0$ being the clean action) rather than a discrete step index.
What it computes: in flow matching, the velocity field $v$ defines the ODE $d\mathbf{a}/dt = v(\mathbf{a}, o, l, t)$ that transforms a sample from the noise distribution at $t=1$ to a clean action at $t=0$. The steered velocity $\hat{v}$ adds the guidance gradient $g$ directly, since in the continuous-time formulation there is no $\sqrt{1-\bar{\alpha}_k}$ scaling needed—the ODE integration naturally handles the timescale.
Why this form: the absence of the $\sqrt{1-\bar{\alpha}_k}$ factor is because flow matching parameterizes the transformation directly as a vector field rather than through a noise prediction objective. The gradient $g$ points in the direction of increasing conditional likelihood, and adding it to $v$ biases the flow toward that direction. The sign is positive (unlike the negative sign in the diffusion case) because $v$ directly parameterizes the direction of sample evolution rather than the noise to remove.
The Central Approximation
The paper notes (Section III-C) that $g = \nabla_{\mathbf{a}} \log p((o,l)_{OOD} \mid \mathbf{a}^k)$ is not directly computable in deployment because $p$ is unknown. The entire VLS framework is therefore organized around constructing a differentiable surrogate:
where $\mathcal{R}$ is a VLM-generated programmatic reward function. The guidance gradient is then approximated as $g \approx \nabla_{\mathbf{a}} \mathcal{R}(\mathbf{a}_{t:t+T}^k, (o,l)_{OOD})$.
The paper identifies two requirements for $\mathcal{R}$ (Section III-C): (1) it must correctly interpret the geometry and logical structure induced by the OOD condition, and (2) it must provide dense, informative gradients—meaning $\mathcal{R}$ must be a smooth function over the action space, not a discrete pass/fail signal, so that its gradient reliably points toward constraint satisfaction even far from the optimum.
OOD Input Grounding: From Pixels and Words to 3D Keypoints
The grounding pipeline (Section IV-A1) converts the high-dimensional OOD input $(o_{OOD}, l_{OOD})$ into a compact geometric scaffold $\mathcal{P} = \{p_i\}_{i=1}^n$ where each $p_i \in \mathbb{R}^3$ is a 3D coordinate anchoring a physically meaningful spatial constraint. This compression is essential because the VLM cannot reason about action-space gradients directly from raw pixels—it needs structured spatial variables (distances, containment regions, relative positions) over which to write differentiable reward functions.
Step 1: Object identification via VLM. A VLM (the paper does not specify which exact VLM in the main text, deferring to Appendix for prompt design) is queried with the observation image $o_{OOD}$ and instruction $l_{OOD}$ to identify which objects and regions are relevant to the manipulation task. The output is a set of object labels with corresponding referring expressions (e.g., "the red cube," "the table surface," "the drawer handle").
Step 2: Segmentation via SAM. For each identified object, the Segment Anything Model (SAM, Kirillov et al., 2023) is applied to produce binary segmentation masks $\mathcal{M}$ that isolate the object pixels in the image. SAM is chosen because it generalizes to unseen objects without fine-tuning—it is a promptable foundation model for segmentation that can segment arbitrary objects given point or box prompts, and here the VLM output provides those prompts (by specifying which objects to segment).
Step 3: Semantic feature extraction via DINOv2. Following the approach of ReKep (Huang et al., 2024), DINOv2 (Caron et al., 2021) is used to extract a patch-wise feature map $\Phi \in \mathbb{R}^{H \times W \times d}$ where $H$ and $W$ are the spatial dimensions of the feature map (downsampled from the image resolution) and $d$ is the feature dimension. DINOv2 is a self-supervised vision transformer that produces semantically meaningful dense features—pixels on the same object tend to have similar feature vectors, and corresponding parts across different objects (e.g., the rim of a cup vs. the rim of a mug) have related features. These features are filtered using the SAM masks $\mathcal{M}$, retaining only those feature vectors that correspond to each object of interest.
Step 4: 3D reprojection using depth. The masked pixels are reprojected into 3D using the depth channel of $o_{OOD}$ (which is an RGB-D image). Each pixel $(u, v)$ with depth $d$ is mapped to a 3D point $(x, y, z)$ in the camera frame using the camera intrinsics. Each resulting 3D point is represented by concatenating its DINOv2 feature vector (dimension $d$) with its 3D spatial coordinates $(x, y, z)$, yielding a $(d+3)$-dimensional representation. This representation jointly encodes what the point is (semantic identity via DINO) and where it is (spatial location via coordinates).
Step 5: Clustering into keypoints. The object-centric point clouds (all $(d+3)$-dimensional points belonging to an object) are clustered to obtain a small set of task-relevant keypoints $p_i \in \mathbb{R}^3$. The paper does not specify the clustering algorithm (likely k-means or DBSCAN, deferred to Appendix), but the key insight is that by including DINOv2 features in the representation, the clustering groups points that are both spatially proximal and semantically similar—for example, the handle of a mug forms a distinct cluster from the body of the mug even if they are adjacent in 3D space.
Design rationale for the grounding pipeline. The paper's choice to ground into 3D keypoints via SAM + DINOv2 + depth reprojection, rather than using the VLM directly to output spatial coordinates (as in VoxPoser) or training a keypoint detector, is motivated by three considerations. First, SAM and DINOv2 are foundation models that generalize to unseen objects without task-specific training, maintaining the training-free philosophy. Second, 3D keypoints provide a natural interface for writing differentiable reward functions—distances, containment tests, and alignment metrics are all straightforward to express as PyTorch operations over 3D coordinates. Third, the geometric scaffold $\mathcal{P}$ abstracts away irrelevant visual details (texture, lighting, background) and exposes only the spatial variables that matter for the task, making the VLM's job of writing reward functions tractable.
Reward Function Synthesis: VLM-Generated Differentiable Programmatic Rewards
Once the keypoint set $\mathcal{P}$ is established, the VLM is queried to synthesize reward functions (Section IV-A2). This is the core technical novelty of VLS—not the steering mechanism itself (which is standard classifier guidance), but the automated pipeline that produces the differentiable guidance signal without any task-specific training.
The VLM query. The VLM receives three inputs: (1) the observation image $o_{OOD}$ and language instruction $l_{OOD}$ (so it understands the task), (2) the keypoint set $\mathcal{P}$ with labels (e.g., $p_1$ is "red cube centroid," $p_2$ is "table edge"), and (3) a prompt instructing it to decompose the task into sequential stages and generate differentiable reward functions for each stage. The exact prompt is deferred to Appendix but the structure is described: the VLM identifies $S$ stages (e.g., for "pick up the red cube and place it near the blue plate": Stage 1—approach the red cube, Stage 2—grasp the red cube, Stage 3—move to the blue plate, Stage 4—release near the plate), and for each stage $s \in \{1, \dots, S\}$, produces a reward function:
where $\mathcal{R}_s$ is a differentiable PyTorch function that takes an action proposal $\mathbf{a}_{t:t+T}^k$ (a tensor of shape [T, action_dim] representing a trajectory chunk) and the keypoint set $\mathcal{P}$ as input, and returns a scalar reward value.
What the VLM generates, concretely. The paper constrains the VLM to output programmatic reward definitions—runnable PyTorch code composed of differentiable tensor operations. For example, for an approach stage, the VLM might generate a reward function that computes the negative L2 distance between the gripper position (extracted from $\mathbf{a}_{t:t+T}^k$ at the grasp timestep) and the target object centroid $p_{target}$, combined with a soft penalty for approaching from the wrong direction (computed via dot product with a preferred approach vector). The reward function is instantiated as actual Python code that operates on tensors; gradients flow through the code back to $\mathbf{a}_{t:t+T}^k$ via PyTorch's autograd.
Why programmatic rather than VLM-embedded. This is a crucial design choice. An alternative would be to have the VLM directly score action proposals (i.e., treat the VLM itself as the differentiable $\mathcal{R}$ by querying it with action text descriptions and backpropagating through the VLM). The paper explicitly avoids this: "the VLM itself remains a non-differentiable, off-graph component." The VLM is queried once per stage to generate the reward code; thereafter, gradients only flow through the instantiated PyTorch reward function, never through the VLM. This has three critical advantages:
- Computational efficiency: The VLM is queried
$O(1)$times per stage (at stage boundaries), not$O(B \times K)$times per denoising step (which would be$B$particles ×$K$denoising steps, making it far too slow for real-time control). - Gradient quality: VLMs are not designed to produce smooth gradients over continuous action spaces; their outputs are effectively categorical (token distributions) and backpropagating through them would produce noisy, unreliable gradients. Programmatic reward functions are analytically smooth (composed of differentiable operations like distances and dot products) and produce clean gradients.
- Modularity: The VLM handles the high-level reasoning (what constraints matter, in what order), while the reward function handles the low-level geometry. This separation matches what each component is good at.
Stage decomposition rationale. The paper decomposes tasks into stages because spatial constraints for different phases of a task are qualitatively different—approaching an object requires minimizing gripper-object distance, while placing near an edge requires maintaining a specific offset from the table boundary. A single monolithic reward function would need to encode the appropriate constraint based on the task phase, which is effectively what stage decomposition achieves but in a cleaner, modular way. The VLM is naturally suited to this decomposition because it can reason about task structure at a semantic level.
The formal connection to guidance. For each stage $s$, the reward gradient becomes:
What it computes: the gradient of the stage-specific reward with respect to each dimension of the action proposal. If the reward measures distance to a target, the gradient points in the direction that reduces that distance. If the reward measures alignment with a preferred orientation, the gradient rotates the end-effector pose accordingly. This gradient is what gets plugged into the classifier guidance equations (Equation 4 for diffusion, Equation 5 for flow matching).
Why this approximates $\log p((o,l)_{OOD} \mid \mathbf{a})$: The reward $\mathcal{R}_s$ is designed (by the VLM) to be high when the action trajectory satisfies the spatial constraints implied by the OOD condition and low otherwise. If the reward is well-designed, the gradient $\nabla \mathcal{R}_s$ points toward constraint satisfaction, which is exactly what $\nabla \log p$ would do if $p$ were a proper probabilistic model of constraint satisfaction. The approximation is that $\mathcal{R}_s$ is a hand-designed (by VLM) heuristic rather than a learned likelihood model, but the paper's results show it is effective in practice.
Diverse Proposal Initialization with RBF Repulsive Forces
Before reward-based guidance begins, the denoising process starts by sampling $B$ independent action proposals from pure noise (Section IV-B1):
where $B$ is the batch size (the paper sweeps this hyperparameter; Figure 3 right shows performance at $K=10$, though the notation $K$ in that figure refers to the batch size, not the denoising step count—this is an unfortunate notational collision in the paper). Each $\mathbf{a}_{t:t+T}^K[i]$ is an independent draw from a standard Gaussian, representing a completely uninformative initial action trajectory.
The paper introduces a diversity-promoting repulsive force during the early denoising steps, inspired by particle guidance methods (Corso et al., 2023) and tree-guided diffusion (Jeon et al., 2025). The repulsive gradient for the $i$-th particle at denoising step $k$ is:
where $\|\cdot\|_2$ is the Euclidean distance between two action proposals in the batch, $\epsilon$ is a small constant added for numerical stability (preventing division by zero when two proposals are identical), and the sum is over all other proposals $j \neq i$ in the batch of size $B$.
What it computes: for each particle $i$, the repulsive force pushes it away from every other particle $j$, with the force magnitude inversely proportional to the distance between them. Particles that are close together experience a strong repulsion; particles that are far apart experience negligible force. The gradient is taken with respect to $\mathbf{a}^k[i]$, so it directly modifies the action proposal during denoising.
What it enables: without this term, all $B$ proposals would follow similar denoising trajectories because they are all initialized from the same Gaussian distribution and processed by the same base policy. They would tend to collapse to the same mode of the base policy's distribution—likely an action trajectory that would work in the training distribution but fails under the OOD condition. The repulsive force ensures that the proposals spread out across the action manifold, providing diverse candidates for the subsequent reward-based selection and refinement.
Why this specific form: the $1/(d + \epsilon)$ potential (an inverse-distance kernel) is chosen because it creates strong local repulsion (particles cannot get too close) but weak long-range repulsion (particles that are already far apart are not artificially pushed further). Alternatives like a Gaussian kernel $\exp(-d^2/\sigma^2)$ would create both local and global repulsion, potentially pushing particles outside the support of the base policy's distribution entirely. The inverse-distance form respects the base policy's manifold—it prevents collapse to a single mode while keeping particles within the broader region where the base policy generates plausible actions.
Gradient-Based Refinement via Classifier Guidance
Once diversity is established, the core steering mechanism injects the stage-specific reward gradient $g_s = \nabla_{\mathbf{a}} \mathcal{R}_s$ into the denoising updates (Section IV-B2). This is the direct instantiation of the classifier guidance equations (Equations 4 and 5), with the reward gradient substituting for the true likelihood gradient.
Multiple inner MCMC updates per denoising step. To improve stability under potentially noisy reward gradients, the paper adopts stochastic refinement with multiple inner updates per denoising step, a technique analogous to MCMC-based guidance used in prior work on energy-based diffusion models (Du et al., 2023) and inference-time steering (Du and Song, 2025; Wang et al., 2024). Specifically, the paper sets $MCMC = 4$ for diffusion policies and $MCMC = 1$ for flow-matching policies (Algorithm 1). At each denoising step $k$, instead of applying the reward gradient once, the paper applies it $MCMC$ times in sequence:
For $m = 1$ to $MCMC$:
- Compute
$g_{reward}^k = \nabla_{\mathbf{a}_{t:t+T}^k} \mathcal{R}_s(\mathbf{a}_{t:t+T}^k, \mathcal{P})$ - Use
$g_{reward}^k$as the guidance gradient$g$in Equation 4 (diffusion) or Equation 5 (flow matching) - Update
$\mathbf{a}_{t:t+T}^k$according to the steered denoising step
The intermediate updates within a single denoising step allow the action proposal to explore the reward landscape more thoroughly before advancing to the next noise level. This is analogous to running Langevin dynamics within each denoising step, using the reward function as an energy function.
Design rationale for different MCMC values. The paper uses $MCMC = 4$ for diffusion but $MCMC = 1$ for flow matching. While the paper does not explicitly justify this choice, it likely reflects the different discretization properties of the two frameworks: diffusion uses discrete steps $K, K-1, \dots, 0$ with explicit noise injection at each step, making multiple inner updates beneficial because each step's noise can partially undo the guidance; flow matching integrates a continuous ODE, where multiple gradient injections per integration step might over-correct and destabilize the trajectory. The single gradient step in flow matching is applied continuously throughout the ODE integration rather than at discrete intervals.
Gradient-Free Resampling via Feynman–Kac Steering
In addition to gradient-based refinement, VLS employs a gradient-free resampling mechanism based on Feynman–Kac (FK) steering (Section IV-B3). The paper explicitly motivates this as complementary to gradient-based methods: reward landscapes for spatial constraints can be multi-modal and non-convex (e.g., there might be multiple valid grasp poses around an object, separated by invalid poses). Pure gradient descent can get trapped in a local optimum; the resampling mechanism provides global exploration by replicating promising particles and eliminating poor ones.
Particle potential computation. The $B$ action proposals $\{\mathbf{a}_{t:t+T}^k[i]\}_{i=1}^B$ are treated as an interacting particle system. For the $i$-th particle at denoising step $k$, a scalar potential is computed as:
where $\mathcal{R}_s$ is the stage-specific reward function, $\mathbf{a}_{t:t+T}^k[i]$ is the current (partially denoised) action proposal for particle $i$, and $G_i^k$ is the exponentiated reward.
What it computes: a non-negative scalar that is large when the particle's current action proposal achieves high reward (satisfying the stage constraints) and small (approaching 1 from above) when the reward is low or negative. The exponential map converts the raw reward into a multiplicative weight; particles with reward near zero get weight ~1, particles with strongly negative reward get weight ~0, and particles with strongly positive reward get large weights.
Weight normalization and resampling. The potentials are normalized to form a probability distribution over the particle population:
where $w_i^k$ is the normalized importance weight of particle $i$ at step $k$. Multinomial resampling is then applied: $B$ particles are drawn with replacement from the current population according to the weights $\{w_i^k\}$. High-weight particles are likely to be sampled multiple times (replicated); low-weight particles are likely to be dropped entirely.
What this achieves computationally: after resampling, the particle population is concentrated in high-reward regions of the action space. Particles that were in poor regions (low reward) are eliminated and replaced by copies of successful particles. This is a discrete, non-differentiable operation that complements the continuous gradient-based steering—gradients locally refine each particle, while resampling globally reallocates the particle budget toward promising regions.
Why Feynman–Kac specifically. FK steering is a sequential Monte Carlo (SMC) technique originally developed for filtering in state-space models. In the context of diffusion models, it has been formalized by Singhal et al. (2025) as a general framework for inference-time steering. The key property FK provides is that, under appropriate conditions, the resampled particle population converges to the target distribution $p(\mathbf{a} \mid (o,l)_{OOD})$ as the number of particles increases, even when the base policy's distribution is different. The resampling step "tilts" the transition kernel of the generative process—it modifies the effective dynamics so that the particles follow a distribution proportional to $p_{base}(\mathbf{a}) \cdot \exp(\mathcal{R}(\mathbf{a}))$ rather than just $p_{base}(\mathbf{a})$.
Why combine gradient-based and gradient-free guidance. The paper's key insight is that these two mechanisms address different failure modes:
- Gradient-based refinement is good at local fine-tuning: given a particle that is already near a constraint-satisfying region, gradients efficiently move it to the exact optimum.
- FK resampling is good at global exploration: given a diverse particle population, resampling identifies which particles are in promising basins and which are not, and reallocates computational budget accordingly.
Without gradients, the system would need impractically many particles to cover the action space densely enough that at least one lands in the constraint-satisfying region by chance. Without resampling, gradient descent would pull all particles toward the same local optimum, losing diversity and potentially missing better solutions. The combination is synergistic: resampling ensures the particle budget is spent exploring promising regions, and gradients refine particles within those regions.
Closed-Loop Execution Control and Stage Switching
The guidance mechanisms described above operate within a single action chunk generation. However, real robot tasks are executed across multiple timesteps and multiple stages, and physical uncertainty (object slippage, partial execution, environment changes) means that a stage may not complete as expected. The closed-loop control mechanism (Section IV-C) wraps the denoising guidance in an outer loop that adapts guidance strength and manages stage transitions based on execution feedback.
Adaptive Guidance Strength
Within a single task stage $s$, multiple action chunks $\{\mathbf{a}_{t:t+T}\}$ are generated sequentially (each chunk executed for $T$ timesteps before the next is generated with an updated observation). The guidance strength $\lambda_t$ is adapted for each action chunk $t$ based on how well the current execution is satisfying the stage constraints (Equation 10):
where $\lambda_{\max}$ is the maximum guidance strength (a hyperparameter), $\mathcal{R}_s^t$ is the reward value achieved by the action chunk generated at chunk index $t$ under stage $s$ (computed as the reward of the final denoising step for that chunk), and $\mathcal{R}_s^{base}$ is the reward value achieved by the first action chunk generated for this stage (which serves as a baseline).
What it computes: the ratio $\mathcal{R}_s^t / \mathcal{R}_s^{base}$ measures how well the current chunk's action satisfies the stage constraints relative to the initial chunk. If the current chunk achieves similar or higher reward than the baseline (ratio close to or above 1), then $1 - \text{ratio} \leq 0$, the sigmoid output is small (close to 0), and $\lambda_t$ is small—meaning the base policy is allowed to dominate because execution is on track. If the current chunk achieves much lower reward than the baseline (ratio close to 0), then $1 - \text{ratio} \approx 1$, the sigmoid output is large (close to 0.73 for sigmoid(1)), and $\lambda_t \approx 0.73 \cdot \lambda_{\max}$—meaning strong steering is applied to correct the deviation.
Why this schedule: the adaptive strength implements a coarse-to-fine control strategy. When the robot is far from satisfying the spatial constraints (e.g., the gripper is far from the target object), strong steering aggressively pulls the action toward the constraint. As the robot approaches the target (e.g., the gripper is near the object and just needs fine positioning), the steering relaxes and the base policy's learned motor control takes over. This prevents the reward gradient from continuing to push aggressively when the robot is already in the right region, which could cause overshooting or oscillation. The sigmoid nonlinearity ensures smooth transitions between high and low guidance regimes.
Schmitt-Trigger-Based Stage Switching
To robustly determine transitions between task stages while avoiding oscillatory behavior (constantly switching back and forth near a boundary), the paper adopts a hysteresis mechanism inspired by the Schmitt trigger, a classic electronics circuit used to convert noisy analog signals into clean digital transitions (Equation 11). For stage $s$, two reward thresholds are defined: $R_{high}$ (the "advance" threshold) and $R_{low}$ (the "reinforce" threshold), with $R_{high} > R_{low}$. The switching signal $Q_t$ is determined by comparing the current reward $\mathcal{R}_s^t$ against these thresholds:
What it computes: a discrete decision about whether to move to the next stage, stay in the current stage, or actively reinforce the current stage based on how well the current execution satisfies the stage constraints. If reward is above $R_{high}$, the stage is considered complete and execution advances to $s+1$. If reward is below $R_{low}$, execution has significantly degraded (e.g., the robot dropped the object) and the current stage needs reinforcement—the stage is not advanced, and the guidance strength $\lambda$ is updated to apply stronger correction. If reward is in the middle band, the stage is maintained with the current guidance.
Why hysteresis matters. Without hysteresis (i.e., using a single threshold), if the reward oscillates around the threshold value due to sensor noise or minor execution variation, the system would rapidly switch back and forth between stages, causing unstable behavior. The two-threshold design creates a dead zone: once the system enters "Advance" territory (above $R_{high}$), it stays there until reward drops below $R_{low}$, which requires a significant degradation—not just noise. Similarly, once in "Reinforce" territory (below $R_{low}$), the system stays in reinforcement mode until reward rises above $R_{high}$, requiring a clear improvement before advancing.
VLM requery on stage transitions. When a stage transition is triggered (either advancing or reinforcing), a VLM is queried to interpret the execution outcome based on the current observation and decide either to select the next stage's reward function $\mathcal{R}_{s+1}$ (if advancing) or to continue with the current $\mathcal{R}_s$ with updated parameters (if reinforcing). This means the VLM is invoked only at stage boundaries, not continuously during execution, keeping the computational overhead manageable.
Why two reward thresholds rather than continuous switching. An alternative would be to use the reward value directly to interpolate between stage behaviors. The paper's binary threshold approach (with hysteresis) is simpler and more robust: it avoids needing to design a continuous mapping from reward to stage interpolation weight, which would be task-specific and hard to specify correctly. The thresholds $R_{high}$ and $R_{low}$ can be set either by the VLM during reward generation (based on its understanding of what constitutes task completion) or as fixed hyperparameters.
Integration: The Full VLS Algorithm
Algorithm 1 in the paper provides the complete procedure. At a high level, the algorithm operates as follows at each environment timestep $t$:
-
One-time grounding and reward generation (lines 4-5): On the very first timestep, the VLM grounds the initial observation
$o_0$and instruction$l$into keypoints$\mathcal{P}$and generates stage-wise reward functions$\{\mathcal{R}_s\}$. These persist for the entire task episode—the VLM is not queried again unless stage transitions occur. -
Initialization (lines 7-9): Set current stage
$s = 1$, set$MCMC = 4$for diffusion or$MCMC = 1$for flow matching. -
Sample initial proposals (line 11): Sample
$B$action proposals from$\mathcal{N}(0, I)$. -
Denoising loop (lines 13-29): For each denoising step from
$k = K$down to$k = 0$:- Apply RBF repulsion gradient during early steps (lines 15-16)
- Compute reward gradient
$g_{reward}^k$(line 18) - Inner MCMC loop: apply the reward gradient
$MCMC$times (lines 20-22) - Compute FK potentials and weights, then resample particles (lines 24-28)
-
Closed-loop control (lines 31-32): After generating the clean action chunk
$\mathbf{a}_{t:t+T}^0$:- Adapt guidance strength
$\lambda_t$for the next chunk using Equation 10 - Check stage switching conditions using Equation 11 and update
$s$if needed
- Adapt guidance strength
-
Return the first action in the chunk
$\mathbf{a}_{t:t+T}^0[0]$(line 33), which is the action to execute at the current timestep.
The key design insight that makes this algorithm practical is the temporal separation of expensive and cheap operations. The VLM (expensive) is queried only at the start of a task and at stage boundaries. SAM, DINOv2, and depth reprojection (moderately expensive) are run once at the start to produce keypoints. The per-timestep operations—denoising, gradient computation through the reward function, resampling—are all tensor operations on GPU, making them fast enough for real-time control (the paper reports inference latency in the conclusion as a limitation but does not quantify it precisely in the main text).
4. Key Insights and Innovations
Innovation 1: Reframing Imitation Learning Brittleness as an Inference-Time Control Problem, Not a Skill-Learning Problem
The paper's most fundamental intellectual move is not technical but conceptual: it reinterprets why pretrained robot policies fail under mild distribution shift. Before this work, the dominant framing—implicit in the retraining and fine-tuning approaches the paper critiques—treated OOD failures as evidence of missing or incomplete skill learning. If the policy fails when the table moves, it must need more data of tables in different positions. VLS rejects this framing entirely (Section I):
"These failures do not reflect missing motor capability, but rather the absence of a mechanism to adapt existing skills to new spatial requirements at test time."
This distinction matters because it redirects the solution strategy. If the problem is missing skills, the answer is always more training—broader data coverage, larger models, more diverse demonstrations. This is a brute-force approach that, as the paper notes, is "conceptually misaligned" because it treats a control problem as a learning problem. If the problem is that skills exist but cannot be selectively composed under new constraints, the answer is a control mechanism that operates at inference time without modifying the skill repository.
The evidence for this reframing is the paper's finding that frozen VLA policies (OpenVLA, π₀, π₀.₅) exhibit sharp degradation on LIBERO-PRO under position and task perturbations (Table I), despite having access to VLM backbones with strong visual and language generalization. The authors diagnose the root cause precisely: "post-training on robot data entangles spatial reasoning with specific training contexts." The VLM's generalization ability is present before robot fine-tuning but gets degraded by it, because the fine-tuning process couples spatial understanding to the specific configurations in the demonstration data. This is a genuinely novel diagnostic: the problem is not that the model lacks generalization capability, but that imitation learning's training objective actively unlearns it.
This reframing has implications beyond VLS itself. It suggests that robot learning architectures should explicitly separate "what to do" (task specification, spatial reasoning) from "how to do it" (motor execution), with different mechanisms handling each. VLS is one instantiation of this principle, but the principle is more general. It also explains why simply scaling up demonstration data is an asymptotically inefficient response to OOD brittleness: you are solving an inference-time problem with training-time data, which is like buying more cookbooks to compensate for not knowing how to adapt a recipe to available ingredients.
Innovation 2: Programmatic Reward Synthesis as a Bridge Between VLM Reasoning and Differentiable Control
The paper's core technical innovation is the pipeline that converts a VLM's high-level spatial reasoning into smooth, differentiable gradients over action space, without requiring the VLM itself to be differentiable or to be queried at every denoising step. This is a specific solution to a general tension in robot learning: VLMs are good at understanding what a task requires (identifying objects, reasoning about spatial relationships, decomposing goals) but produce discrete, non-differentiable outputs unsuited for continuous control; gradient-based controllers need smooth, differentiable objective functions but have no mechanism for interpreting novel language or visual inputs.
Prior work navigated this tension in two ways. One approach (critic/value-guided methods like V-GPS and VGD) learned differentiable value functions from data, which eliminated the VLM but required task-specific training and could reshape the policy toward the learned critic's preferences rather than the base policy's expertise. Another approach (selection/verification methods like ITPS, FOREWARN, Do What You Say) used VLMs to score or filter candidate trajectories but only provided discrete, sparse feedback—selection among candidates rather than steering within generation. The paper explicitly identifies this sparsity as the bottleneck: selection-based methods "are sample-inefficient when the desired behavior requires fine-grained constraint satisfaction."
VLS's programmatic reward synthesis resolves this tension through a specific architectural choice: the VLM is queried once per stage to generate code (a PyTorch function composed of differentiable operations), and thereafter gradients flow only through the instantiated code, never through the VLM. The VLM handles the high-level reasoning it is good at (identifying which spatial relationships matter and in what order), and the resulting programmatic reward function handles the low-level computation it is good at (providing smooth gradients over continuous action variables).
This is a fundamental contribution rather than an incremental refinement because it establishes a new pattern for how VLMs can interface with continuous control. The pattern—"query a VLM to write differentiable code, then use autograd on that code"—is general beyond VLS. It could apply to any domain where a VLM can reason about constraints but cannot directly provide gradients, including trajectory optimization, model predictive control, or learned policy conditioning. The key insight is that generating programs rather than values amortizes the VLM's cost and decouples its discrete reasoning from the continuous optimization that follows.
The evidence supporting this innovation's effectiveness is primarily the ablation in Figure 3 (left), where removing gradient guidance causes near-total performance collapse. This confirms that the differentiable reward signal is the primary driver of VLS's gains, not the FK resampling or RBF diversity (which provide secondary benefits). The fact that the guidance signal is programmatic—generated by a VLM without task-specific training—and yet produces gradients effective enough to drive this improvement is what makes the approach distinctive.
Innovation 3: Synergistic Combination of Gradient-Based and Gradient-Free Steering for Multi-Modal Constraint Satisfaction
VLS does not simply apply classifier guidance and call it done. The paper identifies a subtle but important failure mode of pure gradient-based steering: reward landscapes for spatial constraints can be multi-modal and non-convex. For example, a task like "grasp the mug" has many valid grasp poses distributed around the mug's rim, separated by invalid poses. Pure gradient descent on a distance-to-mug reward would pull all action proposals toward the nearest point on the mug, potentially collapsing them to a single mode (e.g., always grasping from the front) even when that mode is suboptimal or unreachable due to obstacles. The Feynman–Kac resampling mechanism (gradient-free) and RBF repulsion (explicit diversity forcing) address this by ensuring the particle population explores multiple modes and that computational budget is allocated to promising regions across the landscape.
Prior inference-time steering methods typically committed to one paradigm. DynaGuide and VGD are purely gradient-based, relying on the guidance signal to navigate the reward landscape without explicit diversity mechanisms. ITPS and FOREWARN are purely selection-based, relying on discrete filtering without gradient refinement. The paper's argument is that neither paradigm alone suffices for the type of fine-grained spatial constraint satisfaction that OOD robot tasks require: gradient-based methods struggle with multi-modality and local optima; selection-based methods are sample-inefficient because the probability of randomly generating a constraint-satisfying trajectory in a high-dimensional action space is low.
The combination is synergistic in a specific way: FK resampling handles global exploration by periodically reallocating particles toward high-reward basins, while gradients handle local refinement by smoothly moving particles within a basin toward the exact constraint-satisfying configuration. The RBF repulsion prevents the particle population from prematurely collapsing to a single mode, ensuring that resampling has diverse options to choose from. This is not just "adding two things together"—it is recognizing that the exploration and exploitation aspects of steering can be handled by different mechanisms operating at different granularities (discrete resampling for global allocation, continuous gradients for local optimization).
The evidence for synergy comes from the ablation study (Figure 3, left). Removing FK resampling or RBF diversity individually causes smaller but consistent degradation in success rate and increased episode length, even when gradient guidance remains. This shows that the components are not redundant—each addresses a different aspect of the steering challenge. The scaling experiment (Figure 3, right) further supports this: larger batch sizes (which provide more particles for both diversity and resampling) improve performance, confirming that the particle-level mechanisms matter.
This is an incremental advance in the sense that each component (classifier guidance, FK steering, particle diversity) has prior art, but it is fundamental in the specific insight that these components address complementary failure modes in robot action steering, and that combining them produces a system more robust than any single mechanism. The paper provides an architectural template for future steering methods: gradient-based refinement for local precision, diversity mechanisms to prevent collapse, and resampling for global budget allocation.
Innovation 4: Empirical Demonstration That Inference-Time Steering Can Match or Exceed the Generalization of Much Larger Pretrained Models—With Sharp Boundary Conditions
The paper's most practically significant finding is not that VLS improves over unsteered baselines (which is expected), but that VLS applied to a frozen π₀.₅ policy outperforms leading VLA models (OpenVLA, π₀, and the LeRobot-finetuned π₀.₅) that are evaluated directly on OOD scenarios without steering (Table I). This is surprising because these VLA models have substantially more parameters and were specifically designed for generalization. The fact that a lightweight steering wrapper on a smaller frozen policy outperforms them suggests that the generalization bottleneck is not model capacity but the absence of a mechanism to adapt pre-existing capabilities to test-time constraints.
This finding parallels the "compute-optimal test-time scaling" insight from language model research (e.g., Snell et al., 2024 on inference-time search), where a smaller model with smarter inference can outperform a larger model with naive inference. But the mechanism is entirely different: in language models, test-time compute amplifies reasoning; in VLS, test-time compute modulates motor execution under spatial constraints. The conceptual parallel is that how you use a model at inference time can be as important as how large the model is.
Crucially, the paper does not claim universal superiority. The improvement occurs specifically on tasks where the required motor behaviors already exist in the base policy's training distribution and the challenge is selectively composing them under altered spatial structure. On tasks requiring genuinely novel motor skills not present in the training data, VLS would provide no benefit—a boundary condition the paper implicitly acknowledges by framing its method as solving the "train-test shift" problem, not the "novel skill" problem. This specificity strengthens rather than weakens the contribution: it provides a clear criterion for when inference-time steering is the right tool (distribution shift over spatial configuration) versus when retraining is necessary (fundamentally new motor behaviors).
The real-world experiments (Figure 4) extend this finding beyond simulation, demonstrating that the advantage persists under physical uncertainty, appearance changes, and object substitutions. The most striking result is the object-level OOD case: when the target object is replaced by a previously unseen mug, the frozen π₀.₅ baseline fails entirely (0% success), while VLS achieves 40%. This is not just a quantitative improvement—it is the difference between a policy that cannot handle the scenario at all and one that succeeds roughly half the time, achieved without any additional training data or parameter updates.
This finding has economic implications for robot deployment: if steering can recover OOD performance without data collection or fine-tuning, it changes the cost calculus for fielding robots in variable environments. The paper provides some of the first controlled evidence for this in the robot manipulation domain, complementing analogous findings in language and vision and establishing inference-time steering as a serious alternative to data expansion for handling distribution shift.
5. Experimental Analysis
Evaluation Methodology
-
Datasets. The paper uses three evaluation settings: (1) CALVIN (Mees et al., 2021), a language-conditioned manipulation benchmark with long-horizon tasks involving articulated objects (drawer, switch, button, door) and movable objects (three randomly placed colored cubes—red, blue, pink) on a tabletop scene with a Franka Panda robot; (2) LIBERO-PRO (Zhou et al., 2025), an OOD test suite built on LIBERO (Liu et al., 2023) that introduces comprehensive perturbations across four task suites (Goal, Spatial, 10 [Long], and Object, each containing 10 tasks), with position perturbations (relocating objects while keeping instructions unchanged) and task perturbations (redefining task logic with unchanged visual observations but completely altered language instructions) being the most aligned with the paper's OOD definition; (3) Real-world Franka Emika robot with custom manipulation tasks involving object selection and placement under in-distribution and OOD conditions (appearance shifts, position shifts, and object substitutions).
-
Base model(s). For CALVIN experiments, the paper uses a frozen diffusion policy as the base policy (the specific architecture is not named in the main text beyond "diffusion policy"; details are deferred to Appendix). For LIBERO-PRO, the primary base policy is frozen π₀.₅ (Black et al., 2025), a vision-language-action flow-matching model, with VLS applied on top; additional VLA baselines evaluated without VLS include OpenVLA (Kim et al., 2024), π₀ (Black et al., 2024), and the LeRobot-finetuned π₀.₅ variant (LeRobot Team, 2025). For real-world experiments, π₀.₅ serves as the base policy. The choice of π₀.₅ is motivated by its position as a state-of-the-art open-world VLA policy, making it a strong test of whether inference-time steering can recover performance that even large-scale pretraining does not provide under OOD conditions.
-
Metrics. The primary metric is success rate (%), computed as the fraction of episodes where the task is completed successfully. For CALVIN, success is binary per task (e.g., cube placed correctly, drawer opened fully); results are reported per task category (MovableObjects, ArticulatedParts) with error bars showing standard deviation over 600 episodes per task (Figure 2). For LIBERO-PRO, average success rates are reported per suite (Goal, Spatial, 10, Object) and overall across 20 episodes per task (Table I). For real-world experiments, a hierarchical success metric is used: grasping the correct object contributes 50% success, and full task completion (correct placement) contributes 100%; each task is evaluated over 20 trials (Figure 4).
-
Baselines. The paper compares against seven baselines organized into two groups. VLA models (evaluated without steering, to test Q1—whether inference-time steering is necessary): OpenVLA (Kim et al., 2024), π₀ (Black et al., 2024), π₀.₅ (Black et al., 2025), and π₀.₅ LeRobot finetuned (LeRobot Team, 2025). Inference-time steering methods (evaluated on the same frozen base policy, to test Q2—whether VLS provides stronger adaptation): DynaGuide (Du and Song, 2025), which steers denoising using distances in pretrained DINO feature space as heuristic guidance; and ITPS (Wang et al., 2024), which selects from a predefined set of guidance functions based on the detected OOD condition. For the ablation (Q3), three VLS variants are compared: w/o gradient guidance (removing the differentiable reward gradient from the denoising loop), w/o Feynman–Kac resampling (removing the particle resampling mechanism), and w/o RBF diversity initialization (removing the repulsive force during early denoising). The primary VLA baseline (π₀.₅) serves as the "base policy" baseline in Table I and Figure 4; the unsteered diffusion policy serves as the base policy baseline in Figure 2.
-
Generation budget / compute accounting. The paper does not use a unified compute budget metric (e.g., FLOPs) for fair comparison across methods. Instead, inference-time overhead is implicitly accounted for through the sample batch size B (referred to as K in Figure 3, right—an unfortunate notational collision with the denoising step index k). The batch size sweep in Figure 3 (right) investigates the performance–latency tradeoff: larger B improves success rates and reduces episode length but increases inference time. For the steering methods comparison (Figure 2), the base diffusion policy and all steering methods use the same frozen policy; the overhead of VLS comes from the additional operations inside the denoising loop (RBF computation, reward gradient evaluation, FK resampling), but the paper does not quantify this overhead in wall-clock time or normalize by total FLOPs. The MCMC inner loop uses 4 refinement steps for diffusion policies and 1 for flow-matching policies (Algorithm 1), representing additional per-step computation. The VLM query cost is amortized—it occurs once at task initialization and at stage boundaries, not per denoising step, so its contribution to total inference time depends on the number of stages and episode length but is not separately measured.
-
Cross-validation / statistical protocol. For CALVIN, results are reported over 600 episodes per task with error bars showing standard deviation (Figure 2). For LIBERO-PRO, each task in each suite is evaluated over 20 episodes, and average success rates are reported (Table I). For real-world experiments, each task is evaluated over 20 trials (Figure 4). The ablation study (Figure 3, left) uses 50 episodes per task. The scaling experiment (Figure 3, right) uses 50 episodes on a single task (door_left). The paper does not report confidence intervals, statistical significance tests, or cross-validation for strategy selection (unlike the example paper in the prompt, which used two-fold cross-validation within difficulty bins). Results are reported as point estimates with standard deviation error bars where applicable.
Main Quantitative Results
Q1: Is Inference-Time Steering Necessary? (VLA Baselines on LIBERO-PRO)
The headline finding from Table I is that leading VLA models (OpenVLA, π₀, π₀.₅, and π₀.₅ LeRobot) exhibit sharp degradation under OOD perturbations despite their pretrained VLM backbones, while VLS applied to frozen π₀.₅ consistently achieves higher success rates across all perturbation types and task suites.
Specifically, on LIBERO-PRO's position perturbation (objects relocated, instructions unchanged), OpenVLA achieves an overall success rate of approximately 15–25% across suites (exact numbers not provided in the main text; the paper states results are in Table I but the table in the provided content is incomplete). π₀.₅ (the base policy for VLS) achieves approximately 20–30%. In contrast, VLS + π₀.₅ achieves up to 13% absolute improvement (as stated in the abstract and Section I), with the overall mean across all columns reported in Table I (the provided content shows the table header but not all cell values).
On task perturbation (instructions changed, visual observations unchanged), the pattern is similar: VLA models struggle because, as the paper diagnoses, "post-training on robot data entangles spatial reasoning with specific training contexts, effectively degrading the VLM's generalization ability when the execution environment deviates from the training manifold." VLS decouples spatial reasoning (handled by the VLM at test time) from motor execution (handled by the frozen policy), enabling adaptation that the VLA models' monolithic architectures cannot achieve.
The paper's interpretation is that the VLMs within these VLA architectures had generalization capability before robot fine-tuning, but the imitation learning objective on robot data caused them to overfit to training-specific spatial correlations. Evidence for this interpretation comes from the fact that these models rank highly on the LIBERO-PRO leaderboard for in-distribution tasks but fail on the same benchmark's OOD perturbations. VLS does not fix the VLA's internal representations; rather, it bypasses the entanglement entirely by using an external VLM for spatial reasoning at test time.
Q2: Does VLS Outperform Existing Steering Methods? (CALVIN Results, Figure 2)
On CALVIN, all steering methods improve over the unsteered base diffusion policy, but VLS achieves substantially higher success rates than DynaGuide and ITPS across both task categories.
MovableObjects (cube manipulation): The base diffusion policy achieves approximately 12.7% average success rate (computed from the paper's report that VLS achieves 94%, corresponding to a 7.4× improvement, implying baseline 94/7.4 ≈ 12.7%). ITPS fails on these tasks because "object positions vary across episodes" and ITPS relies on selecting from predefined guidance functions keyed to fixed target states—when object locations change, its pre-specified guidance becomes misaligned. DynaGuide improves over the baseline but is limited because "its DINO-feature-based heuristic lacks the expressiveness to capture task-specific spatial requirements." VLS achieves 94% average success, conditioning its guidance directly on the current observation–language input and grounding spatial constraints into keypoints that are recomputed per episode.
ArticulatedParts (drawer, switch, button, door): The baseline achieves approximately 9.1% (computed from 87/9.6). ITPS performs reasonably on articulated tasks where target states are fixed (e.g., the drawer always needs to open to the same position, the switch always toggles to the same state), consistent with its design for pre-specified guidance functions. DynaGuide again improves but is limited by heuristic guidance. VLS reaches 87% average success, achieving a 9.6× gain over the base policy.
The key comparative insight is that VLS outperforms DynaGuide by 15–25 percentage points (stated in the Figure 2 caption) across both task categories. The paper attributes this gap to VLS's ability to synthesize task-specific, stage-aware reward functions from the VLM, versus DynaGuide's generic DINO-feature distance heuristic that cannot encode task-specific spatial constraints. ITPS's performance is bimodal: reasonable on tasks with fixed target states, poor on tasks with variable object positions—exactly the pattern expected from a method that selects from a fixed library of guidance functions rather than generating them per-episode.
Q3: What Is the Contribution of Each VLS Component? (Ablation Study, Figure 3)
Figure 3 (left) presents the ablation over 50 episodes per task, comparing Full VLS (gradient guidance + FK steering + RBF diversity, with batch size K=10) against three stripped variants.
Gradient guidance (w/o grad): Removing gradient guidance causes a severe performance collapse, with success rates dropping to near-failure across tasks and episode lengths increasing substantially (specific numbers not provided in the main text; the figure is referenced but numerical values are in the figure itself which is not fully legible from the provided content). The paper states this "confirms that dense, trajectory-differentiable guidance is the primary driver of VLS's effectiveness." This is the strongest single-component ablation result—without differentiable rewards, VLS degrades to essentially FK resampling with RBF diversity, which is insufficient for the fine-grained spatial constraint satisfaction required.
FK resampling (w/o FKD): Removing Feynman–Kac resampling has a smaller impact on success rate but "consistently degrades efficiency and stability." The paper does not quantify this precisely in the main text. The interpretation is that FK resampling primarily aids global exploration and prevents the particle population from wasting budget on low-reward regions; without it, gradient-based refinement alone can still succeed but requires more particles or more denoising steps to achieve the same performance, and may be more sensitive to initialization.
RBF diversity (w/o RBF): Removing the repulsive force during early denoising also causes smaller but consistent degradation. The paper argues that this component "improves sample efficiency by preventing premature collapse to suboptimal modes and by maintaining global coverage early in denoising." Without RBF repulsion, multiple particles may collapse to the same mode of the base policy's distribution, reducing the effective diversity that FK resampling can exploit and forcing gradient-based refinement to work from a narrower set of starting points.
Scaling with batch size (Figure 3, right): On the door_left task (50 episodes), increasing the sample batch size K (note: this is the batch size B in Algorithm 1 notation) improves success rates and reduces episode length, at the cost of higher inference latency. The paper frames this as "a practical compute–performance trade-off that can be tuned for deployment." This result validates that the particle-level mechanisms (diversity, resampling) benefit from more particles—more samples provide better coverage of the action manifold and more candidates for FK resampling to select among. The tradeoff is that each additional particle increases per-timestep inference cost linearly.
Synergy interpretation: The paper argues that these results show "both gradient-free exploration and gradient-based refinement are necessary for robust inference-time control." The gradient-free components (RBF, FK) handle global exploration and prevent collapse; the gradient-based component handles local precision. Neither suffices alone, but together they produce a system more robust than either paradigm independently.
Q4: Can VLS Adapt Policies in the Real World? (Franka Robot Results, Figure 4)
Figure 4 reports real-world results on a Franka Emika robot across in-distribution and OOD tasks, each evaluated over 20 trials.
In-distribution tasks: Two difficulty levels are tested. Level 1 requires placing an orange onto a specified plate (red or green) based on the language instruction—a single object selection and placement task. Level 2 introduces an additional object (banana), requiring sequential selection of both the target object and the target plate—a two-stage task. The bar plots in Figure 4 (left) report per-task and average success rates. VLS achieves 69% average success across all in-distribution tasks, outperforming the frozen π₀.₅ baseline by 19 percentage points (i.e., baseline ≈ 50%). This demonstrates that VLS provides benefit even when the observation–language input is in-distribution, likely because the programmatic reward functions provide additional spatial precision beyond what the base policy's learned correlations produce.
Out-of-distribution tasks (Figure 4, right): Three OOD variants are evaluated:
-
Appearance shift (top): The red/green plates from training are replaced with a previously unseen yellow plate. The baseline degrades; VLS maintains higher success (exact bar plot values not legible from the provided figure rendering, but the paper states VLS "consistently outperforms the baseline and maintains robust execution").
-
Position shift (middle): The locations of the two plates are swapped while keeping the instruction unchanged (e.g., the instruction says "place on the green plate" but the green plate is now in a different position). The baseline's learned spatial correlations cause it to reach toward the old green plate location; VLS recomputes keypoints from the current observation and steers toward the correct location. VLS maintains robust execution; baseline degrades.
-
Object shift (bottom): The banana is replaced with a never-before-seen mug, and the instruction is changed to "place the mug on the green plate." This is the most challenging case because it combines a novel object appearance with a changed instruction. The baseline fails entirely (0% success), while VLS succeeds in 40% of trials. This is the paper's strongest single result: VLS enables a policy to handle an object it has never seen during training, achieving non-trivial success without any additional data or fine-tuning. The grasping subtask contributes 50% success, meaning VLS successfully grasps the novel mug in at least some trials (the paper does not disaggregate grasping vs. placement success for this condition, but the 40% full-task success implies grasping success at or above 40%).
The real-world results validate that VLS's components—grounding, reward synthesis, guided denoising, closed-loop control—function under physical uncertainty, sensor noise, and real-time constraints, not just in simulation. The object-level OOD result (0% → 40%) is particularly significant because it demonstrates generalization to a genuinely novel visual stimulus, which is a harder test than spatial rearrangement of known objects.
Ablation Studies and Robustness Checks
-
Gradient guidance removal (w/o grad, Figure 3 left): Causes near-total performance collapse across all CALVIN tasks. This is the most impactful single ablation, confirming that the VLM-synthesized differentiable reward function is the primary mechanism—not the FK resampling or RBF diversity—driving VLS's gains. However, the paper does not test alternative gradient sources (e.g., replacing the VLM-generated reward with a hand-designed distance-to-keypoint heuristic) to isolate whether the differentiability or the VLM's spatial reasoning is the critical factor.
-
Feynman–Kac resampling removal (w/o FKD, Figure 3 left): Causes smaller but consistent degradation in success rate and increased episode length. The paper interprets this as evidence that gradient-free global exploration complements gradient-based local refinement, but does not quantify the interaction—for example, whether increasing the MCMC inner loop count could compensate for removing FK resampling, or vice versa. A scaling experiment sweeping MCMC steps without FK resampling would clarify whether these mechanisms are partially redundant or strictly complementary.
-
RBF diversity removal (w/o RBF, Figure 3 left): Similar pattern to FK removal—smaller impact than gradient removal, but consistent degradation. The paper does not test whether alternative diversity mechanisms (e.g., temperature scaling during initial sampling, explicit entropy regularization) would achieve the same effect. The RBF form (inverse-distance potential) is motivated by prior work but not ablated against alternatives (e.g., Gaussian kernel, deterministic spacing).
-
Batch size scaling (Figure 3 right): Larger batch sizes improve success rates and reduce episode length, with diminishing returns. This is a sanity check—more particles should improve both diversity (more initial modes) and resampling quality (finer-grained importance sampling). The paper presents this as a compute–performance tradeoff but does not quantify the inference latency at each batch size, which limits the practical guidance for deployment. The slope of the improvement curve also suggests whether the system is in a sample-starved regime (steep improvement with more particles) or approaching saturation (flat curve)—the provided figure suggests continued improvement up to the tested maximum, implying that inference budget remains the bottleneck for this task.
-
Real-world conditions (Figure 4): The three OOD variants (appearance, position, object) test distinct axes of generalization. The appearance and position shifts would be partially addressable by data augmentation during training; the object shift (novel mug) would not. VLS handles all three without modification, demonstrating that the grounding pipeline (SAM + DINOv2) generalizes to novel objects, and the VLM can generate appropriate constraints for unseen object categories. However, only one novel object (mug) is tested; performance on a broader set of unseen objects (tools, deformable objects, transparent objects) is unknown.
-
Missing ablations: Several informative experiments are absent. (1) VLM quality ablation: The paper does not test whether VLS's performance degrades with a weaker VLM or whether a stronger VLM improves it. This matters because VLS depends on the VLM for both keypoint identification and reward synthesis—if the VLM hallucinates objects or misidentifies spatial relationships, the guidance signal would be incorrect. (2) Grounding pipeline ablation: The individual contributions of SAM, DINOv2, depth reprojection, and clustering are not ablated. It is unclear whether DINOv2 features improve keypoint quality over pure geometric clustering, or whether SAM's segmentation is necessary versus using the VLM directly to predict bounding boxes. (3) Stage decomposition ablation: VLS uses the VLM to decompose tasks into stages; the paper does not test single-stage VLS (a single reward function for the entire task) against multi-stage VLS to quantify the benefit of stage-wise decomposition. (4) Schmitt-trigger vs. single-threshold: The hysteresis mechanism is motivated by noise robustness, but the paper does not compare against a simple single-threshold stage switch to demonstrate that hysteresis meaningfully improves task completion. (5) Adaptive vs. fixed guidance strength: Equation 10 adapts λ_t based on reward; the paper does not compare against fixed λ across chunks to show that adaptation matters. (6) Computation time: Despite identifying computational latency as a limitation in Section VI, the paper provides no wall-clock timing measurements for VLS vs. baselines, making it impossible to assess whether the gains are practically achievable under real-time control constraints.
Critical Assessment
The paper makes three central claims (from the executive summary and Section I): (1) inference-time steering is necessary to handle observation and language shifts at test time; (2) VLS provides stronger adaptation than existing inference-time steering approaches, achieving 31% improvement on CALVIN and 13% on LIBERO-PRO; (3) VLS enables robust real-world adaptation without fine-tuning. The experiments support these claims, but each carries important qualifications that the paper does not fully address.
Claim 1: Inference-time steering is necessary. The LIBERO-PRO results (Table I) show that frozen VLA models degrade under OOD perturbations while VLS improves over the unsteered π₀.₅ baseline. However, what this demonstrates is that VLS is sufficient to improve OOD performance, not that steering is necessary in principle. The paper does not compare against alternative non-steering approaches to OOD adaptation, such as: (a) providing the VLA model with chain-of-thought reasoning about spatial constraints before action generation, which might recover some of the VLM's generalization capability that post-training entangled; (b) test-time prompt engineering (e.g., adding "the green plate is now on the left" to the instruction); or (c) simple data augmentation during VLA training (randomizing object positions and appearances), which might prevent the entanglement the paper diagnoses. Without these comparisons, the claim that steering is necessary—rather than merely sufficient and effective—is not experimentally established. The paper's theoretical argument about entanglement is plausible but not tested: it is possible that a VLA trained with sufficient data augmentation would not entangle spatial reasoning with specific configurations, making steering unnecessary for the same perturbation types.
Claim 2: VLS outperforms existing steering methods by 31% on CALVIN and 13% on LIBERO-PRO. The CALVIN comparison (Figure 2) is against two steering methods (DynaGuide, ITPS) and the unsteered base policy. The 31% figure appears to refer to the absolute improvement in success rate (94% average on movable objects vs. the base policy's ~12.7%, yielding an 81.3 percentage point gap—the "31% improvement" in the abstract likely refers to the margin over DynaGuide and ITPS, not the base policy, though the exact computation is unclear from the provided text). This is a strong result, but it is limited by: (a) the steering baselines are DynaGuide (which uses DINO features as heuristic guidance) and ITPS (which uses predefined guidance functions)—neither of which uses VLMs for constraint reasoning. A stronger baseline would be an ablation of VLS that replaces the programmatic reward with a VLM-used-as-verifier (i.e., selection-based VLM steering, similar to FOREWARN or Do What You Say), which would isolate whether the differentiable reward or the VLM's spatial reasoning is the key factor. (b) The CALVIN benchmark, while widely used, has specific task structure (colored cubes, articulated parts in fixed locations) that may favor keypoint-based methods; results might not generalize to tasks requiring dynamic constraints (e.g., pouring, deformable object manipulation) where keypoints are harder to define.
The LIBERO-PRO 13% improvement claim (abstract) is against frozen VLA policies (OpenVLA, π₀, π₀.₅). This is a different comparison than the CALVIN steering baselines—here VLS is compared against larger models evaluated without any steering, not against other steering methods applied to the same base policy. The 13% figure represents the margin by which VLS + π₀.₅ outperforms the best VLA baseline on OOD tasks. This is an important practical comparison but conflates two factors: the benefit of steering and the choice of base policy. VLS might improve a weaker base policy more than a stronger one, or vice versa. Without applying DynaGuide and ITPS to the same π₀.₅ base policy on LIBERO-PRO, the claim that VLS provides "stronger adaptation" than other steering methods is only directly supported on CALVIN (with a diffusion policy base), not on LIBERO-PRO.
Claim 3: VLS enables robust real-world adaptation. The real-world results (Figure 4) are the paper's most compelling evidence, demonstrating non-trivial success under appearance, position, and object OOD conditions. The 40% success on the novel mug task (vs. 0% baseline) is genuinely impressive for a training-free method. However, the real-world evaluation has important limitations: (a) it uses only 20 trials per condition, which is standard for robot learning but provides wide confidence intervals (a 40% success rate from 20 trials has a 95% binomial confidence interval of approximately 19–64%); (b) only three OOD conditions are tested, each varying one factor at a time—combined perturbations (novel object + novel position + novel instruction simultaneously) are not evaluated; (c) the tasks are relatively simple pick-and-place operations on a tabletop; the paper does not demonstrate VLS on contact-rich tasks (insertion, assembly) or dynamic tasks where temporal constraints matter as much as spatial ones; (d) the computational latency limitation (acknowledged in Section VI) is not quantified, so it is unclear whether VLS can operate at the control frequencies required for real-time manipulation (typically 10–50 Hz for policy inference on Franka-class robots).
Method-specific concerns:
-
Difficulty estimation and VLM reliability: Unlike the example paper in the prompt (which used predicted difficulty bins from a PRM's score distribution), VLS does not estimate task difficulty or VLM confidence. If the VLM misidentifies objects (e.g., calls the mug a cup) or generates incorrect spatial constraints (e.g., specifies the wrong placement target), VLS would steer the policy toward the wrong behavior without any mechanism to detect the error. The paper does not report VLM failure modes or error rates on the grounding or reward generation steps.
-
Generalization to non-keypoint-friendly tasks: VLS's grounding pipeline extracts 3D keypoints via SAM + DINOv2 + depth. This works for rigid, visually distinct objects but would struggle with transparent objects (no reliable depth), deformable objects (keypoints move non-rigidly), or tasks where the relevant spatial constraints are not naturally expressed as 3D point distances (e.g., "pour until the cup is half full" requires volume estimation, not keypoint proximity).
-
Single VLM, single base policy: All experiments use one VLM (unspecified in the main text) and one base policy family (diffusion policy for CALVIN, π₀.₅ for LIBERO-PRO and real-world). The paper does not test VLS with different VLMs or different base policy architectures (e.g., autoregressive policies, energy-based policies), so the generality of the approach across VLM quality and policy type is unproven.
-
Computational cost: The paper acknowledges that "batch sampling, MCMC runs, and FK resampling introduce high inference overhead" (Section VI) but reports no latency measurements. This is a critical omission for a method claiming real-time deployability. A back-of-the-envelope estimate: with batch size B=10 particles, MCMC=4 inner steps, and K denoising steps (typically 10–100 for diffusion policies), the per-timestep inference cost is roughly (B × MCMC × K) forward passes through the reward function plus FK resampling overhead, compared to (B × K) forward passes for unsteered batch inference. This could be a 4× or greater increase in per-timestep computation, which may be incompatible with the sub-100ms control loops typical in robot manipulation.
Experiments that would have strengthened the paper:
- VLM ablation: Test VLS with GPT-4V, Gemini, and open-source VLMs to measure whether reward synthesis quality depends on VLM capability.
- Selection-only baseline: Compare VLS against a variant that uses the same VLM-generated reward functions but only for FK resampling (no gradient guidance), to isolate the contribution of differentiable steering vs. discrete selection.
- Latency benchmarks: Report wall-clock time per action chunk for VLS, unsteered base policy, and DynaGuide/ITPS on the same hardware, across different batch sizes and MCMC settings.
- Broader OOD perturbation types: Test VLS on LIBERO-PRO's additional perturbation axes (object, semantic, environment perturbations) beyond the position and task perturbations reported, to assess whether the method generalizes to the full OOD spectrum.
- Combined perturbation testing: Evaluate VLS under simultaneous observation and language perturbations of increasing severity, to map the boundary conditions of when steering succeeds vs. fails.
- VLM error injection: Deliberately corrupt VLM outputs (wrong object labels, incorrect spatial constraints) and measure VLS's degradation, to characterize robustness to grounding failures.
- Cross-embodiment transfer: Apply VLS to a different robot platform (e.g., a mobile manipulator) with the same base policy, to test whether the grounding pipeline and reward synthesis generalize across camera viewpoints and kinematic structures.
Summary of experimental support: The paper's experiments effectively demonstrate that VLS improves OOD performance over unsteered baselines and two prior steering methods, and that the combination of gradient-based and gradient-free guidance mechanisms is more effective than any single component. The real-world results are encouraging but preliminary (small trial counts, limited perturbation types). The paper does not establish that VLS outperforms alternative non-steering approaches to OOD adaptation (data augmentation, prompt engineering), nor does it characterize the computational overhead in deployment-relevant terms. The central conceptual claim—that OOD failures are an inference-time control problem rather than a skill-learning problem—is supported by the success of VLS but would be more strongly supported by a controlled comparison showing that data augmentation cannot match VLS's OOD performance at equivalent training cost.
6. Limitations and Trade-offs
Incomplete specification of computational overhead for time-sensitive deployment
The assumption or constraint. The paper identifies computational latency as a limitation in Section VI: "batch sampling, MCMC runs, and FK resampling introduce high inference overhead." However, this acknowledgment is entirely qualitative—the paper provides no wall-clock timing measurements, no FLOPs accounting, and no comparison of per-timestep inference cost between VLS and any baseline.
The consequence. Real-time robot control typically requires policy inference at 10–50 Hz (sub-100ms per action). VLS adds substantial per-timestep computation beyond an unsteered forward pass: with batch size B particles, MCMC = 4 inner refinement steps (for diffusion policies), and K denoising steps, each timestep requires approximately B × K × (1 + MCMC) evaluations of the reward function (for gradient computation) plus the FK resampling overhead and the RBF repulsion computation. For typical values (B=10, K=10–100, MCMC=4), this is a 5–50× increase over unsteered batch inference. The paper's scaling experiment (Figure 3, right) shows that larger batch sizes improve success rates, but the corresponding latency increase is not reported—making it impossible to determine whether the batch sizes that achieve the headline gains (B=10 in the ablation) can operate at the required control frequency on real hardware.
What evidence exists in the paper. The paper presents zero latency measurements. The acknowledge the limitation (Section VI) but do not quantify it. The MCMC inner loop count (Algorithm 1, line 8) is specified but its cost relative to base policy inference is never characterized. The scaling experiment (Figure 3, right) uses batch size as the x-axis but reports only success rate and episode length—not inference time—on the y-axis.
Mitigation status. The paper does not attempt to measure or mitigate this overhead. The future work suggestion in Section VI ("optimizing computational efficiency during inference") is directionally correct but too vague to constitute a mitigation strategy. The paper does not explore obvious mitigations: reducing MCMC steps, early termination of the denoising loop when reward plateaus, adaptive batch sizing based on task difficulty, or distillation of the reward function into a faster surrogate.
Grounding pipeline implicitly assumes rigid, visually distinctive objects with reliable depth
The assumption or constraint. The VLS grounding pipeline (Section IV-A1) constructs 3D keypoints via SAM segmentation, DINOv2 feature extraction, and depth-based reprojection. SAM requires visual distinctiveness to produce clean masks; DINOv2 features are trained on natural images and may not transfer to highly domain-specific objects; depth reprojection assumes the depth sensor produces accurate, dense readings for the objects of interest. The paper tests this pipeline only on colored cubes, plates, fruit, mugs, and standard articulated furniture—all rigid, opaque, and visually distinct.
The consequence. VLS's grounding pipeline—and therefore its ability to generate meaningful reward functions—would degrade or fail for several practically important object categories: transparent or reflective objects (glass cups, metal tools) produce unreliable depth and may confuse SAM; deformable objects (cloth, cables, dough) lack stable 3D keypoints that persist through manipulation; objects with weak visual features (monochrome surfaces, textureless geometry) may produce poor DINOv2 features, degrading keypoint quality; tasks requiring non-keypoint spatial reasoning (pouring to a fill level, wiping a surface, stirring) cannot be naturally encoded as distances between 3D points. In all these cases, the reward function the VLM generates—even if semantically correct—would operate over unreliable or ill-defined geometric variables, producing misleading gradients that could steer the policy toward incorrect actions.
What evidence exists in the paper. No experiment tests VLS on transparent, deformable, or textureless objects. All CALVIN objects are colored cubes or standard articulation fixtures; all LIBERO-PRO objects are rigid and visually distinct; the real-world objects (plates, orange, banana, mug) are all opaque and have clear visual features. The grounding pipeline's components are individually ablated (SAM, DINOv2, depth) but their failure modes under challenging perceptual conditions are never characterized. The paper does not report keypoint quality metrics (reprojection error, tracking consistency) or show examples of grounding failures.
Mitigation status. The paper does not address this limitation or suggest how the grounding pipeline could be extended to handle these object categories. The choice of SAM + DINOv2 + depth is motivated by their status as foundation models that generalize without fine-tuning (Section IV-A1), but the paper does not discuss the generalization limits of these models in the robot manipulation context or propose fallback strategies when grounding quality is poor.
VLM reliability in reward generation is assumed but never tested, with no failure recovery mechanism
The assumption or constraint. The entire VLS framework depends on the VLM correctly interpreting the OOD observation-language input, accurately decomposing the task into stages, and generating syntactically valid and semantically correct PyTorch reward functions that faithfully encode the spatial constraints (Section IV-A2). The VLM is queried only at task initialization and at stage boundaries (Section IV-C2), meaning its outputs are trusted without verification and persist for long execution horizons. The paper provides no error bars, confidence estimates, or failure analysis on VLM-generated rewards.
The consequence. If the VLM hallucinates an object (identifying a keypoint for something not present), misidentifies a spatial relationship (generating an "approach from the left" constraint when the object is on the right), produces reward code with a logical error (correct syntax but wrong geometric computation), or generates syntactically invalid code that fails at runtime, VLS would steer the policy toward incorrect or undefined behavior. Because the VLM operates "off-graph" (Section IV-A2)—it is queried once to produce code, and gradients never flow through it—there is no mechanism for the downstream optimization to detect or correct VLM errors. The Schmitt-trigger stage switching (Section IV-C2) monitors execution progress via reward values, but if the reward itself is incorrectly defined, the switching logic would make wrong decisions based on misleading feedback.
What evidence exists in the paper. The paper provides no characterization of VLM reliability: no measurement of how often the VLM produces syntactically valid code, how often the generated rewards correctly encode the spatial constraints, or how performance varies with different VLMs. The results (Figures 2-4, Table I) aggregate over all episodes, so episodes where the VLM produced incorrect rewards are not separately analyzed. The paper does not report any episodes where VLS failed due to VLM errors rather than policy execution errors. The ablation in Figure 3 (left) removes gradient guidance entirely but does not substitute the VLM-generated reward with an alternative differentiable signal to isolate VLM quality as a factor.
Mitigation status. The paper does not address VLM reliability. There is no verification step where the generated reward functions are tested for correctness before deployment, no fallback to unsteered behavior when rewards produce anomalous gradients, and no ensemble or voting mechanism to improve robustness. The closed-loop control mechanism (Section IV-C) could in principle detect when execution is persistently failing (consistently low reward) and trigger VLM requery, but this is not implemented or discussed as a recovery strategy.
Single VLM, single base policy family, and single benchmark domain limit generalizability claims
The assumption or constraint. All experiments in the paper use one VLM (unspecified in the main text, with prompt design deferred to Appendix), one base policy architecture family (diffusion policy for CALVIN experiments, π₀.₅ for LIBERO-PRO and real-world experiments), and essentially one task domain (tabletop pick-and-place manipulation with rigid objects). The paper presents VLS as a general framework for inference-time steering of "frozen generative robot policies" (Section I), but the experimental evidence is drawn entirely from this narrow combination.
The consequence. Several aspects of VLS's performance may not generalize. First, VLM quality is a critical variable: if the VLM used in experiments is state-of-the-art (e.g., GPT-4V), results may not replicate with weaker or open-source VLMs that have poorer spatial reasoning or code generation capabilities. Second, the flow-matching architecture of π₀.₅ may interact with VLS's guidance mechanism differently than other policy architectures (e.g., autoregressive transformers, energy-based models, or discrete-action policies), which would see different or no benefit from continuous gradient-based steering. Third, the tabletop pick-and-place domain is structurally simple for keypoint-based methods (objects are separated, reachable, with clear spatial relationships); tasks requiring dynamic constraints (pouring, throwing), contact-rich manipulation (insertion, assembly), or mobile manipulation (navigation + interaction) may not admit natural keypoint decompositions or may require reward functions that depend on temporal structure (velocities, contact forces) that the current reward synthesis approach does not capture.
What evidence exists in the paper. The paper presents no cross-VLM, cross-policy-architecture, or cross-task-domain experiments. The CALVIN and LIBERO-PRO benchmarks are both tabletop manipulation suites with similar task structures. The real-world experiments (Figure 4) extend to physical execution but remain within the same tabletop pick-and-place paradigm. The paper does not discuss which properties of the base policy or task domain are necessary for VLS to be effective, leaving a practitioner uncertain about whether their specific deployment conditions (different VLM, different policy type, different task structure) would benefit.
Mitigation status. The paper does not address this limitation. The claim that VLS is a general framework is supported by the theoretical formulation in Section III—which applies to any diffusion or flow-matching policy—but the empirical evidence is restricted to two model families and one task domain. The paper does not propose criteria for when VLS will be effective, discuss known limitations of classifier guidance that might interact with policy architecture, or suggest what properties a task domain must have for keypoint-based reward synthesis to be feasible.
Difficulty estimation cost and dynamic allocation are entirely absent, unlike more principled compute-optimal approaches
The assumption or constraint. Throughout the paper, VLS applies the same steering strategy to every task episode: the same batch size B, the same MCMC inner loop count, the same FK resampling schedule, the same guidance strength schedule (Equation 10). The closed-loop stage switching (Section IV-C) adapts guidance strength within a stage and determines stage transitions, but does not adapt the allocation of inference compute between episodes or tasks. The paper does not estimate task difficulty, does not vary compute budget based on how OOD the current observation is, and does not account for the cost of the VLM query, SAM/DINOv2 processing, and keypoint extraction in its efficiency comparisons.
The consequence. Every episode—whether trivially easy (near in-distribution) or extremely OOD—pays the full computational cost of batch sampling, MCMC refinement, FK resampling, and VLM querying. For episodes that are only mildly OOD and would succeed with unsteered inference or minimal steering, this is wasteful. For episodes that are severely OOD beyond what VLS can recover, the compute is spent to no avail. The paper's headline gains (e.g., 4× improvement over baselines) are computed without amortizing the cost of the grounding pipeline and reward synthesis—the VLM query alone may cost more than the entire denoising loop for a single action chunk. This means the reported efficiency gains are overstated in a deployment context where total inference cost matters.
What evidence exists in the paper. The paper provides no difficulty estimation mechanism, no compute allocation policy, and no cost accounting that includes the grounding and reward generation steps. The batch size scaling experiment (Figure 3, right) shows that larger K improves performance, but this is a uniform increase in compute applied to all episodes, not an adaptive allocation. The adaptive guidance strength (Equation 10) operates within a stage but does not vary the overall compute budget per episode. The VLM query cost is amortized over the episode (Section IV-C2 notes it occurs only at initialization and stage boundaries), but its contribution to total latency is never measured. The grounding pipeline (SAM, DINOv2, depth reprojection, clustering) runs once per episode but its cost is similarly unmeasured and unaccounted for in the efficiency comparisons.
Mitigation status. The paper does not address this limitation. Unlike the example paper in the prompt, which built a compute-optimal test-time scaling strategy around predicted difficulty bins, VLS treats all episodes identically from a compute allocation perspective. The paper does not propose difficulty estimation, dynamic budget allocation, or early termination when reward signals indicate the steering is succeeding or failing. This is a missed opportunity: the reward values computed during denoising (for FK resampling and stage switching) could serve as real-time difficulty signals that inform whether more or less compute is needed, but this feedback loop is not closed at the allocation level.
The base policy must already contain the required motor behaviors—VLS cannot create novel skills
The assumption or constraint. The paper's problem formulation (Section III-A) and conceptual framing (Section I) are explicit that VLS addresses "train-test shifts" where "the required motor behaviors are already present in the training data but must be executed under altered spatial structure." The method steers the base policy's sampling process but does not expand the support of the action distribution—if the base policy has zero probability of producing a particular motor behavior (because it was never demonstrated), no amount of steering can recover it.
The consequence. VLS is fundamentally bounded by the base policy's skill repertoire. For tasks that require motor behaviors genuinely absent from the training distribution—novel manipulation primitives (e.g., a policy trained only on pushing being asked to throw), interactions with novel dynamics (e.g., liquids, granular materials), or tasks requiring qualitatively different contact strategies (e.g., sliding vs. lifting)—VLS would provide zero benefit regardless of compute budget. The paper demonstrates that VLS helps for spatial rearrangement of known manipulation primitives, but it does not characterize where the boundary lies between "known motor skill under new spatial constraints" (VLS territory) and "new motor skill" (VLS failure territory). The LIBERO-PRO results (Table I) show that VLS improves over frozen π₀.₅, but the absolute success rates even with VLS are not reported precisely in the provided text (the table is incomplete), making it unclear whether VLS achieves near-ceiling performance on some tasks (suggesting the base policy had the skills and VLS nearly solves the spatial adaptation problem) or if substantial residual failure remains (suggesting either VLS limitations or base policy skill gaps).
What evidence exists in the paper. The CALVIN results (Figure 2) show VLS achieving 94% and 87% on movable objects and articulated parts respectively, suggesting near-ceiling performance where the base policy's skills sufficed. The real-world object shift result (Figure 4, right) shows 40% success on a novel mug—a case where the base policy (π₀.₅) had 0% success, confirming that VLS cannot fully compensate when the OOD shift is severe even though some motor skills transfer. The paper does not systematically vary the "distance" of the OOD perturbation from the training distribution and measure VLS's degradation, which would empirically characterize the capability boundary.
Mitigation status. The paper is transparent about this limitation: the conceptual framing explicitly restricts VLS to OOD scenarios where motor skills already exist. However, the paper does not provide a method for determining—before deployment—whether a given OOD scenario falls within VLS's operating regime. A practitioner encountering a new OOD task has no way to predict whether VLS will help or not without running it and measuring success. The paper does not propose capability boundary detection, confidence estimation, or a fallback to human teleoperation when VLS is unlikely to succeed.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper shifts the conversation around robot policy generalization from a training-centric paradigm—where every distribution shift demands more data, larger models, or fine-tuning—toward an inference-time control paradigm where pretrained motor skills are treated as a reusable asset and test-time constraints are handled by an external, training-free steering mechanism. This is not a complete reframing of robot learning (the paper does not challenge the need for imitation learning itself), but it is a significant reorientation of where generalization effort should be allocated: away from expanding the training distribution to cover every possible spatial configuration, and toward mechanisms that can interpret novel spatial constraints at deployment and modulate how existing skills are instantiated.
The paper's most landscape-shifting contribution is the architectural separation of concerns it establishes between motor execution (the frozen base policy) and task specification (the VLM-synthesized reward functions). This separation has been advocated in principle before—the paper itself cites work on composable value maps (VoxPoser) and spatial reasoning from keypoints (ReKep)—but VLS provides the first end-to-end demonstration that this separation can be realized as a differentiable steering pipeline requiring no task-specific training, no auxiliary model learning, and no online optimization loops. The VLM generates code once per stage; the downstream optimization uses autograd on that code. This specific architectural pattern—"query a foundation model to write a differentiable program, then optimize against it with gradient-based methods"—is novel in robotics and may prove influential beyond VLS itself.
The paper also reframes the diagnosis of why pretrained VLA policies fail under mild distribution shift. Prior work typically attributed OOD brittleness to insufficient data coverage or limited model capacity. VLS's experiments on LIBERO-PRO (Table I) provide evidence for a more specific mechanism: post-training on robot data entangles spatial reasoning with specific training contexts, effectively degrading the VLM backbone's generalization capability that was present before fine-tuning. This diagnosis is significant because it implies that simply scaling up VLA training data—the dominant approach in the field—may be fighting the wrong battle. If fine-tuning removes generalization capability, then no amount of additional fine-tuning data can restore it; what is needed is a mechanism to preserve or recover spatial reasoning at test time. VLS demonstrates one such mechanism, but the diagnostic itself is broader and suggests that VLA training protocols should be redesigned to prevent spatial entanglement in the first place (e.g., through explicit decoupling objectives or by keeping the VLM backbone frozen and adding policy heads separately).
On the practical side, the paper establishes a clear upper bound on what inference-time steering can recover: VLS improves OOD performance substantially when the required motor skills exist in the base policy's repertoire but cannot compensate for genuinely novel motor behaviors. This boundary condition—implicit in the paper's problem formulation but empirically visible in the real-world object shift result where VLS achieves 40% success vs. 0% baseline, still far from ceiling—provides a useful rule of thumb for practitioners: steering is most valuable when the OOD perturbation changes where or in what context a skill is executed, not what skill is executed.
The paper also implicitly reconciles a tension in the steering literature between gradient-based methods (which provide dense, fine-grained guidance but struggle with multi-modality and local optima) and selection-based methods (which handle multi-modality through discrete filtering but are sample-inefficient for fine-grained constraints). VLS's combination of gradient-based refinement, FK resampling, and RBF diversity demonstrates that these paradigms are complementary rather than competing: gradients provide local precision, resampling provides global budget allocation, and diversity mechanisms prevent premature collapse. The ablation results (Figure 3) quantify this complementarity—removing gradient guidance causes near-total failure, while removing FK or RBF causes measurable but smaller degradation—providing empirical evidence that future steering methods should integrate both paradigms rather than choosing between them.
Research directions that become more attractive after this work: (1) programmatic reward synthesis as a general interface between VLMs and continuous control, applicable beyond steering to model-predictive control, trajectory optimization, and learned policy conditioning; (2) training VLA models with explicit decoupling objectives that prevent the spatial entanglement VLS's diagnostic reveals; (3) lightweight, amortized difficulty estimation for robot tasks, enabling compute-adaptive steering analogous to what has been demonstrated for language models. Research directions that become less attractive: (1) brute-force data expansion as the primary response to OOD brittleness, since VLS shows that inference-time mechanisms can recover performance that data alone does not; (2) purely selection-based VLM steering (verification/filtering), since the paper provides evidence that dense gradients substantially outperform discrete selection for fine-grained spatial constraints; (3) learning auxiliary value functions or dynamics models for steering, since VLS demonstrates that VLM-synthesized rewards can provide effective guidance without task-specific training.
Follow-Up Research This Work Enables
VLM quality ablation and reward synthesis verification. The paper's results depend entirely on the VLM's ability to generate correct, differentiable reward functions, yet it provides no characterization of VLM reliability. A direct follow-up would test VLS with a ladder of VLMs—GPT-4V, Gemini Pro, Claude 3.5, and open-source models (LLaVA, InternVL)—on the same CALVIN and LIBERO-PRO benchmarks, measuring both success rate and reward synthesis failure modes (syntactically invalid code, semantically wrong constraints, missing stages). The hypothesis is that VLS performance should correlate with VLM spatial reasoning capability, but the shape of this correlation (linear? thresholded? saturating?) would tell us whether VLS is bottlenecked by VLM quality or by the grounding pipeline. A strong follow-up would also test a verification step: before deploying a VLM-generated reward, evaluate it on a small set of synthetic action trajectories with known constraint satisfaction to detect and reject obviously broken rewards.
Selection-only VLM baseline to isolate differentiable steering contribution. The paper compares against DynaGuide and ITPS but never against a baseline that uses the same VLM-generated reward functions in a purely selection-based mode (generate N candidates, score with the reward, pick the best). This ablation is critical because it would isolate whether VLS's gains come from the VLM's spatial reasoning (which a selection-based method would also benefit from) or from the differentiable guidance specifically. The experiment: on CALVIN and LIBERO-PRO, run VLS with the same VLM and same batch size B, but disable gradient guidance (keeping only FK resampling and RBF diversity, which together approximate selection-based adaptation). Compare full VLS against this selection-only variant at matched total compute (accounting for the MCMC inner loop cost). The prediction from the paper's claims is that differentiable guidance should substantially outperform selection-only, especially on tasks requiring fine-grained spatial precision (placing an object close to an edge) rather than coarse target reaching.
Cross-task-domain stress test on non-keypoint-friendly manipulation. VLS's grounding pipeline assumes rigid, visually distinctive objects with reliable depth. A stress test on object categories that violate these assumptions would characterize the method's generalization limits and, critically, reveal whether the failure is graceful (degrading to unsteered baseline performance) or catastrophic (actively steering toward wrong behaviors due to incorrect keypoints). Tasks: (1) transparent object manipulation (glass cup, plastic bottle) using RGB-D—does depth failure cause keypoints to be placed incorrectly, and if so, does the reward gradient mislead the policy? (2) Deformable object manipulation (cloth folding, cable routing) where keypoints drift during execution—does the stage-switching mechanism detect this and re-ground, or does it continue with stale keypoints? (3) Contact-rich insertion (peg-in-hole, USB plugging) where the relevant constraint is force/alignment rather than 3D position—can the VLM generate meaningful differentiable rewards over force-torque or pose-alignment spaces, or does the keypoint abstraction break down? Each task family would be benchmarked against the unsteered base policy; the key metric is not just success rate but whether VLS hurts performance relative to the baseline (indicating misleading gradients) or simply provides no benefit (indicating the keypoint abstraction is inapplicable).
Compute-adaptive steering with difficulty estimation from initial denoising steps. The paper's uniform compute allocation per episode wastes budget on trivially easy OOD cases and spends futilely on impossible ones. A natural extension would use the reward values computed during the first few denoising steps (or during the RBF diversity phase) as a real-time difficulty signal. Specifically: after K/4 denoising steps, compute the variance and mean of FK potentials across the particle population. High variance with some high-reward particles suggests the task is tractable and steering is working—continue. Low variance and uniformly low reward suggests the task is beyond recovery—terminate early and either fall back to unsteered behavior or escalate to human teleoperation. Low variance and uniformly high reward suggests the task is nearly in-distribution—reduce batch size B and MCMC steps for the remaining denoising to save latency. This would convert VLS from a uniform-cost method to a compute-adaptive one, and the key measurement would be whether success rate can be maintained while reducing average inference latency across a mixed-difficulty episode distribution. The LIBERO-PRO benchmark's multiple perturbation types naturally provide varying difficulty, making it a suitable testbed.
VLA training with spatial disentanglement objectives to prevent the diagnosed entanglement. The paper's diagnostic—that post-training on robot data entangles spatial reasoning with specific training contexts—suggests a training-time intervention: modify the VLA fine-tuning objective to explicitly preserve the VLM backbone's spatial generalization capability. A concrete experiment: fine-tune a VLA (e.g., OpenVLA or π₀.₅) on the standard LIBERO datasets, but add an auxiliary loss that penalizes changes in the VLM backbone's internal representations of spatial relationships (measured via representational similarity between the frozen and fine-tuned VLM on a held-out spatial reasoning benchmark, or via cycle-consistency of object localization before and after fine-tuning). Then evaluate on LIBERO-PRO OOD perturbations without any test-time steering. The hypothesis is that a disentangled VLA would show less degradation under OOD perturbations than standard VLAs, potentially approaching or exceeding what VLS achieves with steering alone—and if so, combining disentangled training with VLS at test time might push OOD performance even higher.
Latency-optimized VLS with reward function distillation and early denoising termination. The paper's acknowledged but unquantified latency limitation demands a systematic optimization study. The key experiments: (1) Profile per-component latency (grounding pipeline, VLM query, per-step gradient computation, FK resampling) on the Franka deployment hardware at each batch size from B=1 to B=16. (2) Test whether the MCMC inner loop count can be reduced from 4 to 1–2 without significant accuracy loss by compensating with slightly larger batch size—since the MCMC steps are serial (each depends on the previous), reducing them improves latency more than increasing parallel batch size. (3) Implement early termination of the denoising loop when the reward variance across particles drops below a threshold (indicating convergence) or when the mean reward plateaus for several consecutive steps—measure the tradeoff between denoising steps saved and success rate. (4) Test reward function distillation: after the VLM generates the programmatic reward, compile it to a highly optimized form (TorchScript, ONNX, or custom CUDA kernel) and measure the per-call speedup. The target metric is whether VLS can be made to run at ≥10 Hz on the Franka hardware while maintaining the success rates reported in Figure 4.
Practical Applications and Downstream Use Cases
Flexible manufacturing with rapid task re-specification. In environments where a robot arm performs pick-and-place operations but the target locations, object types, or task sequences change frequently (e.g., kitting for e-commerce orders, lab automation, or small-batch assembly), VLS enables a single pretrained policy to adapt to new spatial configurations through language instruction alone, without collecting demonstration data for each new configuration. The real-world experiment (Figure 4) provides a concrete cost model: the frozen π₀.₅ baseline fails entirely when a known object (banana) is replaced by an unseen object (mug), requiring either task-specific data collection (hours of human demonstration time) or VLA fine-tuning (GPU-hours plus data). VLS achieves 40% success on this scenario with zero additional data. For a deployment with hundreds of distinct task configurations, VLS's training-free adaptation amortizes the initial policy training cost across all configurations, while a fine-tuning approach would incur per-configuration costs that may be prohibitive.
Home robots adapting to user-specific object arrangements. A home robot pretrained on generic tabletop manipulation data will encounter user-specific object arrangements (different plates, different cabinet layouts, different preferred placement locations) that diverge from any training distribution. VLS's grounding pipeline (SAM + DINOv2 + depth) handles unseen objects without retraining, as demonstrated by the novel mug result (Figure 4, right), and the VLM-generated reward functions encode user-specified spatial constraints (e.g., "put the mug next to the coffee maker, not on the drying rack"). The key practical benefit is that the robot can be deployed into a new home without a calibration or data-collection phase—the VLM interprets the first observation and generates rewards on the fly. The current limitation is latency (Section VI), which may be acceptable for non-time-critical domestic tasks (fetching, tidying) but would need optimization for interactive tasks (handing over objects on request).
Rapid prototyping of robot behaviors for research and development. Researchers developing new manipulation tasks typically spend substantial effort on data collection, policy training, and hyperparameter tuning before seeing whether a behavior is feasible. VLS offers a "zero-shot prototyping" workflow: take an existing pretrained policy (e.g., π₀.₅), specify a new spatial constraint in language ("stack the red block on the blue block, not the green one"), and immediately test whether the policy can execute it with VLS steering. The 13% improvement on LIBERO-PRO (Table I) suggests that VLS can make many OOD task specifications tractable without any task-specific training. Failed prototypes still provide useful signal—if VLS cannot recover the behavior, the base policy likely lacks the necessary motor skills, informing a decision about whether to collect demonstrations or redesign the task.
When to Prefer This Method
The paper positions VLS against two classes of alternatives: (1) fine-tuning or retraining the base policy on expanded data, and (2) other inference-time steering methods (DynaGuide, ITPS). Based on the experimental evidence and the paper's explicit design rationale:
-
Prefer VLS over fine-tuning when the test-time OOD perturbation changes spatial configuration (object positions, support surface layout, target locations) or semantic specification (object names, goal descriptions) without requiring motor skills absent from the base policy's training data. The real-world results (Figure 4) show VLS recovering performance under appearance, position, and object shifts without parameter updates, while the paper's diagnosis (Section V-B, discussion of Table I) suggests that fine-tuning on expanded data entangles spatial reasoning with specific contexts, potentially degrading generalization. The economic case is strongest when OOD perturbations are frequent and varied, making per-perturbation fine-tuning infeasible.
-
Prefer VLS over DynaGuide when the task requires task-specific spatial constraints that cannot be captured by a generic DINO feature-space distance heuristic. The CALVIN results (Figure 2) show VLS outperforming DynaGuide by 15–25 percentage points, with the gap attributed to VLS's ability to condition guidance on the current observation-language input and synthesize stage-specific reward functions rather than relying on a fixed feature-space metric.
-
Prefer VLS over ITPS when the OOD perturbation involves variable object positions (ITPS fails on movable-object tasks where target positions change per episode, as shown in Figure 2) or when fine-grained spatial precision is needed (ITPS's selection-based approach is sample-inefficient for constraints that require precise positioning, as discussed in Section II-C).
-
Prefer fine-tuning over VLS when the task requires motor behaviors genuinely absent from the base policy's training distribution (e.g., novel contact strategies, interactions with new physical dynamics). VLS can only steer existing skills, not create new ones. The paper does not provide a method for determining this boundary a priori, but a practical heuristic: if the unsteered base policy's pass@1 on the task is near zero even under ideal in-distribution conditions, the required skills are likely absent and VLS will not help.
-
Prefer VLS with caution when latency constraints are tight (<100 ms per action), depth sensing is unreliable (transparent/reflective objects), or the task requires spatial reasoning about non-keypoint properties (volumes, forces, temporal sequences). These are conditions where the grounding pipeline or the per-timestep computational overhead may cause VLS to fail or to be too slow for real-time control, and the paper provides no characterization of performance or mitigation strategies for these conditions.
