PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs

ArXiv: 2402.07872

🎯 Pitch

PIVOT turns VLMs into zero-shot robot controllers by having them pick numbered dots on images, then refining their choices—no robot training data needed. Across four real-world navigation tasks, this visual prompting alone scores 75–100%, while bigger VLMs just get better at it.

---

1. Executive Summary

This paper introduces Prompting with Iterative Visual Optimization (PIVOT), a zero-shot method that enables vision-language models (VLMs) to produce continuous spatial outputs—such as robot actions or localization coordinates—by casting tasks as iterative visual question answering. Using GPT-4V and Gemini models without any fine-tuning, PIVOT annotates images with numbered candidate proposals drawn from a distribution, queries the VLM to select the most promising ones, fits a new distribution to those selections, and repeats—conceptually analogous to the cross-entropy method applied entirely in visual space. The approach achieves non-zero success across real-world mobile manipulator navigation (75–100% on four goal-directed tasks with 3 iterations and 3 parallel calls), real-world manipulation (67% grasp rate on "pick coke can" with 3 iterations and 3 parallel calls), and RefCOCO visual grounding (strong accuracy even after a single iteration), while offline ablations show that the combination of iterations and parallel calls consistently outperforms either alone. Scaling experiments across four sizes of the Gemini model family reveal monotonic performance improvement with larger VLMs, establishing that PIVOT's zero-shot spatial reasoning capabilities improve with underlying model scale while remaining fundamentally bounded by the VLM's inability to reliably reason about 3D depth from 2D annotations alone.

2. Context and Motivation

The Core Problem: VLMs Produce Text, But Robotics and Spatial Tasks Require Continuous Actions

The fundamental challenge this paper addresses is a mismatch between what state-of-the-art vision-language models (VLMs) can output and what embodied systems need: VLMs generate text tokens, but robotic control, navigation, and spatial localization require continuous coordinates, trajectories, or low-level action commands. A robot arm needs to output (x, y, z, gripper) deltas in Cartesian space; a mobile robot needs 2D waypoints in pixel or world coordinates; a visual grounding system needs bounding box centers. These are fundamentally continuous outputs residing in a metric space, not discrete tokens from a vocabulary. The paper observes:

"most VLMs still only output textual answers, seemingly limiting such interactions to high-level question answering. Many real-world problems are inherently spatial: controlling the trajectory of a robotic arm, selecting a waypoint for a mobile robot, choosing how to rearrange objects on a table, or even localizing keypoints in an image."

This gap exists because VLM architectures are designed for vision-language understanding (captioning, VQA, reasoning in language), not for regressing to spatial coordinates. The output head of these models is a language modeling head—it produces probability distributions over tokens, not continuous-valued vectors. Directly asking a VLM to output "move to (0.34, -0.12, 0.05)" requires the model to generate precise floating-point numbers token by token, which is far outside its training distribution and fails catastrophically in practice. The paper does not belabor this point with explicit failure-rate tables for direct coordinate generation, but it is the implicit motivation for the entire approach.

Why This Problem Matters

The significance of bridging this gap extends across multiple dimensions:

Practical robotics deployment. If VLMs can be adapted to produce robot actions, it unlocks a path toward generalist robot controllers that leverage internet-scale pretraining without requiring robot-specific training data. The paper emphasizes this zero-shot property repeatedly:

"our approach enables zero-shot control of robotic systems without any robot training data, navigation in a variety of environments, and other capabilities."

This is genuinely transformative if it works reliably because robot data is expensive and scarce compared to internet data. The RT-X dataset [38], which the paper uses for offline evaluation, represents one of the largest robot demonstration collections, yet it covers a tiny fraction of the tasks and environments that internet-trained VLMs have been exposed to. A method that can extract actionable spatial knowledge from VLMs without fine-tuning could dramatically reduce the data requirements for building capable robot systems.

Theoretical significance for foundation model capabilities. The paper also positions itself as a probe into what VLMs already know about spatial reasoning, even though they were never explicitly trained on robot control or precise localization:

"Our aim is not necessarily to develop the best possible robotic control or keypoint localization technique, but to study the limits and potentials of such models."

This is important because understanding the current boundaries of VLM spatial intelligence informs both model development (what training data is missing?) and deployment strategies (what tasks can we reasonably expect to work today?). The paper's identification of specific failure modes—3D depth reasoning, interaction-under-occlusion, greedy multi-step behavior—provides a diagnostic for the VLM research community.

Unification of vision-language and embodied AI. The paper frames PIVOT as an attempt to

"unify internet-scale general vision-language tasks with physical problems in the real world by representing them in the same input space."

This is a broader intellectual agenda: rather than treating VLMs and robot policies as separate model classes, the idea is to find methods that make the VLM's existing visual reasoning machinery directly applicable to embodied tasks. If successful, this collapses two research communities that have largely operated in parallel.

Where Prior Approaches Fall Short

The paper identifies several lines of prior work and explains why each is insufficient for the zero-shot spatial reasoning problem:

1. High-Level Reasoning Only (SayCan-Style)

Early work on LLMs/VLMs for robotics (Ahn et al. [1], Huang et al. [21], Huang et al. [22]) used language models for semantic planning: given a high-level instruction, decompose it into subgoals ("pick up the apple," "move to the table"), then hand each subgoal to a trained low-level policy. This is effective but requires pre-existing low-level skills (navigation policies, grasping policies) trained on robot data. PIVOT's goal is to go directly from VLM to low-level actions without any intermediate trained policy, which means it operates at a fundamentally different (and harder) level of the control stack. The paper notes this limitation implicitly by positioning PIVOT as producing "low-level control of multiple real robot platforms" without any robot data.

2. Learned VLM-to-Action Models (RT-2-Style)

A more recent approach is to fine-tune VLMs to directly output action tokens alongside language tokens. Brohan et al. [4] (RT-2) co-fine-tunes a VLM on both internet vision-language data and robot demonstrations, outputting both text and arm-action tokens. Padalkar et al. [38] (RT-X) extends this across multiple embodiments. These models achieve strong in-distribution results but require fine-tuning on robot data and do not work zero-shot. The paper is explicit about this distinction:

"Unlike these works, we show how VLMs can be applied zero-shot to low-level control of multiple real robot platforms."

This is a critical difference: RT-2 and similar approaches learn a mapping from observations to actions through supervised training on robot demonstrations. PIVOT aims to extract this mapping entirely from the VLM's pretrained knowledge, without any robot-specific parameter updates.

3. Visual Prompting Without Iterative Refinement

The paper builds most directly on works like Yang et al. [59] (Set-of-Mark prompting), which showed that GPT-4V can understand numbered visual annotations overlaid on images and use them for visual reasoning tasks. Other works applied similar ideas to web navigation (Koh et al. [26], Yan et al. [57], Zheng et al. [65]) by annotating UI elements and asking the VLM to select them. The key limitation of these approaches, as the paper frames it, is that they treat proposals as given or generated by a separate perception system:

"instead of taking proposals as given or generating the proposals with a separate perception system, PIVOT generates proposals randomly, but then adapts the distribution through iterative refinement."

A single round of visual prompting with fixed proposals has limited spatial precision: if you annotate 10 candidate arrows on an image, the VLM can only choose among those 10, and the best one might still be far from the correct action. PIVOT's iterative refinement addresses this by progressively concentrating samples around promising regions, achieving finer precision at each round. This is what enables the jump from "select among coarse options" to "produce relatively precise continuous actions."

4. Prompt Optimization in Language Space

Several works have explored optimizing prompts for LLMs/VLMs, either through gradient-based tuning (Lester et al. [28], Li and Liang [29]) or through automatic search in language space (Pryzant et al. [39], Yang et al. [58], Zhou et al. [66]). These methods optimize the textual prompt to produce better outputs. PIVOT is fundamentally different because it optimizes the visual input—the annotated image changes at each iteration, while the text prompt stays largely fixed:

"A major difference between these prior methods and ours is that our iterative prompting uses refinement of the visual input, by changing the visual annotations across refinement steps. We optimize prompts 'online' for a specific query rather than offline to identify a fixed prompt."

This is a conceptual shift: rather than searching for a better prompt that generalizes across examples, PIVOT searches for a better visual representation of candidate solutions for the specific query at hand, analogous to cross-entropy method (CEM) optimization [11].

5. Code-as-Policies and Reward Design Approaches

Other paradigms for using LLMs/VLMs for control include generating code that invokes perceptual APIs (Liang et al. [30], Singh et al. [48]) or generating reward functions that are optimized within a simulator (Huang et al. [23], Yu et al. [62], Ma et al. [35]). These methods are powerful but make strong assumptions: code-writing requires a library of pre-defined perception and control primitives, and reward writing requires a simulator for optimization. PIVOT avoids both requirements—it needs no primitives beyond the ability to project sampled actions into image space (which uses only camera matrices, not learned perception) and no simulator beyond the real world itself.

How the Paper Positions Itself

PIVOT is not presented as a complete solution to zero-shot robotic control, but rather as an approach that establishes what is currently possible and identifies what is not. The paper's abstract is unusually honest about performance:

"Although current performance is far from perfect, our work highlights potentials and limitations of this new regime and shows a promising approach for Internet-Scale VLMs in robotic and spatial reasoning domains."

This framing is important: it positions the work as a capability probe and a methodological contribution, not a benchmark-saturating system. The paper explicitly structures its research questions around both strengths and weaknesses (Section 4, questions 1–5), and devotes an entire section (4.6) to systematic limitations.

The paper also positions itself at the intersection of multiple research communities. It draws from:

• Visual annotations with VLMs [60, 46, 57, 65] for the core idea of rendering choices on images
• Prompt optimization [66, 58, 39] for the iterative refinement methodology
• Foundation models for robotics [13, 19] for the broader agenda of leveraging pretrained models for control
• Cross-entropy method [11] for the optimization algorithm that underlies the iterative likelihood-as-VLM-selection process

By connecting these threads, PIVOT proposes a novel synthesis: visual-space iterative optimization using VLM queries as the selection operator, which is a genuinely new way to extract continuous outputs from discrete-output models. The paper's scaling experiments (Section 4.5, Figure 8) further position PIVOT as a method whose performance is tied to VLM quality—meaning it improves for free as foundation models advance—which makes it a forward-looking contribution rather than one optimized for today's specific model versions.

The key positioning claim, stated twice for emphasis, is:

"Our aim is not necessarily to develop the best possible robotic control or keypoint localization technique, but to study the limits and potentials of such models."

This is a deliberate choice: by avoiding claims of superiority over fine-tuned approaches, the paper can focus on the zero-shot regime as its unique contribution and can treat failure modes as findings rather than weaknesses. This also makes the scaling results (Figure 8) particularly compelling—they frame current limitations as temporary, bounded by today's VLM quality, with a clear trajectory of improvement as models get larger and more capable.

3. Technical Approach

3.1 Reader Orientation

PIVOT is a system that extracts continuous spatial outputs (like robot arm movements or object locations) from vision-language models that were designed to output only text, and it does this without any fine-tuning by repeatedly drawing candidate solutions on an image and asking the model to pick the best ones. The core problem it solves is the output modality mismatch: VLMs produce discrete tokens, but robotics and spatial reasoning require continuous coordinates — PIVOT bridges this gap by converting the continuous output problem into a sequence of visual multiple-choice questions that the VLM can already answer, then using the VLM's selections to iteratively refine the set of choices toward increasingly precise answers.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a loop:

1. Action Sampler — Given a current distribution over the action space (initialized to a broad Gaussian covering all reasonable movements), generates a set of candidate actions by drawing samples. This component is purely statistical and knows nothing about the task.

2. Visual Projection Module (the mapping Ω) — Takes the candidate actions from the sampler and the current camera image, projects each 3D action into 2D pixel space using the robot's camera matrices, renders each action as a numbered visual marker (an arrow with a circled number at its tip) overlaid on the image, and produces an annotated image. This is purely geometric and requires no learned perception.

3. VLM as Selection Operator — Receives the annotated image and a text prompt describing the task, runs chain-of-thought reasoning, and outputs a ranked list of the most promising action numbers. This is a frozen, unmodified GPT-4V or Gemini model queried through its standard API — no fine-tuning, no gradient access.

4. Distribution Fitter — Takes the top-ranked actions selected by the VLM, discards the rest, and fits a new Gaussian distribution (mean and covariance) to these winners. This tightened distribution becomes the input to the Action Sampler for the next iteration.

Information flows in a loop: broad distribution → sample actions → project onto image → VLM selects best → fit tighter distribution → sample again from tighter distribution. This loop repeats 3–4 times, after which a final action is extracted (typically the mean of the final distribution or the VLM's top choice from the last round). The entire process is an instance of the cross-entropy method (CEM) algorithm, but with the VLM playing the role of the cost/quality function that evaluates candidate solutions.

3.3 Roadmap for the Deep Dive

• First, I'll explain the formal problem setup — what $\mathcal{A}$, $I$, $\ell$, and $\pi$ mean concretely in the context of robot control, because everything else builds on these definitions.
• Second, I'll walk through the visual prompt mapping $\Omega$ in detail — how actions become numbered arrows on images, why this specific visual representation was chosen, what the design alternatives were, and how different action space dimensionalities (2D navigation vs. 4D manipulation) are handled.
• Third, I'll explain the iterative optimization loop — Algorithm 1 line by line — covering sampling, VLM querying, distribution fitting, and termination, including the crucial but subtle relationship to the cross-entropy method that justifies why this iterative procedure should converge to better actions.
• Fourth, I'll detail the parallel call robustness mechanism — why VLM stochasticity necessitates running multiple independent PIVOT instances and how their outputs are aggregated, since this is essential for the practical performance reported in Tables 1–2.
• Fifth, I'll cover the text prompting strategy — chain-of-thought vs. direct, zero-shot vs. few-shot, and prompt ordering, with the specific prompt templates used in the manipulation experiments — because the VLM's selection quality depends critically on how the question is framed.
• Sixth, I'll discuss the distribution parameterization and design choices — why isotropic Gaussians, how the initial distribution is set per embodiment, what happens when the VLM selects zero actions, and how the number of samples $M$ trades off coverage against visual occlusion.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a method paper whose core idea is that a frozen VLM can serve as the selection/quality operator inside a cross-entropy-method optimization loop if the candidate solutions are rendered visually on the input image, enabling zero-shot continuous-output tasks (robot control, keypoint localization) from a discrete-output model.

---

Formal Problem Statement and Notation

The paper defines the problem as producing an action $a \in \mathcal{A}$ from a continuous action space $\mathcal{A}$ given a natural language task description $\ell \in \mathcal{L}$ and an image observation $I \in \mathbb{R}^{H \times W \times 3}$. When $\mathcal{A}$ is the set of robot actions, this amounts to finding a policy $\pi(\cdot \mid \ell, I)$ that produces an action.

Let me unpack this notation concretely for the specific embodiments in the paper because the shape of $\mathcal{A}$ varies dramatically across experiments, and understanding this variation is essential to appreciating the flexibility of the approach:

Mobile manipulator navigation (Section 4.1, Figure 3a): $\mathcal{A}$ is a 2D space — each action $a$ represents a target location in the camera's image plane $(u, v)$. The robot uses its onboard depth camera to project this 2D image point into a 3D world coordinate, then commands the mobile base to move toward that location with a maximum displacement of 1.0 meters per action step. So although the action space is 2D pixels, the executed movement is 3D navigation, with the depth camera handling the missing third dimension implicitly.

Mobile manipulator manipulation (Section 4.1, Figure 3b): $\mathcal{A}$ is a 4D space — three continuous dimensions for relative Cartesian end-effector displacement $(x, y, z)$ plus a binary gripper dimension (open/close). The $(x, y, z)$ delta is expressed in the camera frame, meaning the $x$-axis points into and out of the image plane (depth), $y$ is horizontal across the image, and $z$ is vertical. The gripper action is handled separately through text instructions rather than visual annotations, because the paper found that expressing discrete motion types (grasp, release) through visual markers was less reliable than simply instructing the VLM in text when to close or open the gripper.

Franka arm (Appendix F, Figure 3c): $\mathcal{A}$ is a 4D space identical in structure to the mobile manipulator's manipulation space — relative $(x, y, z)$ Cartesian deltas plus gripper — but the camera is wrist-mounted (first-person view) rather than head-mounted (third-person view), which changes the visual perspective dramatically.

RAVENS simulation (Appendix E, Figure 3d): $\mathcal{A}$ is a 4D space representing pick and place locations in the overhead camera's pixel coordinates — $(u_{\text{pick}}, v_{\text{pick}}, u_{\text{place}}, v_{\text{place}})$. This is a higher-level action space where a single action specifies both the grasp point and the release point, in contrast to the relative-delta formulation where each action moves the arm by a small increment.

RefCOCO localization (Section 4.3): $\mathcal{A}$ is a 2D pixel space $(u, v)$ representing the center of the target object's bounding box. This demonstrates that the method generalizes beyond robot control to any spatial reference task.

The VLM being queried has the interface $P_{\text{VLM}}(\cdot \mid w_p, I)$, where $w_p$ is a textual prefix (the prompt, which can include system instructions, task descriptions, and formatting requirements) and $I$ is the input image. The model outputs a distribution over textual completions — sequence of tokens. The fundamental challenge is that $\mathcal{A}$ is continuous, but the VLM can only produce tokens from a discrete vocabulary. PIVOT's solution is to not ask the VLM to generate actions directly, but instead to generate visual representations of candidate actions and ask the VLM to select among them using its existing visual reasoning capabilities.

---

The Visual Prompt Mapping $\Omega$

The visual prompt mapping is the bridge between the mathematical world of continuous actions and the VLM's world of images and text. It is defined as:

$(\hat{I}, w_{1:M}) = \Omega(I, a_{1:M})$

where $I \in \mathbb{R}^{H \times W \times 3}$ is the original camera image, $a_{1:M} = \{a_1, a_2, \ldots, a_M\}$ is a set of $M$ candidate actions, $\hat{I}$ is the output annotated image (same dimensions as $I$, but with visual markers overlaid), and $w_{1:M} = \{w_1, w_2, \ldots, w_M\}$ is a set of textual labels where $w_j$ is the string representation (typically a number like "0", "1", …, "9") that the VLM can output to refer to action $a_j$.

What it computes: given a list of candidate actions and the current visual observation, it produces a new image where each action is visually rendered as a mark in the image, and simultaneously produces the set of text tokens that the VLM should use to refer to those marks. The output pair $(\hat{I}, w_{1:M})$ makes the VLM's input-output cycle exactly a visual multiple-choice question: look at the annotated image, read the task description, and output the numbers corresponding to the best actions.

Why this form: the mapping separates the geometric projection (which actions in continuous space become which pixel locations) from the rendering style (what visual markers look like). This separation is crucial because it means the action sampling and distribution fitting happens entirely in the continuous action space $\mathcal{A}$, while the VLM only ever sees and reasons about the rendered image. The VLM never needs to understand coordinates, inverse kinematics, or depth — it just needs the general visual reasoning capability that it acquired during pretraining to recognize which numbered arrows point toward task-relevant objects or directions.

The paper investigates several specific rendering choices for $\Omega$:

Arrow representation for robot actions (Figure 1, Figure 2): Each candidate action is rendered as an arrow emanating from either the robot's end-effector position in the image (for manipulation, where the arrows originate from the gripper's current pixel location) or from the bottom center of the image (for navigation, where the embodiment isn't always visible). The arrow's direction and magnitude in pixel space correspond to the action's $(x, y)$ components after projection through the camera matrix. Each arrow terminates in a circled number label that identifies it — the VLM selects actions by outputting these numbers. The arrow representation was chosen because it provides an intuitive visual metaphor for "which direction should the robot move?" that aligns with how humans think about motion, and because modern VLMs have been shown to understand arrow annotations [46, 59, 60].

Depth encoding via color and size (Section 3.4, Appendix A): For 3D manipulation tasks where actions include a $z$ (depth) component, the paper uses two redundant visual cues to encode the third dimension. Color: arrows use a red-to-blue spectrum — red arrows indicate movement away from the camera (increasing depth, forward motion), while blue arrows indicate movement toward the camera (decreasing depth, backward motion). Marker size: the terminal circles have varying sizes — smaller circles represent actions farther from the camera (forward), while larger circles represent actions closer to the camera (backward). The text prompt explicitly describes this encoding to the VLM:

"Red arrows move the arm farther away from the camera, blue arrows move the arm closer towards the camera. Smaller circles are further from the camera and thus move the arm farther, larger circles are closer and thus move the arm backwards."

This dual encoding (color + size) is a redundancy strategy: if the VLM fails to interpret one cue, the other still carries the depth information. However, Section 4.6 reports that this encoding is not fully reliable — "none of the VLMs we tested are capable of reliably choosing actions based on depth" — which represents a fundamental limitation of the 2D-only training data these models receive.

Numbered circles for RefCOCO (Section 4.3, Appendix H.1): For spatial localization, the visual markers are simply numbered circles placed at candidate pixel coordinates, without arrows. This is because localization requires selecting a point, not a direction — the action is a position, not a displacement vector. The circles are drawn at the projected 2D locations of sampled candidate points.

Gripper actions handled in text (Section 3.4): The gripper open/close dimension is explicitly not rendered visually. Instead, the text prompt instructs the VLM about when to grasp: "The robot can only grasp or move objects if the robot gripper is close to the object and the gripper fingers would stably enclose the object." The decision to lift (close gripper) or release (open gripper) is inferred from positional context: if the VLM selects arrows that bring the gripper to an object, the system executes a grasp after reaching; if the task requires placing, the system releases after arriving at the target location. This design choice reflects a practical judgment: rendering grasp states as visual annotations would add complexity without clear benefit, and VLMs are already proficient at understanding textual descriptions of grasp affordances.

Number of samples $M = 10$ (Section 4.4, Figure 7): The paper ablates over the number of candidate actions rendered per iteration and finds a non-monotonic effect. With few samples (e.g., $M = 3$), the initial coverage of the action space is poor — the VLM may not see any good candidate even on the first round. With many samples (e.g., $M \geq 15$), visual clutter becomes a problem: arrows overlap, occlude task-relevant objects, and confuse the VLM. The optimum is $M = 10$, which balances distributional coverage against visual clarity. This is a hardware-capability-dependent hyperparameter specific to current VLM visual resolution limits; future models with higher-resolution image processing or better robustness to visual clutter might shift this optimum upward.

Annotation style sensitivity (Appendix G, Tables 6–7): The paper tests how sensitive VLM arrow understanding is to rendering parameters like color, thickness, arrowhead size, and direction. On a synthetic blank-background dataset (Table 6), GPT-4V achieves 88–100% accuracy across most color and size variations for classifying absolute arrow directions — the model can reliably tell which way an arrow points in isolation. On a realistic object-referential dataset (Table 7), where the task is "select the arrow pointing at [object]" among distractors, performance drops dramatically: accuracy ranges from 17% to 83% depending on color, size, and object difficulty, with "Very Hard" objects (brushes, eccentric items) receiving 0–44% across all color conditions. This reveals that while VLMs can understand arrows as directional symbols, using arrows to ground references to specific objects in cluttered scenes is substantially harder, especially for visually unusual objects. The paper does not, however, ablate whether different visual marker types (crosshairs, dots, oriented lines, heatmaps) might be more effective than arrows — this remains an open design space.

---

The Iterative Optimization Loop (Algorithm 1)

Algorithm 1 describes a loop that executes $N$ times (typically $N = 3$ for real-robot experiments or $N = 1-3$ for ablations) to progressively refine from a broad initial action distribution to a concentrated one around the VLM's preferred actions. The algorithm can be understood as an instance of the cross-entropy method (CEM) [11], though the paper uses this connection loosely as a conceptual analogy rather than a formal equivalence — the VLM's selection mechanism is not a mathematically defined quality function with known properties, so the CEM convergence guarantees do not strictly apply.

Step-by-step walkthrough:

Step 1 — Initialization: A distribution $P_{\mathcal{A}^{(0)}}$ over the action space is initialized. The paper approximates this as an isotropic Gaussian (circularly symmetric, equal variance in all dimensions) with its mean set to a neutral action (zero displacement for manipulation; center-bottom of image for navigation) and its variance set to cover the plausible range of actions. The paper does not report specific variance values for each embodiment in the main text, but the principle is that the initial distribution should have high enough variance to include good actions but not so high that most samples are physically impossible or project off-screen. The initialization is the only part of PIVOT that is embodiment-specific beyond the camera projection — different robots have different physical action limits, and these are encoded in the Gaussian's variance.

Step 2 — Sample actions (line 4): $M$ actions $a_{1:M}^{(i)}$ are drawn i.i.d. from the current distribution $P_{\mathcal{A}^{(i)}}$. At iteration $i = 0$, these samples are spread broadly across the action space; at later iterations, they concentrate around previously selected winners.

Step 3 — Project and annotate (line 5): The $M$ sampled actions are fed through $\Omega$ along with the current image $I$ to produce annotated image $\hat{I}^{(i)}$ and associated label strings $w_{1:M}^{(i)}$. This is purely a rendering operation — no VLM involvement yet.

Step 4 — Query the VLM (line 6): The VLM receives the annotated image $\hat{I}^{(i)}$, the task description $\ell$ (e.g., "Pick up the coke can" or "Help me find a place to sit and write"), and a text prompt that frames the decision as a multiple-choice question over the action labels $w_{1:M}^{(i)}$. The VLM processes the image and text autoregressively, ultimately outputting a ranked list of the most promising action numbers. The paper specifies that the VLM is instructed to use chain of thought reasoning:

"we prompt the VLM to use chain of thought to reason through the problem and then summarize the top few labels" (Section 3.4)

The VLM thus produces both a reasoning trace (explaining why certain arrows are better — e.g., "arrow 3 points toward the coke can, while arrow 7 would move the gripper away from the table") and a structured action list. The VLM is instructed to return 1–4 candidate actions ranked from worst to best (the paper notes "a general rule of thumb is to return 1-4 candidates"). The output format varies by task, but for manipulation the prompt specifies:

"Reason through the task first and at the end summarize the correct action choice(s) with the format, 'Arrow: [<number>, <number>, etc.]'" (Appendix H.4, Direct prompt)

The maximum number of returned candidates matters because it controls how aggressively the distribution shrinks: returning only 1 action per iteration produces rapid convergence but risks over-committing to VLM mistakes; returning many actions is conservative but requires more iterations to achieve precision.

Step 5 — Fit new distribution (line 7): The VLM's selected actions are extracted from its text output by parsing the bracketed number list. These selected actions — the "elite set" in CEM terminology — are used to fit a new Gaussian distribution $P_{\mathcal{A}^{(i+1)}}$. The fitting procedure computes the empirical mean and covariance of the chosen actions. Because the distributions are approximated as isotropic Gaussians, the fitting reduces to:

• Mean: the component-wise average of the selected action vectors
• Variance: the component-wise variance of the selected action vectors, constrained to be the same for all dimensions (isotropy assumption)

The paper does not specify whether a minimum variance floor is enforced — in standard CEM, a small minimum variance is typically maintained to prevent premature convergence and allow the sampler to escape from VLM mistakes. Given that the paper reports improvement over iterations (Figure 6, Figure 7) without catastrophic collapse, it is likely that either the VLM's natural stochasticity or an implicit variance floor prevents degenerate distributions.

Step 6 — Iterate (line 8): The counter increments, and the loop repeats from Step 2 using the tightened distribution. The paper uses $N = 3$ iterations for most experiments (Tables 1–2, Figures 6–7), which represents a design tradeoff: each iteration requires one VLM API call (costly in both latency and possibly dollars), and diminishing returns set in after the distribution concentrates. Figure 6 shows that going from 1 to 2 iterations improves performance, and 3 iterations provides further but smaller gains — beyond 3 iterations, the distribution is typically concentrated enough that further refinement yields negligible improvement at the cost of another API query.

Step 7 — Return final action (line 10): After the final iteration, a single action is extracted for execution. The paper describes this as "an action from the VLM best actions" — in practice, this is the mean of the final distribution $P_{\mathcal{A}^{(N)}}$ (the maximum a posteriori estimate under the Gaussian assumption) or the VLM's top-ranked action from the final iteration, depending on the embodiment. For closed-loop control, this action is executed on the robot, a new camera image is captured, and the entire PIVOT process repeats from scratch for the next control step.

The fundamental computational loop: An entire PIVOT cycle (sampling → rendering → VLM query → fitting) is repeated for each action step of the robot. A manipulation task requiring 3 action steps with 3 PIVOT iterations each requires 9 total VLM queries (plus any parallel call overhead — see below). This makes PIVOT substantially more computationally expensive than learned policies (which produce actions in a single forward pass) but the cost is the price of zero-shot capability.

---

The Optimization Formulation (Equation 2)

The paper frames PIVOT as solving an optimization problem:

$\max_{a \in \mathcal{A}, w} P_{\text{VLM}}(w \mid \hat{I}, \ell) \quad \text{s.t.} \quad (\hat{I}, w) = \Omega(I, a)$

where $P_{\text{VLM}}(w \mid \hat{I}, \ell)$ is the probability (or, more accurately, the unnormalized preference score) that the VLM assigns to textual label $w$ given the annotated image $\hat{I}$ and the task description $\ell$, the constraint requires that $(\hat{I}, w)$ is exactly the visual mapping of the action $a$, and the maximization is over both the continuous action $a$ and the discrete label $w$ that corresponds to it.

What it computes: we want the single action $a$ whose visual representation $w$ the VLM would select as the best choice for the task when shown among candidates. More precisely, given the VLM's probability of outputting each label token, we want the action whose associated label achieves the highest probability — the VLM is most likely to say "3" when arrow 3 is the correct action.

Why this form: this formulation converts the problem of "generate a continuous action" into "find the action whose visual marker the VLM prefers," which is a problem that can be attacked with black-box optimization because we can evaluate $P_{\text{VLM}} (w \mid \hat{I}, \ell)$ for any candidate action $a$ by rendering it and querying the VLM. The constraint $(\hat{I}, w) = \Omega(I, a)$ ensures that we only consider actions that can be rendered — we cannot ask the VLM about a label that doesn't correspond to any rendered marker.

However, the paper does not actually use the VLM's token probabilities $P_{\text{VLM}}(w \mid \hat{I}, \ell)$ directly. GPT-4V and Gemini, as accessed through their APIs, do not expose full token-level log probabilities for all label candidates — they only provide the sampled text output. So PIVOT uses the VLM's ordinal ranking output ("Arrow: [3, 7, 1]") as a proxy for the underlying probabilities, treating any action that appears in the ranked list as "selected" and any that doesn't as "rejected." This is a lossy but practical approximation: the VLM's internal probability ordering over the labels is reduced to a binary in/out decision plus a rank order within the selected set.

---

Parallel Call Robustness (Section 3.3)

A critical practical issue is that VLMs are stochastic: the same image and prompt can produce different outputs on different calls due to sampling in the autoregressive decoding process. A single PIVOT instance can therefore converge to a suboptimal action if the VLM makes a ranking error in an early iteration — once the distribution tightens around the wrong region, subsequent samples are unlikely to explore the correct region again even if the VLM would have preferred it.

To mitigate this, the paper introduces a parallel call strategy:

1. Run $E$ independent PIVOT instances in parallel (where $E$ ranges from 2 to 3 in experiments). Each instance receives the same image and task description but has independent randomness in both the action sampling (Step 2) and the VLM query (Step 4). The paper refers to these as "parallel calls" — each runs the full Algorithm 1 loop to convergence independently of the others.

2. Aggregate the $E$ final actions from the parallel instances using one of two strategies:

   • Distribution-based aggregation: Fit a single Gaussian to the $E$ action candidates returned by the parallel PIVOT runs, and take the mean of this meta-distribution as the final action. This treats each PIVOT instance as a "vote" for a region of the action space and combines them democratically.
   • VLM re-selection: Render the $E$ final actions on the image (as numbered markers) and query the VLM one final time to select the single best action among them. This adds one additional VLM call but allows the model to make a direct comparison between the converged outputs of different parallel runs, which can be more reliable than statistical averaging when the parallel runs have converged to very different regions.

The paper reports that parallel calls "effectively improve the robustness of PIVOT and mitigate local minima in the optimization process" (Section 3.3), and Tables 1–2 show that configurations with 3 parallel calls generally outperform those without.

Computational cost of parallels: $E$ parallel instances with $N$ iterations each require $E \times N$ VLM queries (plus one additional query for VLM re-selection aggregation). For $E=3$ and $N=3$, this is 9 queries per action step — approximately a 10–30 second latency bottleneck depending on API response times, which is why the paper also reports efficiency metrics like "Steps" (number of action steps to completion) — faster convergence to the target means fewer total VLM queries despite the per-step cost.

---

Text Prompting Strategy and Design Choices

The text prompt is the VLM's only non-image input and critically shapes which actions it selects. The paper ablates several prompting dimensions (Section 4.4, Appendix D, Figures 13–14):

Zero-shot chain of thought performs best (Figure 13). The paper compares four prompt types in offline manipulation evaluation:

• Zero-shot direct: Ask the question with no reasoning instruction and no examples. Example (Appendix H.5): "Task: What are the best arrows for the robot follow to pick white coat hanger? ... direct answer with the format, Arrow: [<number>, <number>, etc.]"
• Zero-shot chain of thought: Ask the VLM to reason before answering. Example: "Reason through the task first and at the end summarize the correct action choice(s) with the format, Arrow: [<number>, <number>, etc.]"
• Few-shot direct: Provide example task-image pairs with the correct arrow answers, but no reasoning chain. Example (Appendix H.5): Three examples showing "Task: Erase the writing on the whiteboard. Arrow: [5, 10]" then the target task.
• Few-shot chain of thought: Provide examples with explicit reasoning chains. Example: "Task: Erase the writing on the whiteboard. The robot is holding an eraser, so it should move it over the marker on the whiteboard. The following arrows look promising: 5. This arrow moves the eraser over the writing and away from the camera and thus towards the whiteboard. 10. ... Arrow: [5, 10]"

The finding: zero-shot CoT achieves the best performance, but few-shot direct prompting is "close and more token efficient." This is counterintuitive — one might expect few-shot examples to help the VLM understand the task format and visual annotation conventions — but the paper hypothesizes that zero-shot avoids distribution shift from the specific few-shot examples (which show different scenes, objects, and tasks with potentially different arrow conventions). The chain of thought instruction improves performance because it forces the VLM to ground its choices in visual features ("arrow 3 points toward the coke can") rather than picking numbers by superficial pattern matching.

Prompt ordering matters marginally (Figure 14). The prompt has three distinct elements:

• Preamble: A system-level description of the setup, including what the arrows represent and how to interpret color/size cues. Example: "The arrows are actions the robot can take. Red means move the arm forward (away from the camera), blue means move the arm backwards (towards the camera). Smaller circles are further from the camera and thus move the arm forward, larger circles are closer and thus move the arm backwards."
• Image: The annotated image $\hat{I}^{(i)}$ — included inline in the VLM's input.
• Task: The specific instruction, e.g., "Pick up the coke can" or "What are the best arrows for the robot to follow to pick up the coke can?"

The paper tests all six possible orderings of these three elements and finds that preamble → image → task performs best, though "by a small margin" — the differences are not dramatic. The rationale: putting the image after the preamble provides context for interpreting the visual annotations (the VLM has already read that red = forward, blue = backward when it encounters the colored arrows), and putting the task last focuses the VLM's generation on the specific question rather than losing it in preamble details.

Task-specific prompt templates (Appendix H): Each embodiment/domain has its own prompt template, adapted to the specific visual encoding and task format:

• Navigation: "I am a wheeled robot that cannot go over objects. This is the image I'm seeing right now. I have annotated it with numbered circles. Each number represent a general direction I can follow. ... Choose {K} best candidate numbers. Do NOT choose routes that goes through objects."
• RefCOCO: "I have annotated the image with numbered circles. Choose the 3 numbers that have the most overlap with the OBJECT. If there are no points with overlap, then don't choose any points."
• RAVENS: "which number markers are closest to the {OBJECT}? Reason and express the final answer as 'final answer' followed by a list of the closest marker numbers."

These prompts are designed through manual engineering — the paper does not report automated prompt optimization or a systematic prompt search. This is a limitation: it is possible that better prompts exist that would significantly improve performance, and different VLMs might prefer different prompt styles.

---

Distribution Parameterization and Design Choices

The action distributions $P_{\mathcal{A}^{(i)}}$ in Algorithm 1 are approximated as isotropic Gaussians — Gaussian distributions where the covariance matrix is a scalar multiple of the identity matrix ($\sigma^2 I$), meaning there is equal variance in all dimensions and zero correlation between dimensions.

This is a simplifying approximation rather than a principled choice. A full-covariance Gaussian would allow the distribution to capture correlations between action dimensions (e.g., for manipulation, actions that move forward often also move slightly downward in a coordinated way), but fitting a full covariance matrix from a small number of VLM-selected actions (1–4 per iteration) is statistically unreliable. The isotropic assumption means the distribution shrinks as a sphere around the VLM's preferred actions, which is a conservative choice that avoids over-committing to spurious correlations estimated from few samples.

The fitting procedure from selected actions to new distribution handles special cases:

• If the VLM selects exactly one action: the mean is that action, and the variance is heuristically reduced (likely halved, though the paper does not specify the exact shrinkage factor) to concentrate samples around the winner.
• If the VLM selects multiple actions: the mean and variance are computed empirically from the selected set. The variance naturally shrinks as the selected actions become more clustered across iterations.
• If the VLM selects zero actions: this is not explicitly discussed but would require a fallback — likely maintaining the previous iteration's distribution or expanding variance to explore more broadly.

The initial distribution's variance is embodiment-specific and must be set manually: too large, and samples project off-screen or outside the robot's physical workspace; too small, and the VLM never sees good actions and cannot provide useful feedback. The paper does not provide exact variance values for each embodiment, which is a missing implementation detail that would be necessary for reproduction.

Number of iterations $N$: The paper uses $N = 3$ for most real-robot experiments and ablates up to $N = 3$ in offline evaluations. The choice of 3 appears to be empirically motivated: performance improves from 1 to 2 to 3 iterations in both the navigation offline eval (Table 3) and the manipulation offline eval (Figures 6, 7), but the paper does not test larger $N$ values to determine if returns fully saturate. Given the per-iteration VLM API cost, the practical optimum is likely between 2 and 4, trading off precision against latency and expense.

---

Comparison to Standard Cross-Entropy Method

PIVOT is explicitly analogous to the cross-entropy method (CEM) [11], a general-purpose optimization algorithm for continuous spaces. In standard CEM:

1. Sample candidates from a parameterized distribution (typically Gaussian)
2. Evaluate each candidate with a scalar quality function $f(a) \in \mathbb{R}$ (higher is better)
3. Select the top $k$% of candidates (the "elite set")
4. Fit a new distribution to the elite set
5. Repeat

PIVOT maps onto this structure with the VLM serving as the quality function, but with several crucial differences:

• The VLM is not a scalar function. Instead of returning $f(a) \in \mathbb{R}$ for each action, the VLM returns a ranked subset — a list of the top 1–4 actions. This means there is no notion of "how much better" one action is than another, only "in the selected set" vs. "not in the selected set." Standard CEM's elite fraction $\rho$ is therefore implicitly determined by the VLM's output format: if the VLM selects $k$ out of $M$ candidates, the effective elite fraction is $k/M$.

• The VLM is stochastic and biased. A standard CEM quality function is deterministic (same action always returns the same score). The VLM can return different rankings for the exact same annotated image on different calls due to sampling in autoregressive decoding. Moreover, the VLM can make systematic errors — preferring actions that are visually salient but task-irrelevant, misunderstanding depth cues, or hallucinating justifications. These errors do not have the statistical properties (zero-mean, independent) that CEM's convergence proofs assume.

• The VLM evaluates actions jointly, not independently. In CEM, each candidate is scored independently: $f(a_j)$ depends only on $a_j$, not on the other candidates. The VLM, however, sees all $M$ candidates rendered simultaneously on the image and makes comparative judgments — "arrow 3 is better than arrow 7." This means the VLM's selection is context-dependent: the same action might be selected when surrounded by poor alternatives but rejected when flanked by even better ones. Joint evaluation is actually an advantage for PIVOT, because relative comparisons are often more reliable than absolute scoring, but it breaks the standard CEM assumption of independent evaluations.

• The quality function operates in visual space, not action space. The VLM never sees the actions $a_{1:M}$ directly — it sees the rendered image $\hat{I}$. This means the VLM's quality assessment depends on the rendering quality, which can introduce artifacts: an action that is objectively correct might be rendered with its label partially occluded by an object, making the VLM less likely to select it; conversely, a suboptimal action might be rendered in a visually prominent region and selected due to visual salience rather than task suitability.

Despite these differences, the CEM analogy provides the conceptual justification for why the iterative procedure should work: each iteration concentrates probability mass around actions the VLM prefers, which increases the density of good candidates in the next round, which in turn makes it easier for the VLM to identify even better actions among even more refined options. The convergence behavior — improving from iteration 1 to 2 to 3 and then plateauing — is consistent with CEM-like distribution collapse.

4. Key Insights and Innovations

Innovation 1: Reframing Continuous Action Generation as Visual-Space Cross-Entropy Optimization

The paper's most fundamental conceptual move is not the specific rendering choices or the parallelization scheme, but the reframing of what it means to extract a continuous output from a discrete-output model. Before PIVOT, the dominant strategies for getting VLMs to produce spatial outputs fell into two camps: (1) learn a mapping from VLM representations to actions via fine-tuning on robot data (RT-2 [4], RT-X [38], VIMA [25]), which works well but requires in-domain supervision, or (2) ask the VLM to generate coordinates as text (e.g., "output (x=342, y=218)"), which fails because precise numerical regression in token space is far outside the VLM's training distribution. Yang et al. [59] showed that VLMs can select among visually annotated options, but treated proposals as given by an external perception system, not as something to be optimized over.

PIVOT introduces a genuinely new third category: treat the VLM as the quality evaluator inside a black-box optimization loop that operates entirely in visual space. This is a conceptual synthesis that reinterprets the VLM's visual reasoning capability — which was designed for tasks like "describe this image" or "answer this question about the scene" — as a learned scoring function for spatial proposals. The VLM is never told coordinates, never generates actions, and never receives gradients; it simply looks at pictures of candidate actions and says which ones look promising. The cross-entropy method [11] provides the outer optimization wrapper, and the VLM's preference judgments drive the distribution updates.

What makes this framing distinctive is that it collapses the distinction between perception and control. In a standard robotics pipeline, you have a perception module that identifies objects/regions, a planning module that decides what to do, and a control module that produces motor commands. PIVOT's VLM does all three simultaneously from a single query: it perceives the scene, reasons about the task, and selects the motor-relevant arrow — but it does so through a unified visual reasoning process that was never explicitly trained for any of these subtasks. The fact that this works at all (non-zero success across real robots with GPT-4V, no fine-tuning) is evidence that the VLM's pretraining has implicitly learned spatial affordances and task-relevant visual preferences, even without robot data in the training mixture.

This reframing also changes the research question itself. Before, the question was "how do we train VLMs to output actions?" — a supervised learning problem. After PIVOT, the question becomes "how well can VLMs evaluate candidate actions when they're rendered visually, and what are the limits of this evaluation capability?" — a capability assessment problem. The paper's scaling experiments (Section 4.5, Figure 8) are a natural consequence of this reframing: if VLM evaluation quality improves with model scale, then PIVOT performance inherits these gains without any change to the algorithm. This makes PIVOT a forward-looking contribution whose value increases with foundation model progress, rather than a fixed-capability system.

The significance of this reframing extends beyond robotics: any domain where a VLM needs to produce structured continuous outputs — molecular conformations, architectural layouts, UI design coordinates, sports strategy positions — could potentially be cast as an iterative visual optimization problem. The paper demonstrates this generality by applying the same algorithm to RefCOCO keypoint localization (Section 4.3) without any domain-specific modification, establishing that the approach is not robot-specific but is a general method for spatial reasoning from VLMs.

Innovation 2: Demonstrating That Visual Annotation Quality Is the Primary Bottleneck, Not VLM Reasoning Capability

A second major conceptual contribution is the paper's diagnostic decomposition of failure modes, which reveals that current VLMs fail at spatial tasks primarily because of limitations in understanding the visual annotations themselves, not because of failures in task reasoning or object recognition. This is a subtle but important distinction that redirects where improvement efforts should focus.

The paper provides converging evidence for this claim from multiple angles:

Appendix G (Tables 6–7) directly tests VLM ability to parse arrow annotations as a function of rendering parameters. The results are revealing in their asymmetry: on a blank background with a single arrow (Table 6), GPT-4V achieves near-perfect accuracy at classifying the arrow's absolute direction across colors, thicknesses, and sizes — the model clearly understands what an arrow is and which way it points. But in the realistic setting with object-referential arrows (Table 7), accuracy collapses to 17–83% depending on color and object type, with "Very Hard" objects (brushes, eccentric items) scoring 0–44% across all color conditions. This means the VLM's problem is not arrow comprehension per se — it is disambiguating which arrow points at which object when multiple arrows and multiple objects overlap in a cluttered scene. The visual annotation itself introduces a perceptual bottleneck that is independent of task understanding.

The depth encoding failure (Section 4.6) provides a second diagnostic angle. The paper carefully designed a dual-channel depth encoding (color spectrum + circle size) and explicitly described it in the text prompt, yet reports that "none of the VLMs we tested are capable of reliably choosing actions based on depth." This is not a reasoning failure — the VLM was told the rules and can presumably understand them — but a perception failure: the depth cues as rendered in 2D do not reliably convey 3D information in a way the VLM can extract. This is fundamentally a training data limitation: these VLMs were trained on 2D images from the internet, where depth must be inferred from monocular cues like perspective and occlusion, not from explicit color-coded depth maps.

The visual clutter ablation (Section 4.4, Figure 7) provides a third angle: performance degrades when more than 10 samples are rendered because "the region of the image around the correct answer gets crowded and causes significant issues with occlusions." This is a visual capacity limit, not a reasoning one — the VLM can reason about the task perfectly well but cannot disambiguate closely spaced, overlapping visual markers.

This diagnostic decomposition challenges the natural assumption that the bottleneck is VLM reasoning quality (scaling model size will fix it) and instead points toward visual representation design as the critical lever. The implications are concrete: future work should prioritize developing visual annotations that VLMs can robustly parse under clutter, that convey 3D information in ways compatible with monocular pretraining, and that remain discriminable when objects are partially occluded. Improving the VLM's reasoning about which object to approach is less urgent than improving the VLM's ability to see which arrow points to that object.

This insight also explains why the text-only baseline performs so much worse (Figure 6, Figure 7): when actions are expressed as discrete language choices ("move right," "move left," "top middle region"), the VLM must translate language into spatial coordinates, a task at which it is even less reliable than parsing visual annotations. Visual prompting is essential not because VLMs are good at understanding arrows (they're mediocre) but because language-based spatial reasoning is worse — the visual channel at least provides a direct perceptual link between the candidate action and the scene, even if that link is noisy.

Innovation 3: Parallel Calls as a Practical Robustness Mechanism for Stochastic Black-Box Optimizers

While the idea of running multiple trials and aggregating results is not novel in general, PIVOT's specific application of parallel independent CEM instances followed by VLM re-selection is a distinctive contribution to the problem of optimizing with a stochastic, non-differentiable, and potentially adversarial quality function. The VLM-as-quality-function has several pathological properties that standard optimization algorithms are not designed to handle:

• Stochasticity with unknown distribution: The VLM can return different rankings for identical inputs, but the noise is not zero-mean or symmetric — it can be systematically biased toward visually salient but task-irrelevant options.
• Catastrophic early errors: An error in iteration 1 (selecting the wrong region) causes all subsequent iterations to sample from a tightened distribution around the wrong region, with essentially zero probability of recovery because the distribution has moved away from the true optimum.
• Context-dependent evaluation: The same action can be selected or rejected depending on what other candidates it competes against visually, making the effective quality landscape non-stationary across iterations.

The parallel call strategy (Section 3.3) addresses these pathologies through a simple but effective mechanism: run E independent PIVOT instances from the same initial distribution and aggregate their final outputs. Since each instance has independent sampling noise and independent VLM stochasticity, they explore different regions of the action space. If some instances fall into local minima (VLM errors steer them toward suboptimal regions), others may still converge toward the correct action. The aggregation step — either fitting a meta-distribution to the E final actions or using the VLM to directly compare them — surfaces the consensus while suppressing outliers.

The evidence supports this mechanism: Tables 1–2 show consistent improvements from adding parallel calls. For example, in navigation, "Go to orange table with tissue box" achieves 25% with no iterations and no parallel, 50% with 3 iterations and no parallel, but 75% with either no iteration and 3 parallel, or 3 iterations and 3 parallel — suggesting that parallel exploration alone can compensate for the lack of iterative refinement, and the combination is best. In manipulation (Table 2), "Pick coke can" goes from 0% grasp rate (no iteration, no parallel) to 33% (3 iterations, no parallel) to 67% (3 iterations, 3 parallel) — the parallel calls roughly double the grasp success beyond what iterations alone achieve.

This is not simply "majority voting" over VLM outputs — it is a hierarchical optimization strategy where the lower level (each PIVOT instance) runs local optimization via CEM, and the upper level (aggregation) combines the results of independent local searches. This two-tier structure is conceptually similar to population-based optimization methods (e.g., evolutionary strategies with restarts), but adapted to the specific properties of VLM-based evaluation. The fact that it works with only E = 2–3 parallel instances — rather than the hundreds typical in population methods — reflects that the VLM's evaluation quality is high enough that each local search has a reasonable probability of finding a good region, and the aggregation mainly needs to filter out the occasional catastrophic failure rather than search a vast space.

The implication for future work is that test-time compute scaling for VLM-based optimization may follow a parallel-then-aggregate pattern analogous to how best-of-N sampling works for language outputs — spend more compute by running more independent optimization trajectories and selecting among them, rather than by making any single trajectory longer (more iterations). The paper does not fully explore this parallel-vs-iterations tradeoff surface, but the existing results suggest that parallel calls provide robustness benefits that additional iterations cannot match due to the non-recoverable nature of early optimization errors.

Innovation 4: Establishing VLM Spatial Reasoning as a Scaling Phenomenon with Sharp Capability Boundaries

The paper's scaling experiments (Section 4.5, Figure 8) demonstrate that PIVOT's spatial reasoning performance improves monotonically with VLM size across four variants of the Gemini model family, on both manipulation and navigation tasks. This is significant not because it is surprising — larger models generally perform better — but because it establishes that zero-shot spatial reasoning from VLMs is a genuine emergent capability that scales with model capacity, not a brittle artifact of a particular model's idiosyncratic training. It reframes PIVOT as a capability probe whose results at any given moment represent a lower bound on what will be possible with next-generation models.

However, the paper's contribution here goes beyond the standard "bigger is better" scaling narrative. The more distinctive insight is the identification of capability boundaries that may persist even with scaling. The depth reasoning failure (Section 4.6) — "none of the VLMs we tested are capable of reliably choosing actions based on depth" — is particularly diagnostic. This is not a failure of insufficient capacity; it is a failure of insufficient training signal. Internet images lack explicit depth annotations, so even a perfectly scaled VLM trained on internet data would have no reason to learn that red = forward and blue = backward in a color-coded action visualization. The encoding is arbitrary and task-specific — no amount of model scale can recover information that is not present in the training distribution.

This creates a scaling taxonomy for VLM spatial capabilities:

• Capabilities that scale with model size: object recognition, semantic understanding of tasks, coarse spatial reasoning ("that arrow points roughly toward the coke can"), and relative spatial comparisons ("arrow 3 is closer to the target than arrow 7"). These benefit from larger models with better visual representations and more extensive pretraining.
• Capabilities that require new training data, not just more parameters: 3D depth reasoning, understanding of arbitrary visual encodings (color-to-semantic mappings, marker size conventions), fine-grained interaction physics (when to grasp, how occluded objects behave), and multi-step planning where actions have delayed consequences. These cannot be recovered by scaling alone because the necessary information is absent from the pretraining corpus.

The paper's interaction analysis (Figure 9) provides concrete evidence for the second category: PIVOT's performance degrades systematically when the robot approaches objects for grasping, precisely because grasping involves physical interactions (contact, occlusion, force) that are poorly represented in static internet images. The performance recovers after the grasp when the robot moves toward a visible placement target, suggesting that the degradation is not a general control problem but specifically an interaction-perception problem.

This taxonomy has important implications for the VLM research agenda. It suggests that bridging the gap between VLMs and embodied tasks requires not just scaling existing architectures but deliberately introducing training data modalities that convey the physical and spatial concepts missing from internet images — depth maps, egocentric video with action labels, interaction trajectories, or multi-view data. PIVOT's value as a diagnostic tool is that it localizes precisely which capabilities are missing, providing a roadmap for what training data would be most impactful to collect or synthesize.

Finally, the scaling results in Figure 8 are notable for what they do not show: the paper does not claim that PIVOT with the largest Gemini model approaches the performance of fine-tuned robot policies. The scaling curves are upward-sloping but far from saturation, and the absolute performance levels even for the largest model remain substantially below what RT-2 [4] achieves with in-domain fine-tuning. This honesty about the gap between zero-shot and fine-tuned performance is itself a contribution — it calibrates expectations for what current VLMs can do zero-shot and quantifies the performance premium that robot-specific training data still commands.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on four distinct domains, each with its own data: (1) Real-world mobile manipulator navigation: four goal-directed tasks designed by the authors, evaluated on a physical robot with success-rate metrics (Table 1). (2) Real-world mobile manipulator manipulation: three tabletop tasks evaluated on a physical robot with reach, steps, and grasp metrics (Table 2). (3) Real-world Franka arm manipulation: seven pick-and-place tasks evaluated on a physical Franka robot with XY/YZ reaching and steps metrics (Table 5, Appendix F). (4) RAVENS simulation [63]: five instances of "pick the {fruit} and place it in the {color} bowl" evaluated in simulation with L2 distance to ground-truth pick/place locations (Figure 15, Appendix E). (5) RefCOCO [61]: a random subset of 1000 examples from the testA split for keypoint localization, evaluated with normalized distance and bounding-box containment accuracy (Figure 5, Section 4.3). (6) Offline navigation dataset: 60 examples from prior robot navigation logs with human-labeled ground-truth targets, divided into in-view, semantic, and out-of-view categories (Appendix B). (7) Offline manipulation dataset: 10 episodes of pick demonstrations and 30 episodes of move-near demonstrations from the RT-X mobile manipulator dataset [38], used for offline metric computation (Appendix D, Section 4.4).

• Base model(s). The primary model is GPT-4V (GPT-4 with vision) [37], accessed through its standard API without any modification, fine-tuning, or gradient access. For scaling experiments (Section 4.5, Figure 8), the paper additionally evaluates four sizes of the Gemini model family[17], labeled "a" through "d" with progressively more parameters, again without any fine-tuning. The paper states that GPT-4V is used "unless otherwise noted" (Section 3.4) and positions these models as "representative of the capabilities of many contemporary LLMs" (Section 4, though this quote actually comes from the PaLM context in the example — the PIVOT paper does not use this exact phrasing but implies similar reasoning). The choice of GPT-4V is motivated by its state-of-the-art visual reasoning capabilities and public API availability, which makes the zero-shot claim testable by external researchers.

• Metrics. The paper uses domain-specific metrics that capture both success and precision:

  • Navigation (real robot, Table 1): Binary success rate — whether the robot reaches the target destination specified by the language instruction. Evaluated across 4 trials per task configuration.
  • Manipulation (real robot, Table 2): Three metrics: (i) Reach — binary success rate for whether the robot end-effector makes contact with or arrives at the relevant object; (ii) Steps — average number of action steps before successful termination (efficiency metric, lower is better); (iii) Grasp — binary success rate for whether the robot successfully grasps the target object (only reported when the task requires grasping).
  • Franka manipulation (real robot, Table 5): XY and YZ — binary success rates for reaching the correct object proximity in the camera's horizontal and vertical planes respectively; Steps — average action steps for successful trials.
  • Offline manipulation (Figures 7, 8, 13, 14): Cosine similarity between the action vector selected by PIVOT and the ground-truth expert action from the RT-X dataset [38], computed in the camera frame. The paper explains this choice: "For example, a 0.5 cosine similarity in 2D space corresponds to arccos(0.5) = 60°. As our actions can be executed a maximum delta along the chosen Cartesian action direction, we have found this metric more informative than others, e.g., mean squared error." (Section 4.4). Cosine similarity ranges from 0 (orthogonal/wrong direction) to 1 (perfectly aligned with demonstrator).
  • Offline navigation (Figures 6, 8, Table 3): Normalized L2 distance between the selected action and the point of interest in the camera frame, normalized by the image width. Lower is better.
  • RefCOCO (Figure 5): (i) Normalized distance between the center of the selected circle and the center of the ground-truth bounding box (lower is better); (ii) Accuracy — binary measure of whether the selected circle lies within the ground-truth bounding box.
  • RAVENS (Figure 15): L2 distance (in pixels) between predicted pick/place locations and ground-truth locations, averaged across pick and place, tracked iteration by iteration.

• Baselines. The paper compares against several internal variants of its own method rather than external published baselines, consistent with its framing as a capability probe rather than a benchmark effort:

  • No Iteration, No Parallel: A single PIVOT instance with 1 iteration of sampling and VLM selection — essentially, "show the VLM 10 annotated arrows once and pick the best." This tests whether iterative refinement provides gains over a single VLM query.
  • No Iteration, N Parallel: N parallel independent single-iteration PIVOT instances, aggregated. Tests whether parallel exploration alone compensates for lack of iterative refinement.
  • N Iterations, No Parallel: A single PIVOT instance run for N iterations without parallel redundancy. Tests the isolated contribution of iterative optimization.
  • N Iterations, N Parallel: The full PIVOT configuration with both iterative refinement and parallel redundancy. Tests the combined effect.
  • Text-only baseline (Section 4.4, Appendix B): For navigation, the VLM is given the same image and asked to "imagine that the image is split into 3 rows and 3 columns of equal-sized regions and output the name of one of those regions (e.g., 'top left', 'bottom middle')." For manipulation, the VLM selects from language actions "move right," "left," "up," and "down." This tests whether visual prompting provides benefits over purely textual spatial reasoning. Results in Figure 6, Figure 7, and Table 4.
  • The paper explicitly does not compare against learned approaches (RT-2 [4], RT-X [38], etc.) because "our focus is on zero-shot understanding" and "many such approaches would perform well in distribution on these tasks, but would have limited generalization on out of distribution tasks" (Section 4.4). This is a deliberate scope limitation, not an oversight.

• Generation budget / compute accounting. The paper does not use a standardized FLOPs or token count metric. Instead, the relevant budget is measured in number of VLM API calls per action step, since these dominate wall-clock time and monetary cost. The cost of one PIVOT instance with N iterations is N VLM queries (one per iteration). The cost with E parallel calls and N iterations is E × N queries, plus optionally one additional query for VLM-based aggregation. The paper also treats number of action steps to completion as an efficiency metric (Tables 2, 5) — fewer steps means fewer total VLM queries over the entire task even if per-step cost is high. The per-iteration number of candidate actions M = 10 is held fixed based on the ablation in Figure 7, so the generation budget per query is consistent across experiments. This is a practical rather than theoretical cost model — it does not account for GPU utilization, batching opportunities, or API pricing tiers, but it captures the dominant scaling factor for VLM-based methods.

• Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance tests for the real-robot experiments. For real-robot evaluations (Tables 1, 2, 5), each task is evaluated with a small number of trials (typically 2–4 per condition), and the paper reports raw success rates and averages, not standard errors. For offline evaluations (Figures 6–7, 13–14, Appendix B), results are reported with standard deviations across 3 runs over the entire dataset, as indicated in Table 3 and the Figure 13 caption. The RefCOCO evaluation uses a "random subset of 1000 examples from the RefCOCO testA split" (Section 4.3) but does not report confidence intervals on the accuracy metrics. The RAVENS evaluation (Figure 15) averages over "five random instances" per task but does not report variance bars. This relatively lightweight statistical reporting is consistent with the paper's framing as an exploratory capability probe rather than a rigorous benchmark submission, but it does mean that some of the reported differences between configurations (particularly in the real-robot tables with 2–4 trials per cell) may not be statistically robust.

---

Main Quantitative Results

Real-World Navigation (Table 1)

The paper evaluates PIVOT on four goal-directed navigation tasks using the mobile manipulator platform (Figure 3a) with 2D action space:

• Go to orange table with tissue box: 25% (no iteration, no parallel) → 50% (3 iterations, no parallel) → 75% (no iteration, 3 parallel) → 75% (3 iterations, 3 parallel)
• Go to wooden bench without hitting obstacle: 25% → 50% → 75% → 50%
• Go to the darker room: 25% → 50% → 75% → 100%
• Help me find a place to sit and write: 75% → 50% → 100% → 75%

The headline finding is that the "3 iterations, 3 parallel" configuration achieves the highest or tied-for-highest success on 3 of 4 tasks, with the best result being 100% on "Go to the darker room." However, the pattern is not fully consistent: on "Go to wooden bench," the best configuration is "No Iteration, 3 Parallel" at 75%, and "3 Iterations, 3 Parallel" drops to 50%. On "Help me find a place to sit and write," the "No Iteration, 3 Parallel" at 100% outperforms "3 Iterations, 3 Parallel" at 75%. This inconsistency — where adding iterations sometimes hurts performance — may reflect the VLM over-committing to an incorrect early selection when the distribution tightens, exactly the failure mode that parallel calls are designed to mitigate. However, with only 4 trials per cell, the differences (e.g., 75% vs. 50%) could be driven by a single trial's outcome.

The text-only baseline (Table 4) for navigation achieves substantially worse performance: normalized L2 error of 0.26 ± 0.15 for in-view tasks (vs. 0.21 ± 0.002 for image-based prompting), 0.35 ± 0.14 for semantic tasks (vs. 0.23 ± 0.012), and 0.46 ± 0.31 for out-of-view tasks (vs. 0.44 ± 0.04). The text baseline's high standard deviations reflect the inconsistency of language-based spatial reasoning — the VLM sometimes guesses correctly and sometimes catastrophically misses.

Real-World Manipulation — Mobile Manipulator (Table 2)

Three tabletop tasks are evaluated on the mobile manipulator with 4D action space (3D Cartesian delta + gripper):

"Pick coke can": The full configuration (3 iterations, 3 parallel) achieves 100% reach, 3.0 steps average, and 67% grasp success. This compares to:

• No iterations, no parallel: 50% reach, 4.5 steps, 0% grasp — the robot never successfully grasps the can
• 3 iterations, no parallel: 67% reach, 3.0 steps, 33% grasp
• The jump from 0% to 67% grasp rate when adding parallel calls (from "3 iterations, no parallel" to "3 iterations, 3 parallel") is the largest single improvement in the table, suggesting that grasp-critical approach angles require robustness to VLM errors that single-instance PIVOT cannot provide.

"Bring the orange to the X": 80% reach with 3 iterations and no parallel (best reach), 67% reach with 3 iterations and 3 parallel. The parallel version achieves slightly worse reach but the same step efficiency (3.5 steps). Since this task does not require grasping (it involves pushing/moving an object that is already on the table), no grasp metric is reported.

"Sort the apple": 100% reach with 3 iterations and no parallel (best), 75% reach with 3 iterations and 3 parallel. The efficiency improves from 3.25 to 3.0 steps with parallel calls, suggesting that when the parallel-aggregated action is correct, it converges faster, but the parallel instances sometimes diverge to different regions and the aggregation produces a compromise that misses.

A key pattern across Table 2: the "3 iterations, no parallel" configuration achieves the best reach rate on two of three tasks (80% and 100%), while "3 iterations, 3 parallel" achieves the best grasp rate (67%) and generally lower step counts. This suggests a precision-vs-robustness tradeoff: single-instance PIVOT converges more aggressively to a specific action (good reach, fewer steps), while parallel PIVOT explores more broadly and is less likely to miss the correct approach entirely (better grasp), but may average across competing modes and produce less decisive actions.

Real-World Manipulation — Franka Arm (Table 5, Appendix F)

Seven pick-and-place tasks are evaluated on the Franka arm with wrist-mounted camera (Figure 3c). Results are reported per-task and averaged across all seven tasks (bottom row):

The combined configuration achieves the highest XY and YZ reach rates, but the improvement is much larger in YZ (38% → 59%) than XY (25% → 34%). YZ reach captures the vertical approach dimension (the camera's z-axis is up/down), which is precisely the dimension that the paper identifies as the hardest for VLMs due to depth reasoning limitations (Section 4.6). The large YZ improvement with parallel calls suggests that vertical errors are stochastic rather than systematic — the VLM can sometimes get depth right, and parallel calls surface the runs where it does. The XY improvement is more modest because horizontal direction is easier to infer from 2D arrows and benefits less from multiple attempts.

Per-task variation is extreme: "Place peppershaker on the pink plate" achieves 100% XY and YZ reach even with no iterations and no parallel (8.0 steps) — suggesting this is an easy task with clear visual affordances — while "Grasp the pink cup" and "Grasp the blue cup" achieve 0% XY reach across all configurations. These grasp-only tasks (no placement target) may require the VLM to identify the correct approach angle for a successful grasp, which is precisely where the interaction limitations (Figure 9) are most acute.

RefCOCO Visual Grounding (Figure 5, Section 4.3)

On a random 1000-example subset of RefCOCO testA, PIVOT with GPT-4V achieves:

• Iteration 1: approximately 0.055 normalized distance (the leftmost bar in Figure 5 left) and approximately 48% accuracy (the leftmost bar in Figure 5 right)
• Iteration 2: marginal improvement to approximately 0.050 normalized distance and approximately 49% accuracy
• Iteration 3: further marginal improvement to approximately 0.048 normalized distance and approximately 49% accuracy

The key finding is that even the first iteration achieves strong performance, with only modest gains from subsequent iterations. This contrasts with the robot control results, where iterations provide substantial gains (Tables 1–2). The difference likely reflects the nature of the task: localization is a single-shot spatial reference problem — "find the object" — where the VLM can identify the correct region from a single broad sampling, whereas robot control requires precise directional actions where coarse initial samples are often insufficient.

Offline Navigation Evaluation (Figure 6, Table 3)

On the offline navigation dataset of 60 examples, PIVOT achieves:

• 1 iteration, 0 parallel: L2 distance 0.21 ± 0.002 (in-view), 0.23 ± 0.012 (semantic), 0.44 ± 0.04 (out-of-view)
• 3 iterations, 3 parallel: L2 distance 0.17 ± 0.009 (in-view), 0.19 ± 0.01 (semantic), 0.39 ± 0.05 (out-of-view)

Iterations and parallel calls both improve performance, but the relative improvement decreases with task difficulty: in-view tasks see a ~19% reduction in L2 error from baseline to best configuration, while out-of-view tasks see only ~11% reduction. The out-of-view category (where "the object of interest is not visible from the current view with arrow annotations, but can be seen in past images from different locations") is fundamentally harder because the VLM must reason about navigation strategy ("the object is behind me, I should turn around") rather than directly perceiving the target. The paper does not report whether the VLM has access to past images or must infer the target's location from the language instruction alone; if the latter, this is a pure semantic reasoning task where visual prompting provides limited leverage.

Figure 6 (left) shows the ablation over iterations and parallel calls: the curve for 3 iterations and 3 parallel calls is consistently below (better than) all other configurations across the budget range, with the largest gap appearing at higher iteration counts. Figure 6 (right) shows that the text-only baseline (L2 ~0.35, estimated from Figure 6) is substantially worse than PIVOT with 1 iteration (L2 ~0.21), confirming that visual prompting provides a large absolute gain over language-based spatial reasoning.

Offline Manipulation Evaluation (Figure 7)

On the offline manipulation dataset (10 pick episodes from RT-X), PIVOT achieves:

• 1 iteration, 0 parallel: cosine similarity approximately 0.42
• 3 iterations, 3 parallel: cosine similarity approximately 0.58

This represents a ~38% relative improvement from the baseline to the full configuration. The curves in Figure 7 show that both iterations and parallel calls contribute roughly additively: going from 1 to 3 iterations (at 0 parallel) improves cosine similarity by approximately 0.08, while going from 0 to 3 parallel calls (at 1 iteration) also improves by approximately 0.08, and doing both together yields approximately 0.16 total improvement.

Figure 7 also shows the sample-count ablation: M = 10 achieves the best performance across iterations, with M = 5 and M = 15 both underperforming. The paper explains this as a coverage-vs-clutter tradeoff: "more samples leads to better initial answers, but worse optimization. Intuitively, a large number of samples supports good coverage for the initial answer, but with too many samples the region of the image around the correct answer gets crowded and causes significant issues with occlusions." (Section 4.4).

The text-only baseline for manipulation (Figure 7, estimated from the text description) achieves "much lower" cosine similarity than the image-based approach, consistent with the navigation findings.

Scaling Results (Figure 8, Section 4.5)

Four sizes of the Gemini model family (labeled a through d, increasing in parameters) are evaluated on both manipulation and navigation offline metrics using first-iteration visual prompting (no iterative refinement, to isolate model capability):

• Manipulation (pick objects): cosine similarity increases from approximately 0.28 (model a) to 0.35 (model b) to 0.42 (model c) to 0.48 (model d) — monotonic improvement with each model size increase
• Manipulation (move near): cosine similarity increases from approximately 0.22 (model a) to 0.27 (model b) to 0.38 (model c) to 0.42 (model d) — also monotonic, but the gap between c and d is smaller than between b and c
• Navigation: L2 distance decreases (improves) from approximately 0.38 (model a) to 0.33 (model b) to 0.27 (model c) to 0.24 (model d) — again monotonic

The key finding is that performance improves monotonically across all model sizes on all three metrics, with no observed saturation or diminishing returns within the tested range. The paper frames this as evidence that PIVOT "scales well with improved VLMs" (Section 4.5). However, even the largest model (d) achieves absolute performance (cosine similarity ~0.48 for manipulation, L2 ~0.24 for navigation) that remains substantially below what fine-tuned approaches achieve, and the scaling curves show no sign of approaching the fine-tuned performance ceiling within the tested model size range. This suggests that scaling alone, at least within the range tested, may not close the zero-shot gap.

RAVENS Simulation (Figure 15, Appendix E)

On five instances of "pick {fruit} and place in {color} bowl" with three fruits and three bowls visible:

• Iteration 0 (initial sample): L2 error ranges from roughly 50 to 200+ pixels depending on the instance
• Iteration 1: substantial improvement across most instances, with error dropping to 30–100 pixels
• Iterations 3–4: plateau or slight improvement, with final errors of 20–80 pixels

The paper notes that "in most settings the chosen pick and place locations are close to the desired objects, yet the VLM lacks the often ability to precisely choose points that allow it to execute the task successfully in one action" (Appendix E). The L2 errors at convergence (20–80 pixels on what appears to be a ~500×500 image) mean the selected points are in the general vicinity of the correct objects but may not be precisely centered for successful grasping. This is consistent with the coarse-to-fine convergence pattern of CEM: iterations bring you to the right region, but the final precision is limited by the visual resolution at which the VLM can discriminate candidate markers.

---

Ablation Studies and Robustness Checks

Number of iterations and parallel calls (Tables 1–3, Figures 6–7): Adding iterations improves performance across both offline metrics (Figures 6 left, 7) and most real-robot tasks (Tables 1–2), but returns diminish after 2–3 iterations. Adding parallel calls provides improvements that are roughly additive with iterations. The combination (3 iterations, 3 parallel) consistently achieves the best or near-best performance. Table 3 breaks this down by task category for offline navigation: in-view tasks improve from 0.21 to 0.17 normalized L2 (3 iterations, 3 parallel), semantic tasks from 0.23 to 0.19, and out-of-view tasks from 0.44 to 0.39. The out-of-view improvement is notably smaller both absolutely and relatively, suggesting that iterative refinement helps most when the VLM can perceive the target, rather than when it must reason about an occluded object's likely location.

Number of candidate samples M (Figure 7, Section 4.4): Tested at M = 5, 10, 15. M = 10 achieves the best performance. M = 15 degrades performance despite providing better action-space coverage, which the paper attributes to visual clutter and occlusion — "with too many samples the region of the image around the correct answer gets crowded." M = 5 provides insufficient initial coverage. This is a non-obvious finding: one might expect more samples to monotonically improve VLM selection quality by providing finer-grained options, but the visual clutter penalty dominates beyond a threshold. This threshold is likely VLM-specific (depends on image resolution, attention span, and robustness to occlusions) and scene-specific (simpler scenes can accommodate more markers before confusion sets in).

Text-only vs. visual prompting baseline (Figure 6 right, Figure 7, Table 4): The visual prompting approach outperforms text-based spatial reasoning by a large margin. For offline navigation (Table 4), text-based spatial reasoning (asking the VLM to output a 3×3 grid region) achieves 0.26 ± 0.15 L2 error on in-view tasks vs. 0.21 ± 0.002 for image-based prompting — the higher mean and higher standard deviation indicate that language-based reasoning is both less accurate and less reliable. For offline manipulation, the text baseline ("move right/left/up/down") is described as achieving "much lower" cosine similarity than PIVOT, though exact numbers are not provided in the text. This ablation establishes that visual annotations are essential for the spatial precision PIVOT achieves — the VLM cannot translate language into spatial coordinates with comparable accuracy.

Text prompting strategy (Figure 13, Appendix D): Compared across zero-shot vs. few-shot and chain-of-thought (CoT) vs. direct prompting on the offline manipulation dataset:

• Zero-shot CoT achieves the highest cosine similarity (~0.55 at 3 iterations)
• Few-shot direct is close behind (~0.53) and is "more token efficient" since it avoids the verbose reasoning trace
• Zero-shot direct and few-shot CoT achieve lower performance (~0.48–0.50)

The finding that few-shot prompting does not clearly outperform zero-shot CoT is noteworthy: providing the VLM with correct examples of arrow selection does not reliably improve its selections on new scenes. This may reflect distribution shift — the few-shot examples show different objects, scenes, and tasks, and the VLM may overfit to superficial features of the examples rather than learning the general selection principle. Chain of thought helps because it forces the VLM to ground its selection in visual evidence ("arrow 3 points toward the coke can") rather than pattern-matching to the format.

Prompt ordering (Figure 14, Appendix D): Six orderings of preamble, image, and task are tested on the offline manipulation dataset:

• Preamble → Image → Task achieves the highest cosine similarity (~0.48)
• Other orderings achieve ~0.43–0.47
• "By a small margin," meaning the effect is present but not large enough to be a primary driver of performance

The paper hypothesizes that preamble-first order "most closely mirrors the training mixture" of internet-scale VLM pretraining data, where a system prompt or description often precedes the image. The marginal nature of the effect suggests that VLM prompt ordering for visual tasks is relatively robust, though a specific ordering provides a slight edge.

Visual annotation style robustness (Appendix G, Tables 6–7): On a synthetic blank-background arrow classification task (Table 6), GPT-4V achieves 88–100% accuracy across arrow colors (red, orange, yellow, green, blue, purple), thicknesses (2, 4, 6 pixels), arrowhead sizes (ratios 0.1, 0.3, 0.5), and directions (up+right, down+right, up+left, down+left). However, performance is direction-dependent: "down+right" and "up+left" show some dips (75%, 50% for certain colors) compared to 100% for "up+right" across most colors, suggesting a systematic bias in arrow direction classification on blank backgrounds — certain diagonal directions are harder for the VLM to classify, possibly because they are less common in VLM training data.

On the realistic object-referential arrow dataset (Table 7), where the task is "select the arrow that points at [specific object]" among distractors, performance drops dramatically:

• Accuracy ranges from 17% to 83% across all color/size/direction combinations
• Color effects are smaller than object-difficulty effects: Easy objects achieve 44–100% accuracy; Medium objects achieve 22–100%; Hard objects achieve 0–56%; Very Hard objects achieve 0–44%
• Yellow and green arrows perform slightly better on average than red or blue, but the differences are not large enough to be definitive (ranges heavily overlap across colors)

This ablation is significant because it demonstrates that VLMs can reliably classify arrows as abstract symbols (Table 6) but struggle when arrows must be associated with specific objects in cluttered realistic scenes (Table 7). This suggests that the bottleneck in PIVOT's visual annotation pipeline is not arrow comprehension but visual grounding of arrows to objects — a core vision-language capability that current VLMs have developed to varying degrees depending on object type.

Interaction trajectory analysis (Figure 9, Section 4.6): On "move near" trajectories where the robot must pick up an object and move it near another, PIVOT's cosine similarity with expert demonstrations follows a U-shaped pattern:

• Initially high (robot moves toward the first object, which is clearly visible)
• Decreases to a minimum during the approach-to-grasp phase (objects become occluded by the gripper or arm, fine-grained grasp positioning is required)
• Increases after the grasp as the robot moves toward the second object (the target is visible again)
• Decreases again as the robot approaches the second object for placement

This pattern is presented as evidence for the "interaction and fine-grained control" limitation discussed in Section 4.6. The paper argues that this degradation is due to "occlusions, resolution of the image, but perhaps more crucially, a lack of training data from similar interactions" (Section 4.6).

ReST$^{EM}$ revision training (not applicable — this paper does not use revision models): The paper does not train or use revision models; this ablation category from the example is not relevant to PIVOT.

Aggregation strategy for parallel calls (Section 3.3): Two strategies are described: (1) fitting a new action distribution from the E final actions (distribution-based aggregation), and (2) querying the VLM again to select the single best action from the E candidates (VLM re-selection). The paper states both are used but does not provide a direct ablation comparing them. From the experimental descriptions, the "3 parallel" configuration in Tables 1–2 uses VLM re-selection (it adds one additional VLM call), while the offline results (Figures 6–7) fit a distribution. Since the real-robot results with 3 parallel generally outperform those without, and the offline results also show gains from parallel calls, both strategies appear effective, but the paper does not quantify which is better.

---

Critical Assessment

The experiments demonstrate several claims convincingly, but the scope of evidence is narrower than the paper's framing suggests, and several important validation steps are missing.

Claim: "PIVOT enables zero-shot control of robotic systems without any robot training data." This claim is supported with qualifications for the specific robots and tasks tested. Tables 1, 2, and 5 unambiguously show non-zero task success on real robot platforms with GPT-4V and no fine-tuning — the robot does move, does sometimes reach targets, and does sometimes grasp objects. This is a genuine achievement given that the VLM was never trained on robot data. However, the claim of "control" should be interpreted carefully: these are open-loop action steps selected by the VLM and executed on the robot with closed-loop visual feedback at the next step (the robot takes a picture, PIVOT selects an action, the robot executes it, the cycle repeats). This is "control" in the sense of producing actions that move the robot, but the success rates (25–100% on navigation tasks with only 4 trials each; 0–67% grasp rates; 0–100% Franka reach rates with high variance) fall well short of what is typically required for reliable autonomous operation. The paper acknowledges this ("Although current performance is far from perfect"), but readers unfamiliar with robotics may not appreciate how large the gap is between these success rates and production requirements (typically >95% for repeatable tasks). The experiments demonstrate feasibility, not reliability.

Claim: "parallel calls improves performance and efficiency." Supported for most tasks, but with important inconsistencies. Table 1 shows that "No Iteration, 3 Parallel" achieves the best result on 3/4 navigation tasks, and "3 Iterations, 3 Parallel" achieves best on 2/4. Table 2 shows parallel calls improve grasp rate on "Pick coke can" from 33% to 67%, which is a large effect, but reach rate on "Sort the apple" drops from 100% (3 iterations, no parallel) to 75% (3 iterations, 3 parallel). Table 5 shows parallel calls improve average YZ reach from 31% to 59% but XY reach only from 28% to 34%. The offline results (Figures 6, 7) are more consistently positive. These mixed real-robot results suggest that parallel calls help on average but can sometimes hurt when the aggregation across parallel instances produces a compromise action that satisfies none of them (e.g., averaging two different approach directions yields a direction between them that misses the object entirely). The paper does not analyze specific failure cases of the parallel aggregation, which would strengthen the claim.

Claim: "increasing the number of PIVOT iterations also improves performance." Generally supported in the offline setting (Figures 6, 7, Table 3), but the real-robot evidence is mixed: Table 1 shows "3 Iterations, No Parallel" sometimes outperforming "No Iteration, No Parallel" and sometimes not; Table 2 shows iterations improving reach and grasp in "Pick coke can" but only reaching in "Bring the orange"; Table 5 shows effectively flat XY reach (25% → 28% → 34%) with more pronounced YZ improvement. The inconsistency between offline (where iterations reliably help) and online (where they help variably) may reflect that offline evaluation uses a fixed set of images with known ground truth, while online evaluation involves dynamically changing scenes where the VLM's iteration-1 error can propagate (the robot moves based on a wrong action, changing the scene such that subsequent iterations cannot recover). This distinction is important and not discussed in the paper.

Claim: "PIVOT scales well with improved VLMs" (Section 4.5). The scaling experiments in Figure 8 are among the paper's most important contributions, but they test only first-iteration performance (no iterative refinement), which measures pure VLM visual grounding capability but does not test whether the full PIVOT algorithm (with iterations and parallel calls) scales. It is possible that larger models not only select better initial actions but also make more consistent selections across iterations, improving the iterative refinement process itself. Conversely, they might be more prone to over-optimization (selecting visually salient but task-irrelevant arrows with high confidence, causing the distribution to collapse to a wrong region faster). Testing full PIVOT scaling (with iterations and parallel calls) across model sizes would be a more direct validation of the "PIVOT scales with VLM capability" claim. Additionally, only the Gemini family is tested; scaling behavior may be model-family-specific.

Weakness: extremely small real-robot evaluation. Tables 1 and 2 report success rates based on what appear to be 2–4 trials per condition per task (the paper does not explicitly state trial counts for real-robot experiments, but the fractional success rates like 25%, 50%, 67%, 75% strongly suggest denominators of 4 or 3). With 4 trials per configuration, a single trial's outcome shifts the reported rate by 25 percentage points. The differences between configurations (e.g., 50% vs. 75% in Table 1's "Go to wooden bench") are therefore not statistically distinguishable from noise. The paper should be read as reporting illustrative performance levels rather than precise comparisons between configurations — a limitation that the paper does not explicitly flag for the real-robot results (it does report standard deviations for offline results in Table 3).

Weakness: no comparison to simple learned baselines. While the paper's stated goal is zero-shot evaluation, comparing PIVOT to even a minimal fine-tuned baseline (e.g., a behavior cloning policy trained on 10 demonstrations of each task) would calibrate what "non-zero success" means. A reader unfamiliar with these specific tasks cannot tell whether 25% navigation success is impressive (because the task is genuinely hard and end-to-end policy learning also achieves ~25% with similar data) or weak (because a simple scripted policy using object detectors achieves 100%). The paper's claim that "many such approaches would perform well in distribution on these tasks, but would have limited generalization on out of distribution tasks" is plausible but untested — no OOD generalization experiment is included for the fine-tuned baselines either.

Weakness: missing difficulty analysis. The paper does not categorize tasks by difficulty or analyze how PIVOT's performance varies with task characteristics beyond the coarse "in-view / semantic / out-of-view" split in Table 3. Understanding whether PIVOT succeeds on "easy" tasks (clear visual affordances, unambiguous instructions) and fails on "hard" ones (cluttered scenes, subtle distinctions) would make the results more actionable. The interaction trajectory analysis (Figure 9) is a step in this direction but is limited to a single task type. A systematic difficulty breakdown — analogous to the difficulty bins in the example paper — is absent.

Weakness: cost analysis is entirely missing. The paper does not report wall-clock time per action step, monetary cost of API calls, or total compute required for the real-robot experiments. With 3 iterations × 3 parallel calls = 9 GPT-4V queries per action step, and tasks requiring 3–8 steps (Tables 2, 5), a single task attempt consumes 27–72 VLM queries. At typical GPT-4V API pricing, this is a non-trivial cost per task attempt. For PIVOT to be evaluated as a practical approach (rather than just a capability probe), these costs matter. The paper's claim that PIVOT is "promising" should be contextualized against its computational expense relative to learned policies that require one forward pass.

Missing experiment: adaptive iteration stopping. The paper fixes the number of iterations to N = 3, but a natural extension would be to stop iterating when the distribution has converged (e.g., when the variance falls below a threshold or when the VLM selects the same action as the previous iteration). This would save VLM queries on easy steps where the first or second iteration suffices while preserving precision on harder steps. No such experiment is reported.

Missing experiment: oracle VLM comparison. A useful diagnostic would be to replace the VLM with an oracle that always selects the action closest to the ground truth, and measure the remaining error. This would decompose PIVOT's total error into (a) error from the CEM optimization process itself (sampling noise, distribution fitting approximations) and (b) error from the VLM's imperfect selections. Without this decomposition, it is unclear whether improving the VLM or improving the optimization algorithm would yield larger gains. The scaling experiments (Figure 8) partially address this by showing that model quality matters, but they do not separate optimization error from selection error.

Missing experiment: sensitivity to initial distribution. The initial Gaussian's mean and variance are set manually per embodiment but never ablated. If the initial distribution is too narrow and misses the correct action entirely, no amount of iterations or parallel calls can recover. If it is too broad, early iterations waste samples on physically impossible actions. Quantitative characterization of this sensitivity would be valuable for practitioners deploying PIVOT on new embodiments.

Summary: The experiments demonstrate that PIVOT works — VLMs can indeed serve as the selection operator in a visual CEM loop to produce continuous spatial outputs without fine-tuning — and the ablation studies provide useful empirical guidance (10 samples, 3 iterations, 3 parallel calls, zero-shot CoT prompting). However, the quantitative results should be interpreted as existence proofs rather than precise performance measurements, due to the small number of real-robot trials, the absence of statistical rigor for the online results, and the lack of calibration against learned baselines. The paper's most robust contributions are the methodological framework and the scaling analysis (Figure 8), while the specific success rates in Tables 1–2 and 5 are best understood as illustrative of the current capability ceiling rather than precise benchmarks.

6. Limitations and Trade-offs

Depth Reasoning Is Fundamentally Unreliable

The assumption or constraint. PIVOT represents 3D actions (including depth/displacement along the camera's optical axis) using visual annotations on a 2D image — specifically, a red-to-blue color spectrum and varying circle sizes to encode forward/backward motion (Section 3.4, Appendix A). The text prompt explicitly describes this encoding to the VLM. The implicit assumption is that VLMs pretrained on internet images can reliably interpret these arbitrary visual depth cues when combined with textual descriptions of the encoding scheme.

The consequence. This assumption is falsified by the paper's own findings. Section 4.6 states directly:

"we have observed that none of the VLMs we tested are capable of reliably choosing actions based on depth."

This is not a marginal performance degradation — it is a categorical capability failure. The VLM cannot consistently distinguish between "move forward toward the object" and "move backward away from it" based on the rendered visual cues, even when told what those cues mean. For any manipulation task requiring precise 3D positioning (approaching an object from the correct distance, stopping before collision, making contact for grasping), this means PIVOT's actions in the depth dimension are effectively unreliable. The real-robot results bear this out: Table 5 (Franka arm) shows that YZ reach rate (which includes the vertical/depth approach dimension) improves from 38% to 59% with parallel calls, but XY reach (horizontal only) remains at only 34% even with full PIVOT — both are far below reliable deployment thresholds. Table 2 (mobile manipulator) shows that even the best configuration achieves only 67% grasp rate on "Pick coke can," and the grasp action requires correct depth positioning.

What evidence exists in the paper. The categorical statement in Section 4.6 is the primary evidence, but it is qualitative — no quantitative ablation isolates depth errors from other error sources. The Franka results (Table 5) provide indirect evidence: YZ reach (which includes depth) is worse than XY reach in the baseline (38% vs. 25%), though both are low. The manipulation offline evaluation (Figure 7, cosine similarity ~0.42–0.58) uses a metric that is direction-aware — an action selected in the wrong depth direction would produce a negative cosine similarity, so the positive averages suggest PIVOT gets depth direction roughly correct on average but imprecisely, which is consistent with unreliable rather than random depth selection. Appendix G (Tables 6–7) does not test depth-cue interpretation at all — it only tests arrow direction classification on 2D arrows, leaving the depth encoding failure unquantified. The paper does not report what fraction of errors in manipulation tasks are specifically attributable to incorrect depth choices vs. incorrect horizontal direction choices.

Mitigation status. The paper acknowledges this limitation explicitly (Section 4.6) but offers only speculative mitigations: "more complex visuals (e.g. with shading to give the illusion of depth) may address some of these challenges, but ultimately, the lack of 3D training data in the underlying VLM remains the bottleneck." It further suggests that "training on either robot specific data or with depth images may alleviate these challenges," but this would violate the zero-shot premise. No experiment tests alternative depth encoding schemes, and the failure is presented as a fundamental limitation of current VLM training distributions rather than a fixable engineering issue. For practitioners, this means PIVOT in its current form is not suitable for tasks where depth precision matters — which includes most manipulation beyond 2D tabletop pushing.

---

Difficulty Estimation Requires Prohibitively Expensive Computation (But for Actions, Not Prompts)

The assumption or constraint. PIVOT's iterative optimization loop requires evaluating candidate actions at each iteration by rendering them on images and querying the VLM, then fitting a new distribution and repeating. The paper assumes 3 iterations × 10 samples each = 30 rendered-and-evaluated candidates per action step, plus 3 parallel PIVOT instances for robustness = 90 VLM queries per action step (3 instances × 3 iterations × 10 candidates each = 90 images processed, though the VLM sees one annotated image with 10 candidates per query, meaning 9 total VLM API calls per action step in the "3 iterations, 3 parallel" configuration). This computational structure is inherent to the method — it is the price of extracting continuous actions from a discrete-output model via black-box optimization.

The consequence. PIVOT is two to three orders of magnitude more computationally expensive per action than a learned policy. A standard behavior cloning or RT-2-style policy produces an action in a single forward pass (one inference call, milliseconds on modern hardware). PIVOT requires 9 VLM API calls per action step (for 3 iterations × 3 parallel), each of which takes seconds and incurs monetary cost at commercial API rates. For tasks requiring multiple action steps — Tables 2 and 5 report 3–8 steps for successful trajectories — a single task attempt consumes 27–72 VLM queries. At the time of writing, GPT-4V API pricing is on the order of a few cents per image query, making a single task attempt cost dollars in API fees alone, independent of robot operation time. For real-time closed-loop control where actions must be issued at 10–50 Hz, this latency (seconds per action step for VLM inference plus rendering) is fundamentally incompatible with dynamic tasks requiring rapid responses — PIVOT is effectively restricted to quasi-static tasks where the scene does not change significantly between action steps.

What evidence exists in the paper. The paper does not report wall-clock time, monetary cost, or total compute for any experiment. The number of action steps is reported as an efficiency metric (Tables 2, 5), which provides a multiplier on per-step cost but not the per-step cost itself. The number of VLM queries per step can be inferred from the method description (Section 3.2–3.4): 1 query per iteration, 3 iterations, 3 parallel instances = 9 queries, possibly plus one for VLM re-selection aggregation. The paper does not ablate whether similar performance could be achieved with fewer iterations or fewer parallel calls at higher per-iteration sample counts, which would change the cost structure. The offline evaluations (Figures 6–7) show that 1 iteration with 3 parallel calls achieves performance close to 3 iterations with 3 parallel calls in some settings (Figure 6 left, the gap between "1 iter, 3 par" and "3 iter, 3 par" is modest), suggesting that per-step cost could potentially be reduced by a factor of 2–3× with acceptable performance loss, but this tradeoff is not explicitly analyzed. The scaling experiments (Figure 8) use only first-iteration performance, meaning they measure the cheapest possible PIVOT query, not the full iterative cost.

Mitigation status. The paper does not address computational cost as a limitation in Section 4.6 (Limitations) — it is mentioned nowhere as a concern. The paper frames PIVOT as a capability probe ("Our aim is not necessarily to develop the best possible robotic control") which partially excuses the absence of cost analysis — the goal is to demonstrate what is possible, not to optimize for deployment efficiency. However, for a practitioner considering whether to use PIVOT, the per-step cost is arguably the most important practical consideration after capability. The paper leaves entirely open the question of whether the zero-shot benefit justifies the 10–100× inference cost premium over fine-tuned alternatives, or whether the cost could be reduced by caching, batching, or using smaller/cheaper VLMs for earlier iterations. The method also provides no adaptive stopping criterion — it always runs the full N iterations regardless of whether the distribution has converged, wasting queries on steps where the first or second iteration already identified the correct action.

---

Real-Robot Evaluation Is Too Small to Support Quantitative Performance Claims

The assumption or constraint. The real-robot experiments (Tables 1, 2, 5) evaluate PIVOT on a small number of manually designed tasks with what appears to be 2–4 trials per configuration per task (inferred from the fractional success rates: 25%, 33%, 50%, 67%, 75%, 100% — these denominators strongly suggest 4 or 3 trials). For navigation, Table 1 reports success rates across 4 tasks; for mobile manipulation, Table 2 reports across 3 tasks; for Franka manipulation, Table 5 reports across 7 tasks with 2 trials each (stated explicitly: "Each task is evaluated for two trials," Appendix F). The paper reports no confidence intervals, no standard errors, and no statistical tests for any real-robot result.

The consequence. The headline real-robot numbers — "75–100% on four goal-directed tasks with 3 iterations and 3 parallel calls" — are based on sample sizes where a single trial flip changes the reported rate by 25–33 percentage points. The comparison between configurations is therefore not statistically meaningful. For example, Table 1 shows "Go to wooden bench": 25% (no iter, no par) → 50% (3 iter, no par) → 75% (no iter, 3 par) → 50% (3 iter, 3 par). The drop from 75% to 50% when adding iterations to the parallel configuration could indicate a real degradation (iterations causing over-commitment to VLM errors) or could be noise from 1–2 trial differences. The paper does not discuss this ambiguity. Similarly, Table 2 shows "Sort the apple" reach rate dropping from 100% (3 iter, no par) to 75% (3 iter, 3 par) — a surprising result that would be important to understand if real, but impossible to distinguish from sampling noise with 4 trials per cell. The offline evaluations (Table 3, Figures 6–7) report standard deviations and use 60 examples for navigation and 10–30 episodes for manipulation, providing more statistical reliability — but these measure cosine similarity and L2 distance against demonstration data, not task success rates in the real world. The relationship between offline metric improvement and real-world task success is not calibrated.

What evidence exists in the paper. The trial counts are not stated in the main paper for Tables 1–2 — they must be inferred from the fractional success rates. For Table 5 (Franka), Appendix F states "two trials" per task. For Tables 1–2, the denominators (4 or 3) are the only clue. The results in Table 1 show 4 discrete success levels (25%, 50%, 75%, 100%), strongly suggesting 4 trials. Table 2 shows rates of 20%, 33%, 50%, 67%, 75%, 80%, 100%, suggesting a mix of denominators (likely 4 for most entries; 3 for the 33% and 67% entries). The paper does not explain why some tasks had more trials than others (if they did). The offline results do report standard deviations (Table 3, Figure 13 caption), but these are for a different evaluation protocol (fixed image dataset, no real-robot execution) and different metrics. There is no bridging experiment showing that offline cosine similarity improvements predict online success rate improvements. A reader cannot determine whether the configuration rankings in Tables 1–2 would replicate in a larger study.

Mitigation status. The paper does not acknowledge the small sample size as a limitation. The use of 2–4 trials per configuration is characteristic of real-robot evaluation in robotics research — running more trials is expensive and time-consuming — but the paper does not calibrate reader expectations about the statistical reliability of the reported differences between configurations. The paper's framing as a capability probe partially mitigates this concern (the point is to show non-zero success, not to precisely rank configurations), but the abstract and Section 4.2 do make comparative claims ("parallel calls improves performance," "increasing the number of PIVOT iterations also improves performance") that depend on configuration rankings. A simple fix — reporting that differences below a certain threshold are not statistically distinguishable given the sample size, or pooling results across tasks within each configuration to increase effective sample size — would strengthen these claims without requiring more robot time.

---

Performance Degrades Systematically During Interaction-Rich Phases

The assumption or constraint. PIVOT assumes that the VLM can evaluate candidate actions from a single static image. This same image is both the observation of the current scene and the canvas on which action proposals are rendered. The VLM must reason about the effects of actions from visual features alone — it sees arrows overlaid on the current frame, but it does not see what will happen after the action executes, cannot predict physical interactions (contact, force, occlusion), and has no access to temporal dynamics from previous steps (unless explicitly included in a multi-turn prompt, which the paper does not do).

The consequence. PIVOT's action quality degrades specifically during phases requiring fine-grained physical understanding. Figure 9 provides direct evidence: on "move near" trajectories, cosine similarity with expert demonstrations starts high when the robot is approaching a clearly visible object from a distance, drops to a minimum during the approach-to-grasp phase (when the gripper is close to the object and precise positioning matters), recovers after the grasp when moving toward the second object, and drops again when approaching the placement target. The paper attributes this to "occlusions, resolution of the image, but perhaps more crucially, a lack of training data from similar interactions" (Section 4.6). This means PIVOT is least reliable at precisely the moments that determine task success — the grasp itself, the placement, any contact-rich maneuver. The real-robot results corroborate this: Table 2 shows that even with full PIVOT, "Pick coke can" achieves only 67% grasp rate (vs. 100% reach rate), meaning the robot arrives at the can but fails to grasp it one-third of the time. Table 5 shows that Franka grasping tasks (grasp pink cup, grasp blue cup) achieve 0% XY reach even with full PIVOT — the VLM cannot identify correct approach angles for these grasp-only tasks.

This limitation is not specific to grasping — it applies to any task phase where the visual image alone is insufficient to determine the correct action, including pushing objects with unknown friction, navigating tight spaces where depth from monocular cues is unreliable, or manipulating articulated objects (doors, drawers) where kinematic constraints are invisible in a single frame. The paper's evaluation tasks are deliberately chosen to minimize these challenges (tabletop manipulation with rigid objects, open-floor navigation), so the reported success rates represent an upper bound on what PIVOT could achieve on more interaction-heavy tasks.

What evidence exists in the paper. Figure 9 is the primary evidence, showing the U-shaped performance pattern across 30 "move near" trajectories. The figure plots cosine similarity aggregated across trajectory phases — it shows a clear dip during grasp-approach and placement-approach, but the absolute values and variance are not reported numerically. Section 4.6 provides qualitative description of occlusion-related failures ("objects of interest can become no longer visible if the cameras are too close") and notes that "errors are a result of both occlusions, resolution of the image, but perhaps more crucially, a lack of training data from similar interactions." The paper does not quantify what fraction of total errors are attributable to interaction-phase failures vs. other causes, nor does it attempt to mitigate this by, for example, providing multiple camera views, including previous frames as context, or using higher-resolution images during interaction phases.

Mitigation status. The paper acknowledges this limitation in Section 4.6 ("Interaction and fine-grained control") and suggests that "training on embodied or video data may be a remedy." This is a forward-looking suggestion — it points toward what future VLMs would need but offers no mitigation within the current PIVOT framework. A practitioner deploying PIVOT today has no mechanism to improve interaction-phase performance beyond hoping that the underlying VLM happens to handle the specific interaction in the specific scene. The paper does not explore whether adding a second camera angle, providing a short video history, or incorporating simple heuristics (e.g., "when gripper is within X cm of object, close gripper") could partially address this limitation without requiring VLM retraining. The finding that PIVOT fails at interaction phases is important for scoping where the method can be applied, and the paper's transparency about it is a strength, but the absence of any attempted mitigation within the zero-shot paradigm leaves a clear capability gap unaddressed.

---

The Method Has Not Been Tested on a Single Closed-Loop Task Requiring Temporal Reasoning

The assumption or constraint. PIVOT selects actions based on the current image and task instruction only. Each action step is independent — the VLM receives the current annotated frame and the task description, but no history of previous actions, previous observations, or previous VLM selections (except implicitly through the robot's changed position in the new image). The paper's tasks are designed so that greedy, myopic action selection is sufficient: "go to the orange table" can be solved by always moving toward the orange table; "pick up the coke can" can be solved by always moving toward the coke can. The paper acknowledges this limitation in Section 4.6:

"the underlying VLM often displays greedy, myopic behaviors for multi-step decision-making tasks. For instance, given the task 'move the apple to the banana', the VLM may recommend immediately approaching the banana rather than the apple first."

This is presented as an observation about VLM behavior, but it is also a property of the evaluation design — none of the tested tasks require non-greedy multi-step planning. "Move the apple to the banana" was not actually tested; it is offered as a hypothetical example of a failure mode the current approach would exhibit if tested on such a task. Every evaluated task has a single-step-optimal action at each timestep: navigation tasks involve moving toward a single visible target; manipulation tasks involve approaching a single object (pick tasks) or approaching then placing (bring-to tasks, but the sequence "approach object A, then approach location B" can be solved greedily if object A is attended to first, which the VLM can determine from the instruction). No task requires the robot to temporarily move away from the goal to navigate around an obstacle, to first clear a distractor before accessing the target, or to perform a sequence of actions whose individual steps are not obviously goal-directed.

The consequence. The paper provides no evidence about PIVOT's ability to handle tasks requiring memory, planning, or temporal credit assignment. Any task where the correct current action depends on previous actions (e.g., "I already picked up the apple, now I need to place it" vs. "I still need to pick up the apple") requires the VLM to maintain state across action steps. PIVOT as described has no explicit state-tracking mechanism — it relies on the image implicitly containing state information (the apple is in the gripper, so the image shows it there), but this breaks down when state is not visually apparent (did I already check behind the door? Is the drawer unlocked now?) or when the image is ambiguous. The paper's identified "greedy, myopic" failure mode further suggests that even when the image contains sufficient information, the VLM may not correctly sequence actions — it may skip prerequisite steps because the end goal is visually salient.

Even simpler temporal phenomena like momentum, velocity-dependent dynamics, or delayed effects of actions are completely outside PIVOT's scope — the method treats each action step as independent. This restricts applicability to quasi-static tasks where the world state changes only through the robot's discrete action steps and fully stabilizes between steps. Tasks involving dynamic object interactions (throwing, catching, swinging), fluid manipulation (pouring, stirring), or deformable objects (folding, stretching) are categorically inaccessible to PIVOT in its current form.

What evidence exists in the paper. The paper has zero experiments testing multi-step reasoning or temporal state tracking. The navigation tasks (Table 1) are all single-target reaching tasks. The manipulation tasks (Table 2) include one multi-phase task ("Pick coke can" — approach and grasp) but the phases are visually distinct (approaching the can vs. closing gripper), and the gripper action is handled via text prompt rather than visual reasoning. "Bring the orange to the X" and "Sort the apple" involve moving objects between locations, but the paper does not analyze whether the VLM correctly sequences these subtasks or whether failures occur due to sequencing errors. The RAVENS tasks (Figure 15) involve pick-and-place but treat pick and place as independently queried locations — the VLM is prompted separately for "which marker is closest to the {fruit}" and "which marker is closest to the {bowl}," bypassing the need to sequence actions. The Franka tasks (Table 5) include "Place saltshaker on the blue plate" and similar pick-and-place variants, but the paper does not break down errors by phase (approach, grasp, transport, place). The "greedy, myopic" behavior described in Section 4.6 is not experimentally demonstrated or quantified — it is a qualitative observation from informal testing, not a measured failure rate on a multi-step reasoning benchmark.

Mitigation status. The paper mentions this limitation in Section 4.6 ("Greedy behavior") and suggests that "more in-domain examples provided either via fine-tuning or via few-shot prompting with e.g., a history of actions as input context to the VLM to guide future generated actions" could help. This suggestion would require providing action history as additional context in the VLM prompt (multimodal few-shot examples showing action sequences), which the paper does not attempt. A more direct mitigation — simply including the previous N actions and observations as additional images or text in the VLM's input — is not tested. The paper's evaluation design, which deliberately avoids tasks requiring temporal reasoning, means that the reported performance numbers are conditional on a task distribution that excludes the failure mode the paper itself identifies. A practitioner considering PIVOT for a task that requires remembering what step of a plan the robot is on should treat the existing results as an upper bound and expect additional failures from sequencing and state-tracking errors that are not captured in the paper's evaluation.

---

Difficulty Estimation and Strategy Selection Are Absent — No Adaptive Allocation

The assumption or constraint. PIVOT uses a fixed strategy for every task and every action step: always 3 iterations, always 10 samples per iteration, always 3 parallel calls (when using the full configuration). The number of iterations, samples, and parallel calls is determined by offline ablations averaged across all tasks and then applied uniformly in online evaluation. There is no mechanism that estimates the difficulty of the current action step or adapts the computational budget accordingly. A step where the VLM can immediately identify the correct action (clear visual affordances, unambiguous instruction) receives the same 9 VLM queries as a step where the VLM is uncertain and generates inconsistent rankings across parallel instances.

The consequence. PIVOT wastes substantial computation on easy action steps while potentially under-computing on hard ones. Figure 3 (right) and Figure 7 (right) in the paper — using a hypothetical difficulty-breakdown structure the paper does not actually provide — would likely show that easy action steps (robot far from target, clear line of sight) achieve high cosine similarity with 1 iteration and 0 parallel calls, while hard steps (robot near target, partial occlusion, ambiguous instruction) require more iterations and parallel calls. The paper's offline results provide indirect evidence for this: Table 3 breaks navigation performance by task category (in-view, semantic, out-of-view) and shows that PIVOT's relative improvement over the baseline is largest on in-view tasks (~19% L2 reduction) and smallest on out-of-view tasks (~11% L2 reduction), suggesting that the fixed strategy is suboptimally allocated — out-of-view tasks might benefit from a different strategy (e.g., more exploration, fewer iterations) than in-view tasks. Figure 6 (left) shows that the gap between "3 iter, 3 par" and "1 iter, 3 par" narrows as the number of iterations increases, suggesting diminishing returns from iterations that could be detected and used as a stopping criterion.

More broadly, the absence of difficulty estimation means PIVOT cannot make the kinds of compute-optimal allocation decisions that the example paper (on test-time compute scaling) showed were critical. That paper demonstrated 4× efficiency gains by adaptively selecting search strategies based on estimated prompt difficulty. PIVOT's fixed strategy leaves similar efficiency improvements on the table — an easy navigation step that could be solved with 1 iteration and 1 parallel call (1 VLM query) instead consumes 9 queries, while a hard step that might benefit from 5 iterations and 5 parallel calls receives only the default 3 and 3.

What evidence exists in the paper. The paper does not conduct a difficulty-aware analysis. The offline evaluations (Figures 6–7) show average performance across all examples, not per-difficulty breakdowns. Table 3 provides the closest thing — a breakdown by task category — but the task categories were defined by the dataset creators (in-view, semantic, out-of-view for navigation), not by PIVOT's observed performance on those categories. There is no analysis of which action steps within a trajectory were easy vs. hard for PIVOT, no measurement of whether VLM confidence (e.g., agreement among parallel instances, consistency across iterations, the VLM's expressed certainty in its chain-of-thought) predicts action correctness, and no experiment testing whether an adaptive stopping criterion (e.g., stop iterating when the distribution variance falls below a threshold or when the VLM selects the same top action for two consecutive iterations) reduces cost without sacrificing performance. The paper also does not report the variance of performance across trials/configurations, which would indicate whether some steps are consistently hard (suggesting fixed strategy is appropriate) or whether difficulty varies widely (suggesting adaptive strategy would help).

Mitigation status. The paper does not identify the absence of adaptive allocation as a limitation. The fixed-strategy design is an implicit simplification — the paper's goal is to demonstrate that the iterative visual optimization approach works at all, not to optimize its efficiency. However, the practical consequence is that PIVOT in its current form is both computationally wasteful on easy steps and potentially underpowered on hard steps. A natural extension — using VLM confidence signals (agreement across parallel instances, variance of the action distribution, the VLM's own uncertainty expressions in chain-of-thought) to dynamically allocate computation — is not discussed. Given the per-step cost of VLM queries, adaptive allocation would be one of the highest-impact improvements to PIVOT's practicality, potentially reducing average per-step cost by 2–3× while maintaining or even improving success rates on the hardest steps. The paper's scaling results (Figure 8) and iteration ablations (Figures 6–7) provide the raw material for designing such an adaptive strategy, but the synthesis is left to future work.

7. Implications and Future Directions

How This Work Changes the Landscape

PIVOT represents a methodological reframing with diagnostic significance rather than a paradigm shift or a benchmark-saturating system. Its primary contribution is changing the question the field asks about VLMs and spatial tasks: instead of "how do we train VLMs to output actions?" (a supervised learning problem requiring robot data), PIVOT reframes the question as "how well can VLMs evaluate spatially-grounded proposals when those proposals are rendered visually?" — a capability assessment problem that can be studied without any fine-tuning. This is not an incremental improvement on existing VLM-to-action pipelines; it is a categorically different approach that opens an entirely new axis of investigation.

The magnitude of the shift is best understood diagnostically. Before PIVOT, the dominant narrative around VLMs for robotics was that they possess rich semantic knowledge (object identities, task structures, common-sense affordances) but lack the low-level spatial precision needed for control, and that bridging this gap requires in-domain robot data (RT-2) or code-generating primitives (Code-as-Policies). PIVOT demonstrates that the semantic knowledge and the spatial reasoning capability are less separable than previously assumed — when candidate actions are rendered as visual annotations, the same VLM that can describe a scene and answer questions about it can also make reasonably good relative spatial judgments ("arrow 3 points more toward the coke can than arrow 7"), and an outer optimization loop can convert these relative judgments into absolute action coordinates. This reframes the problem from "VLMs lack spatial precision" to "VLMs possess coarse spatial reasoning that can be refined through iterative visual feedback," which is a fundamentally different capability model with different scaling implications.

The paper reconciles several tensions in prior work. The finding that visual prompting dramatically outperforms text-based spatial reasoning (Table 4: L2 error 0.21 vs. 0.26 for in-view navigation; Figure 7: text baseline "much lower" cosine similarity for manipulation) explains why earlier language-only attempts to extract spatial outputs from LLMs/VLMs were disappointing — the visual modality carries spatial information that language fundamentally cannot encode at comparable resolution. The finding that iterative refinement helps on robot control tasks (Tables 1–2) but only marginally on RefCOCO localization (Figure 5) explains why Yang et al. [59] achieved strong single-shot visual grounding results without iteration — when the task is identifying a clearly visible object, one round of sampling suffices, but when the task requires precise directional control with continuous-valued consequences, the coarse initial proposals need refinement. This is not a contradiction between papers but a task-dependent difference in required precision that PIVOT's iterative framework makes explicit.

The paper also changes the attractiveness landscape for research investments:

• More attractive: Research on visual annotation design (arrow styles, depth encoding schemes, marker shapes, resilience to clutter) becomes directly impactful for zero-shot control performance, as Appendix G demonstrates that current VLMs are sensitive to these parameters. Work on improving VLM robustness to visual annotations — not just language prompts — is now a clearly scoped problem with measurable downstream consequences. Research on black-box optimization algorithms for stochastic, non-differentiable quality functions (CEM variants, Bayesian optimization, evolutionary strategies adapted to VLM noise characteristics) becomes relevant to robotics in a way it wasn't when the dominant paradigm was end-to-end policy learning.

• Less attractive: The paper implicitly argues against investing heavily in hand-crafted text prompting strategies for spatial tasks — the finding in Figure 13 that prompt engineering (zero-shot vs. few-shot, CoT vs. direct, prompt ordering in Figure 14) yields only marginal gains compared to the visual annotation design and the optimization algorithm itself suggests that prompt engineering for spatial reasoning faces diminishing returns. Pure language-based spatial reasoning (the text-only baseline) is shown to be sufficiently unreliable that further investment in that direction without a visual component seems unpromising. Most significantly, the paper demonstrates that zero-shot VLM-based control is possible but currently far from reliable, which may redirect some effort away from pushing zero-shot performance toward hybrid approaches that combine PIVOT-style visual reasoning with minimal fine-tuning or learned components — the paper does not advocate this directly, but its honest reporting of limitations (Section 4.6) makes the case implicitly.

The paper's most lasting contribution may be the scaling taxonomy it introduces for VLM spatial capabilities (Section 4.6, elaborated in the prior sections): capabilities that scale with model size (object recognition, coarse spatial comparisons, semantic task understanding) vs. capabilities that require new training data modalities (3D depth reasoning, interaction physics, multi-step planning). This taxonomy provides a principled framework for deciding where to invest in scaling vs. where to invest in data collection, and PIVOT serves as a reusable measurement instrument for tracking progress on both axes as VLMs evolve.

---

Follow-Up Research This Work Enables

1. Quantitative decomposition of PIVOT error into VLM selection error vs. CEM optimization error. The paper demonstrates that PIVOT works but does not measure why it fails when it fails. A critical follow-up would replace the VLM with an oracle selector (always choosing the action closest to the ground-truth expert action from the RT-X dataset or human label) and run the same CEM loop with the same hyperparameters (10 samples, 3 iterations, isotropic Gaussian fitting). The difference between oracle-PIVOT performance and VLM-PIVOT performance isolates VLM selection error; the remaining error in oracle-PIVOT isolates optimization error (insufficient samples, poor initial distribution, premature convergence, distribution fitting approximations). This decomposition would answer: should we invest in bigger VLMs (reduce selection error) or better optimization algorithms (reduce optimization error)? The scaling results in Figure 8 show that larger models improve first-iteration selection, but the decomposition would reveal whether the optimization error floor is low enough that scaling alone can eventually approach oracle performance, or whether algorithmic improvements are necessary regardless of VLM quality. A strong version of this experiment would run on the offline manipulation and navigation datasets (60 and 10–30 examples respectively) with statistical error bars, and would test multiple CEM configurations (sample counts, iteration counts, covariance structures) to identify the Pareto frontier of optimization error.

2. Adaptive budget allocation using VLM confidence signals. The paper's fixed strategy (always 3 iterations, always 3 parallel calls, always 10 samples) is almost certainly suboptimal given the observed variance in difficulty across action steps and task categories (Table 3 shows out-of-view tasks barely benefit from iterations; Figure 6 shows diminishing returns). A natural follow-up would develop and test an adaptive stopping criterion based on signals already available in PIVOT: (a) inter-parallel agreement — if 3 parallel PIVOT instances all select the same or similar actions, the distribution has likely converged and further iterations are unnecessary; (b) distribution variance — when the fitted Gaussian's variance falls below a threshold, stop iterating; (c) VLM-expressed uncertainty — use the chain-of-thought reasoning traces to detect hedging language ("arrow 3 might be acceptable but it's hard to tell") and allocate additional budget when the VLM is uncertain. The experiment would compare adaptive PIVOT to fixed PIVOT on the offline datasets, measuring both accuracy (cosine similarity, L2 distance) and cost (number of VLM queries). The hypothesis is that adaptive PIVOT matches or exceeds fixed PIVOT accuracy while reducing average per-step cost by 2–3×, with the largest savings on easy action steps. This directly addresses the paper's unacknowledged computational cost limitation and would make PIVOT substantially more practical. A strong version would also test whether difficulty can be predicted before querying the VLM — e.g., using a lightweight visual classifier trained on PIVOT's failure patterns — enabling proactive budget allocation rather than reactive stopping.

3. Systematic visual annotation design space exploration with human-calibrated baselines. Appendix G shows that VLMs are sensitive to arrow color, thickness, size, and object type, but explores only a handful of rendering parameters and does not compare alternative visual representations (dots, crosshairs, oriented rectangles, heatmaps, contour lines, animated overlays). A systematic follow-up would: (a) generate a large synthetic dataset varying rendering parameters factorially (arrow style × color × size × number of distractors × scene complexity), (b) establish human performance baselines on the same visual selection tasks to calibrate what is "hard for VLMs" vs. "hard for anyone," and (c) use the resulting error landscape to optimize visual annotation design for VLM perception. The paper's finding that VLMs achieve 88–100% on blank-background arrow classification (Table 6) but 17–83% on object-referential arrows (Table 7) suggests the bottleneck is visual grounding (associating an arrow with a specific object in a cluttered scene), not arrow comprehension. A well-designed follow-up would test whether representations that explicitly link the marker to the referent — e.g., arrows that originate from the object, or markers that are rendered with object-specific colors mentioned in the prompt — close this gap. The specific metric would be selection accuracy as a function of representation type, with the human baseline serving as the ceiling. This work could produce a "visual annotation style guide for VLMs" analogous to the accessibility guidelines that exist for human interface design.

4. Closed-loop multi-step reasoning evaluation with state-tracking mechanisms. The paper identifies greedy, myopic behavior as a limitation (Section 4.6) but provides zero experimental evidence — the evaluation tasks are deliberately single-target and quasi-static. A rigorous follow-up would design a benchmark of tasks that require non-greedy multi-step reasoning, varying the complexity systematically: (a) tasks where the correct action sequence requires temporarily moving away from the visible goal (navigate around an obstacle to reach a target behind it), (b) tasks where state is not fully observable from the current image (drawer opened vs. closed, object already grasped vs. not, previous subgoal completed vs. pending), and (c) tasks where the sequence of subgoals is underspecified by the instruction and must be inferred from common sense ("prepare the table for dinner" → clear objects → wipe surface → set plates). The experiment would compare PIVOT variants that differ in how they handle history: (i) no history (current PIVOT), (ii) text-based history (previous actions and VLM reasoning appended to the prompt), (iii) visual history (previous annotated frames included as additional images in a multi-image VLM input, if the base VLM supports it), and (iv) explicit state tracking (the VLM is prompted to maintain and update a textual state variable across steps). Success rates and specific error categorization (sequencing error vs. spatial error vs. state confusion) would be the primary metrics. This would establish whether the greedy behavior is a fundamental VLM limitation or an artifact of PIVOT's current state-oblivious design, and would produce actionable guidance on how to extend PIVOT to temporally extended tasks.

5. Hybrid zero-shot + minimal fine-tuning Pareto frontier. The paper explicitly studies only zero-shot performance and declines to compare against fine-tuned baselines, which is appropriate for its scope. But a crucial practical question is: given that PIVOT achieves non-zero but unreliable performance (25–100% navigation, 0–67% grasp), how much fine-tuning data is needed to reach a useful reliability threshold (say, 90%+ success), and does PIVOT's visual reasoning structure accelerate this? A follow-up would train behavior cloning policies on increasing amounts of in-domain demonstration data (1, 2, 5, 10, 20, 50 demonstrations per task) and compare to PIVOT with 0 demonstrations, establishing the zero-shot-to-fine-tuned Pareto frontier. The key comparison would be: (a) PIVOT selecting actions zero-shot, (b) a standard behavior cloning policy trained purely on demonstrations, and (c) a hybrid where PIVOT's iterative visual optimization generates candidate actions, but a lightweight learned model (trained on the same demonstrations) serves as the selection/scoring function instead of the VLM. The hypothesis is that (c) might achieve better sample efficiency than (b) because PIVOT's proposal distribution already covers reasonable action regions, and the learned scorer only needs to rank them rather than generate actions from scratch. The experiment would use the same mobile manipulator or Franka tasks from the paper, with the RT-X dataset [38] providing demonstration data. The specific metric would be success rate vs. number of demonstrations, establishing whether PIVOT's structure provides a sample-efficiency benefit even when zero-shot performance is insufficient.

6. Stress-testing VLM annotation robustness under controlled distribution shift. Appendix G tests annotation sensitivity on a single synthetic dataset and a single realistic dataset, but does not systematically vary the type of distribution shift between the VLM's training distribution and the evaluation scenes. A stress-test would evaluate PIVOT on images that systematically depart from internet-image statistics: (a) fisheye or wide-angle camera lenses (common on wrist-mounted robot cameras but rare in internet photos), (b) unusual lighting conditions (low light, harsh shadows, overexposure), (c) motion blur from robot movement, (d) domain-randomized scenes where object textures, backgrounds, and lighting are randomized in simulation to test generalization, and (e) adversarial perturbations designed to cause the VLM to misrank visually annotated actions (e.g., adding visual patterns near specific arrow numbers that bias VLM attention). The experiment would use the simulated RAVENS domain (already set up in the paper, Appendix E) for controlled variation, measuring PIVOT's accuracy degradation as a function of distribution shift magnitude. The goal is not to demonstrate robustness (the paper already shows sensitivity to clutter and object type) but to precisely characterize the fragility surface — under what conditions does PIVOT's performance collapse, and are these conditions rare or common in real-world robot deployment? This would produce a "safe operating envelope" for PIVOT that practitioners can use to decide whether their specific deployment setting is within the method's reliable range.

---

Practical Applications and Downstream Use Cases

1. Rapid prototyping of robot behaviors for research and development. PIVOT enables a robotics researcher to get a new manipulation or navigation task working on a real robot without collecting any demonstration data or training any policy. The paper's results show that for tasks with clear visual affordances and simple single-target structures, PIVOT achieves non-zero (though imperfect) success — 100% reach and 67% grasp on "Pick coke can" with full PIVOT (Table 2). For a researcher who needs to collect a dataset of successful task executions (for later policy training) or who wants to quickly test whether a task is physically feasible before investing in data collection, PIVOT provides an immediate bootstrapping mechanism. The cost in VLM API calls (9 queries per action step, 27–72 queries per task attempt) is negligible compared to the human time that would otherwise be spent teleoperating demonstrations. The specific workflow: use PIVOT to generate 20–50 task attempts, keep the successful ones as seed demonstrations, train an initial policy on those, then iteratively improve. This is analogous to how LLMs are used to generate training data for themselves in self-improvement loops, but for physical robot actions.

2. Emergency or one-off robot deployments where training data cannot exist. Consider a disaster response scenario where a robot is deployed to a novel environment (collapsed building, industrial accident site) and must perform tasks for which no prior training data exists — "open the red valve," "move the debris blocking the doorway," "retrieve the medical kit from the overturned cabinet." These environments and object configurations will never appear in any robot training dataset. PIVOT's zero-shot property means the robot can attempt these tasks immediately using the VLM's general visual reasoning, without waiting for data collection and policy training. The paper's navigation results are directly relevant here: "Go to orange table with tissue box" (75% with 3 parallel) and "Help me find a place to sit and write" (100% with 3 parallel, Table 1) demonstrate that the VLM can interpret open-ended spatial instructions and direct the robot accordingly. The current reliability (50–100% per task) is insufficient for fully autonomous deployment, but in a teleoperated setting where a human supervisor can monitor and intervene, PIVOT could reduce the cognitive load on the operator — the robot proposes actions, the human approves or overrides. The specific benefit is a 3–5× reduction in human attention required per action step compared to full teleoperation, based on PIVOT's success rates suggesting that 2/3 to 3/4 of proposed actions are acceptable and don't require override.

3. Visual grounding pre-screening for large-scale spatial annotation pipelines. The RefCOCO results (Figure 5) show that PIVOT with a single iteration achieves approximately 48% accuracy at placing a marker within the target object's ground-truth bounding box, with normalized distance around 0.055. While this is far from state-of-the-art for supervised localization models, it is achieved zero-shot with no training data. A practical deployment would use PIVOT as a pre-screening filter in a human-in-the-loop annotation pipeline: for each image-question pair, run single-iteration PIVOT to generate a coarse localization; if the VLM's confidence (agreement among the top-ranked points, or the reasoning chain's certainty) is above a threshold, accept the VLM's answer directly; if below, route to a human annotator. Even at 48% accuracy, if the confidence-based threshold can identify the subset where PIVOT is reliable (likely the "Easy" and "Medium" object categories from Table 7, where accuracy reaches 44–100%), a substantial fraction of annotations could be handled automatically, reducing human annotation cost by 30–50% depending on the object-type distribution. The RefCOCO dataset, with its large scale (thousands of examples), is precisely the kind of task where even modest automation yields significant cost savings. The specific metric to track would be the tradeoff between automation rate and annotation quality, as controlled by the confidence threshold.

4. Curriculum generation for robot policy learning. The paper's interaction trajectory analysis (Figure 9) reveals that PIVOT performs well during easy phases (approaching from a distance) and poorly during hard phases (grasping, fine positioning). This suggests a curriculum learning application: use PIVOT to autonomously practice the easy sub-skills of a complex task, gradually building a repertoire of successful partial trajectories, then use human demonstrations or scripted policies only for the hard sub-skills. For example, on a "pick and place" task, PIVOT could autonomously generate hundreds of successful "approach object" trajectories (where its cosine similarity with expert demonstrations is high, Figure 9 left region), which could be used to pretrain an approach policy. The grasp itself could be handled by a small number of human demonstrations or a simple scripted grasp primitive. The transport phase could again use PIVOT-generated data. The specific benefit is reducing the human demonstration burden for complex multi-phase tasks by 50–70%, based on the observation in Figure 9 that PIVOT maintains reasonable performance for approximately 60–70% of the trajectory duration (before and after the interaction dips). The paper does not implement such a curriculum, but the trajectory-phase performance characterization makes the opportunity clear.