SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL

🎯 Pitch

VLMs can be taught to chain multiple 3D vision tools—depth estimators, segmentors, and grasp generators—through interactive reinforcement learning, but only when bootstrapped with teacher demonstrations. Without this curriculum, RL in a 12-tool action space collapses completely, while the two-phase approach lifts a small 3B model to beat GPT‑5 by over 20 points in pose estimation and run real‑robot pick‑and‑place at 86% success.

1. Executive Summary

This paper introduces Double Interactive Reinforcement Learning (DIRL), a two-phase training framework that enables vision-language models to coordinate multiple vision and robotic tools for spatial reasoning tasks. The method trains Qwen2.5-VL-3B-Instruct on a teaching dataset combining single-tool interactive RL traces with frontier-model multi-tool demonstrations, then refines tool coordination through a second interactive RL phase supported by Toolshed, a distributed infrastructure for serving compute-intensive tools such as segmentation, depth estimation, and grasp generation as asynchronous services during training. The resulting model, SpaceTools, achieves state-of-the-art performance across spatial reasoning benchmarks — outperforming the strongest proprietary baseline (Gemini-ER 1.5) by +7.5% on RoboSpatial, exceeding Claude Sonnet 4.5 by +24.4% on pose estimation, and surpassing GPT-5 by +8.3% on grasp prediction — while also demonstrating an 86% success rate on real-world robotic pick-and-place tasks, establishing that multi-tool coordination can be learned through staged interactive RL only when a strong initialization from teacher demonstrations prevents exploration collapse in the large multi-tool action space.

2. Context and Motivation

The Core Problem: VLMs Can See but Not Perceive Spatially

The paper addresses a fundamental limitation of modern vision-language models: while they have achieved strong qualitative visual understanding and can describe scenes in rich natural language, they struggle massively with metrically precise spatial reasoning — the ability to answer questions about exact geometric relationships, 3D positions, distances, object orientations, and physical affordances required for embodied applications like robotics.

This gap is not subtle. As shown in Table 2, general open-source models like Qwen2.5-VL-32B score only 7.28% on pose estimation tasks and achieve 0% on grasp prediction — essentially no better than random guessing. Even frontier proprietary models struggle: GPT-5 reaches only 23.10% on pose estimation and 9.03% on grasp tasks. The paper argues this is because VLMs inherently lack access to the dense geometric representations — depth maps, point clouds, segmentation masks, 3D bounding boxes — that precise spatial reasoning demands.

The critical distinction here is between qualitative recognition ("the cup is on the left side of the table") and quantitative spatial understanding ("the cup is 0.34 meters from the table edge at coordinates (0.67, 0.45)") — the latter being what robotics and embodied systems require for action. The paper's opening Figure 1 illustrates this gap with a concrete example: to determine which pedal is smallest and where to press it requires chaining together object detection, relative size comparison, depth understanding, and precise pointing — a multi-step reasoning pipeline that vanilla VLMs cannot perform reliably.

Why This Problem Matters

The paper motivates this problem along two axes: immediate practical impact and longer-term architectural implications for embodied AI.

Robotics demands precise spatial reasoning for action. For a robot to pick up an object, it needs to: locate the object in 3D, estimate its pose and extent, compute a collision-free grasp pose, and place the object at a specific location. These are inherently geometric operations that require dense 3D representations. As the paper notes in Section 1, "these challenges are amplified in robotics, where perception must seamlessly translate into decision-making and physical action." The real-robot experiments in Section 5.3 demonstrate that when spatial reasoning fails, robots fail — the π0.5 vision-language-action model achieves 0% success on pick-and-place tasks because spatial understanding is not sufficiently grounded.

Conventional fine-tuning approaches don't scale. The existing paradigm for teaching VLMs spatial capabilities involves fine-tuning on task-specific datasets — collecting large-scale annotations for depth estimation, pointing, 3D awareness, etc. (Section 2). The paper argues this approach is fundamentally limited because it requires: (a) extensive data engineering for each new perceptual capability, (b) architectural modifications to output dense predictions from language model heads, and (c) continued retraining as new capabilities are added. Each new perceptual skill (depth, segmentation, pose) essentially becomes a separate data collection and training problem.

The compositional nature of spatial reasoning demands flexible tool coordination. Real spatial reasoning tasks are not satisfied by any single perceptual capability. Determining "which pedal is smallest and where would you activate it" requires object detection to find pedals, depth or 3D estimation to compare sizes, and precise pointing to identify the activation location. These tasks naturally decompose into a sequence of perceptual queries that different specialized models handle best. As the paper puts it in Section 1, tool use "provides access to precise measurements and intermediate geometric representations, can leverage computer vision models from VLM-incompatible settings (e.g., dense prediction), and allows combining the strengths of heterogeneous models to augment base-model capability."

Where Prior Approaches Fall Short

The paper identifies specific limitations in four categories of prior work:

1. Training-free tool prompting strategies are inflexible and brittle.

Several works have equipped VLMs with vision tools through handcrafted prompting strategies — Visual Programming (Gupta & Kembhavi, 2023), Set-of-Mark prompting (Yang et al., 2023), Visual Sketchpad (Hu et al., 2024), and Visual Agentic AI (Marsili et al., 2025). These approaches provide the model with tool APIs at inference time and rely on the model's existing reasoning capabilities to decide when and how to call them.

The paper identifies two fundamental weaknesses with this approach:

Fixed, predefined pipelines limit adaptability. Methods like SpatialPIN (Ma et al., 2024) and APC (Lee et al., 2025) enforce a predetermined sequence of tool calls rather than allowing the model to discover optimal tool-use patterns for each specific query. As Table 1 shows, these approaches use tools but do not permit non-fixed tool pipelines — the model cannot adapt its strategy based on intermediate results.
No learning from tool interaction. When a model encounters tool failures, noisy outputs, or ambiguous results, training-free approaches have no mechanism to improve. The model either hallucinates corrections (e.g., GPT-5 inventing grasp poses or camera intrinsics, as reported in Section 5.3) or produces incorrect answers. The paper demonstrates this empirically in Table 5: adding Toolshed to GPT-5 and Claude Sonnet 4.5 increases their scores on low-level tasks (RefSpatial, pose, grasp) but on high-level reasoning tasks like RoboSpatial, performance actually declines because "models tend to overuse tools and struggle to correctly interpret nuanced tool outputs" (Section 6).

2. Supervised fine-tuning on tool-use traces is insufficient.

TIGeR (Han et al., 2025), a concurrent work, focuses on problem-solving via code generation with tools but derives its supervision from "a predefined synthetic tool pipeline with large-model-based rewriting." The paper argues (Section 2, Table 1) that TIGeR's reliance on precomputed tool outputs and non-interactive training prevents models from learning interactive, state-dependent tool use. The model never experiences tool failures, never learns to recover from errors, and never discovers alternative strategies when a preferred tool fails.

The paper's ablation study in Table 4 makes this point quantitatively: Tool SFT (supervised fine-tuning on multi-turn tool-use traces from the universal teacher, without any interactive RL) achieves a mean score of 39.19 across benchmarks, compared to 52.48 for the full DIRL method — a gap of +13.4 percentage points. This demonstrates that "interactive RL is key to teaching VLMs consistent reasoning over complex tool sequences" (Section 5.4).

3. Naïve interactive RL with many tools fails due to combinatorial action space explosion.

ViGoRL (Sarch et al., 2025) demonstrated that reinforcement learning can enable a VLM to learn grounded reasoning with a single visual tool — specifically a cropping operation. This was an important proof of concept showing that interactive RL for tool use is feasible.

However, the paper identifies a critical scaling challenge that ViGoRL does not address: "scaling to multiple heterogeneous tools poses a fundamental challenge: with 10+ tools, the action space grows combinatorially, causing naive RL exploration to fail" (Section 2). When the model must simultaneously learn which tool to call, in what order, with what arguments, and how to interpret the results — all while exploring through random trial and error — the optimization signal becomes too weak to discover effective policies.

This is not merely a hypothesis. The paper provides direct experimental evidence in Table 9 (Appendix E.2): applying interactive RL directly with all tools on all tasks ("Direct IRL All.") achieves a mean score of only 19.79 across benchmarks, compared to 52.48 for DIRL. On RefSpatial, it scores 3.25 vs. 53.07. The model essentially collapses — it cannot discover productive tool-use strategies from scratch in such a large search space.

4. Spatial VLM fine-tuning approaches require per-capability data engineering.

Specialized spatial VLMs like SpaceLLaVA-13B (Chen et al., 2024), RoboPoint-13B (Yuan et al., 2024), and RoboBrain2.0-7B (Team, 2025) represent the conventional approach: fine-tune on task-specific spatial reasoning datasets to bake perceptual skills directly into model weights. The paper acknowledges these efforts but notes their fundamental limitation: "these methods require large-scale data collection and architecture modifications even to introduce a single low-level perceptual capability such as depth, pointing, and 3D-awareness" (Section 2).

This approach does not scale. Adding depth perception requires one dataset and potentially architectural changes. Adding segmentation requires another. Adding grasp prediction requires yet another. Each capability is siloed, and the model's size grows to accommodate all these skills — skills that specialized computer vision models already perform better. The paper's alternative vision is that VLMs should not become perception models but should orchestrate them.

How This Paper Positions Itself

The paper situates its contribution at the intersection of three research threads that have previously been explored in isolation, arguing that their synthesis is the key to unlocking multi-tool spatial reasoning. This positioning is most clearly articulated in Table 1, which systematically compares prior work across five axes: whether they use SFT, RL, tools, non-fixed tool pipelines, and interactive tool calls during training. No prior work checks all five boxes.

The key insight: staged learning decomposes the combinatorial problem.

The paper's central conceptual contribution — beyond the specific method — is the recognition that learning to coordinate multiple tools can be decomposed into progressive, tractable phases. The intuition, as articulated in the introduction, is that:

Learning to use a single pointing tool is tractable via RL because the action space is constrained (one tool, one type of query), the reward signal is clean (distance to target), and grounding is a prerequisite for using most other vision tools (you typically need to locate an object before you can segment it, estimate its depth, or compute its pose). The paper demonstrates this in Appendix E.1 (Table 8): IRL with just the pointing tool on RoboSpatial achieves 72.3% overall accuracy and — critically — generalizes to 34.3% on RefSpatial, whereas tool-free fine-tuning achieves 0%.
Learning to coordinate many tools requires good initialization. The single-tool RL model provides demonstrations of grounded reasoning that the multi-tool model can imitate, while the frontier model (Claude Sonnet 4.5) provides demonstrations of multi-tool coordination patterns. Together, these create a teaching dataset that covers both grounding (from the IRL teacher) and composition (from the universal teacher). Without this initialization, the model faces the combinatorial explosion described above.
Interactive RL refines but does not invent from scratch. Once the model has basic tool-use competence from the teaching phase, the second IRL phase allows it to discover improved strategies — when to retry a failed tool, when to switch to an alternative tool, when to fall back to self-estimation. But crucially, it is not starting from random exploration; it is refining an already-functional policy.

This staged approach is the paper's answer to why prior work either limited itself to single tools (ViGoRL) or relied on fixed pipelines (SpatialPIN, APC). Neither approach alone could solve the multi-tool coordination problem; together — with the right staging — they can.

The Toolshed infrastructure as an enabling contribution.

The paper is explicit that part of its contribution is systems-level: making interactive multi-tool RL practically feasible at the scale needed for VLM training. Section 4.2 describes how prior work either sidestepped this problem entirely (using precomputed tool outputs, as TIGeR did) or limited to simple tools that could run synchronously within the training loop (cropping in ViGoRL). The paper argues that "naïve application of RL to many tools creates a prohibitively large search space where exploration fails" not only algorithmically but also from a systems perspective — if tool execution bottlenecks the training loop, the number of RL steps per wall-clock hour collapses.

Toolshed solves this by decoupling tool execution from policy inference, running tools as asynchronous services on separate GPU resources, and supporting parallel tool instances. This architecture choice — described in detail in Appendix B — is what makes the two-phase RL paradigm computationally tractable. The paper is positioning Toolshed not as an engineering footnote but as a necessary contribution that enables the learning contribution.

Relationship to the broader tool-augmented reasoning literature.

In the context of tool-augmented reasoning for LLMs (search engines, calculators, code executors), this work extends the paradigm to a domain — spatial reasoning with vision tools — where tools are not lightweight API calls but compute-intensive models (SAM2, DepthPro, GraspGen) that produce dense outputs (segmentation masks, depth maps, point clouds). The challenge is qualitatively different: web search returns text snippets that the LLM can directly ingest; a depth estimator returns a 2D array of floating-point values that the VLM must learn to query, interpret, and combine with other spatial representations.

The paper also positions itself against the concurrent TIGeR work (Section 2), emphasizing that TIGeR relies on precomputed, synthetic tool pipelines whereas DIRL uses "real and stochastic tool outputs into the learning loop, [exposing] models to actual tool behavior, encouraging reasoning about tool reliability and discovering improved ways to query the tools." This distinction — learning from real tool interactions versus learning from idealized teacher traces — is central to the paper's claim that interactive RL enables capabilities (error recovery, tool selection strategy, output interpretation) that pure SFT cannot teach.

The framing as a scalable alternative to dataset-specific fine-tuning.

Throughout the introduction and related work, the paper frames tool-augmented spatial reasoning as fundamentally more scalable than the dominant paradigm of baking perceptual skills into model weights through task-specific fine-tuning. The argument is that:

New capabilities can be added by upgrading tools, not retraining models. If a better depth estimator is released, Toolshed can swap it in without modifying the VLM's weights.
The VLM focuses on reasoning, not perception. The model does not need to learn to estimate depth; it needs to learn when depth information is useful and how to query it. This is a higher-level reasoning skill that generalizes across tools and tasks.
Data requirements shift from annotation to demonstration generation. Rather than collecting thousands of annotated examples for each spatial reasoning subtask, the teaching dataset is generated by teachers (an IRL-trained model and a frontier model) interacting with the same tools the student will use. This is more scalable because tool interactions produce rich supervision automatically.

This framing positions the paper not as an incremental improvement to spatial VLM training but as a rethinking of what capabilities should live in the VLM versus in external tools — a design choice with implications for model architecture, training methodology, and system deployment.

3. Technical Approach

3.1 Reader Orientation

This paper builds SpaceTools, a vision-language model (VLM) that learns to coordinate multiple computer vision and robotics tools — such as depth estimators, segmentation models, 3D bounding box fitters, and robot arm controllers — to solve spatial reasoning tasks that vanilla VLMs fail at. The core idea is that no single training strategy works: teaching a VLM to use many tools requires a staged curriculum where a single-tool specialist trained via reinforcement learning provides grounding demonstrations, a frontier model provides multi-tool coordination demonstrations, and then the student model refines its own tool-use strategies through a second round of interactive reinforcement learning — hence "Double Interactive RL."

3.2 Big-Picture Architecture (Diagram in Words)

The SpaceTools system consists of four major components:

A base VLM (Qwen2.5-VL-3B-Instruct) — the language model backbone with visual encoder that generates reasoning text, tool calls, and final answers in a structured multi-turn conversation format. This is the policy $\pi_\theta$ being trained.
Toolshed — a distributed infrastructure that hosts computationally heavy computer vision tools (SAM2 for segmentation, DepthPro for monocular depth, RoboRefer and Molmo for pointing, GraspGen for grasp generation, 3D bounding box fitting, and image operations) as asynchronous services on separate GPU resources. It also hosts robotic tools (image capture, depth capture, grasp execution, object placement) for real-world manipulation. Toolshed decouples tool execution from policy inference, enabling parallel, non-blocking tool calls during training rollouts.
The teaching dataset generation pipeline — two teacher models (an IRL-trained single-tool specialist using only a pointing tool, and Claude Sonnet 4.5 using all tools) generate multi-turn trajectories of reasoning, tool calls, tool responses, and final answers on spatial reasoning questions. These trajectories form the supervised fine-tuning dataset for Phase 1.
The Double Interactive RL (DIRL) training loop — a two-phase procedure: Phase 1 performs supervised fine-tuning (SFT) on the teaching dataset to establish basic tool-use competence; Phase 2 runs Group Relative Policy Optimization (GRPO) with the full toolset, where the model generates multi-turn tool-use trajectories in Toolshed, receives task-specific rewards based on answer correctness, and updates its policy through advantage-weighted optimization with KL regularization against the Phase 1 checkpoint.

Information flows as follows: an image-text query enters the system → the VLM generates reasoning text and optional tool calls in a structured XML-tagged format → Toolshed executes any called tools asynchronously and returns structured outputs (text, images, numerical variables) → the VLM incorporates tool outputs into its continuing reasoning → this repeats for up to $T_\text{max}$ turns → the VLM produces a final answer in <answer> tags → the answer is scored against ground truth via task-specific reward functions → the reward drives policy updates during RL phases.

3.3 Roadmap for the Deep Dive

First, the problem formulation as a sequential decision-making process (Algorithm 1), because it defines the action space, observation space, and reward structure that all subsequent training components operate within.
Second, the Toolshed infrastructure, because it is the systems foundation that makes interactive tool use during training computationally feasible — without it, the two-phase RL paradigm would be impractically slow.
Third, the DIRL framework's teaching phase — the data generation process, the two teacher models, and the supervised fine-tuning step — because this is what provides the initialization that prevents exploration collapse.
Fourth, the DIRL framework's exploration phase — the GRPO algorithm, reward design, and policy update mechanics — because this is where the model learns tool coordination beyond what teachers demonstrate.
Fifth, the reward functions for each task type, because they shape what behaviors the RL process reinforces and explain why the model learns particular tool-use strategies.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper with a significant systems contribution. The core idea is that learning to coordinate multiple heterogeneous tools for spatial reasoning requires decomposing the learning problem into a teaching phase (which establishes basic competence through imitation of specialist and generalist teachers) and an exploration phase (which refines tool coordination through interactive trial-and-error with real tool feedback). The enabling systems contribution is Toolshed, a distributed infrastructure that makes interactive multi-tool RL computationally tractable at the scale needed for VLM training.

Problem Formulation: Spatial Reasoning as Sequential Tool-Augmented Decision Making

The paper formalizes tool-augmented spatial reasoning as a multi-turn sequential decision-making problem. This formulation is spelled out in Algorithm 1 and serves as the mathematical foundation for both the training objectives and the inference procedure.

State and observation space. At each turn $t$ , the VLM policy $\pi_\theta$ receives a dialogue history $h_t$ containing the full conversation so far: the original user query $I$ (which may be an image-text pair or a robotic manipulation instruction), all previous VLM responses, and all previous tool outputs. The history is initialized as $h_1 = I$ . Critically, tool outputs can include not just text but also structured data — segmentation masks as boolean arrays, depth maps as floating-point arrays, 3D point clouds, grasp poses as $4 \times 4$ transformation matrices, and captured images — allowing the VLM to access rich spatial information that goes beyond what text alone can convey.

Action space. At each turn, the VLM generates a response $a_t \sim \pi_\theta(h_t)$ that can contain three types of content, demarcated by XML-style tags:

Reasoning enclosed in thinking tags — the model's internal chain-of-thought analysis of the current state, what information it needs, and how it plans to proceed. This reasoning is visible in the dialogue history and conditions future decisions.
Tool calls enclosed in <tool call> tags — JSON objects specifying a tool name and arguments. Multiple tool calls can appear in a single response, and they are executed sequentially by Toolshed. Tool calls can reference variables stored from previous tool outputs (e.g., $segmentation_mask, $point_cloud, $focal_length_px), creating dataflow dependencies across turns.
Final answer enclosed in <answer> tags — the model's solution to the original query, produced when the model determines that no further tool calls are needed. The answer format depends on the task: a multiple-choice letter, a set of 2D normalized coordinates, a list of 3D corner points, or a robot action parameterization.

Episode termination. The process continues iteratively — generate response, execute tools, append to history, repeat — until either the model produces an <answer> tag (successful completion) or a maximum number of turns $T_\text{max}$ is reached (forced termination). This creates a natural variable-length execution trace where the model autonomously decides how many tools to call and in what order.

Why this formulation? The sequential decision-making framing is deliberate. It transforms tool coordination from a one-shot planning problem (which would require the model to predict all necessary tool calls upfront) into an interactive, state-dependent process where the model can observe tool outputs, detect failures, and adapt its strategy mid-execution. This is essential for spatial reasoning because tool outputs are often noisy or incomplete — a pointing tool might mislocalize an object in a cluttered scene, or a grasp generator might fail to find collision-free grasps. The multi-turn structure allows the model to learn corrective behaviors (switching to an alternative pointing tool, falling back to manual estimation) that would be impossible in a single-turn, pre-planned pipeline.

The structured conversation format. The requirement that the model follow a specific XML-tagged format serves two purposes. First, it makes tool calls machine-parseable — Toolshed can reliably extract function names and arguments from model outputs without ambiguity. Second, it enables reward computation during RL training: the training system can verify whether the model produced properly formatted tool calls, whether reasoning preceded each tool call, and whether exactly one final answer was produced.

Toolshed: Distributed Infrastructure for Interactive Multi-Tool Execution During Training

Toolshed is the systems foundation that makes interactive multi-tool RL practically feasible. The paper identifies a specific bottleneck that prior work either sidestepped or failed to address: when training a VLM to use computationally heavy vision tools (segmentation, depth estimation, grasp generation), tool execution time dominates the training loop, making interactive RL impractically slow. This is not a minor engineering concern — it is a gating factor. If each tool call takes 500ms and the model makes 3 tool calls per trajectory, a batch of 64 rollouts would spend 96 seconds just waiting for tools, during which the GPU training the VLM sits idle.

The problem with naïve integration. In a straightforward implementation, tool execution would be tightly coupled with the policy's inference loop: the VLM generates a response, the training loop detects a tool call, executes the tool synchronously, appends the output to the history, and feeds the updated history back to the VLM for the next generation step. This serial execution creates two problems:

Blocking stalls. A single slow tool call blocks the entire batch, since all rollouts in a batch must complete before gradients can be computed.
Resource contention. Vision tools like SAM2 and DepthPro require GPU memory and compute. Running them on the same GPU as the VLM during training would cause memory exhaustion or severe contention, forcing the use of smaller models or batch sizes.

Toolshed's architectural solutions. The paper describes five design principles (Section 4.2 and Appendix B) that together solve these problems:

Decoupled execution. Tool invocations run as independent processes separate from the VLM's inference loop. When the VLM generates a tool call, it is dispatched to Toolshed and the training loop continues processing other rollouts. Results are returned asynchronously and inserted into the dialogue history when ready.
Asynchronous parallel workers. Each tool type is served by multiple parallel instances (actors in the Ray distributed framework). The paper specifies concrete resource allocations: point1 and point2 tools get 2 actors each with 0.5 GPU, sam2 gets 4 actors with 0.2 GPU each, depth_estimator gets 4 actors with 0.2 GPU each. This means multiple rollouts can call the same tool simultaneously without queuing.
Resource isolation. Tool instances are assigned dedicated GPU resources separate from the GPU(s) used for VLM training. The paper uses a Ray placement group with {"CPU": 8, "GPU": 8} for Toolshed, distinct from the 8 GPUs used for the VLM training process. This prevents memory contention and allows both the VLM and tools to run at full capacity.
Environment isolation. Each tool type runs in its own Python environment with its specific dependencies, solving the common problem where SAM2 requires one version of PyTorch, DepthPro needs another, and GraspGen needs specific CUDA kernels. Without this isolation, dependency conflicts would make it impossible to host all tools in a single system.
Multimodal data passing. Toolshed supports exchanging not just text but also images (PIL Image objects), structured arrays (numpy ndarrays for depth maps, segmentation masks, point clouds), and scalar variables (focal lengths, coordinates) between the VLM and tools, even when they run on different physical machines.

Implementation details. Toolshed is built on the Ray distributed execution framework, which provides lightweight task scheduling, actor management, and high-throughput message passing. For interactive RL training, Toolshed integrates with the VERL framework: "VERL's asynchronous multi-turn rollouts align naturally with Toolshed's asynchronous tool actors, enabling us to parallelize expensive perception, generation, and simulation steps without slowing down rollouts" (Appendix B). The paper reports this results in "significantly higher steps-per-second compared to monolithic or synchronous training setups."

The mock robot system. For robotic manipulation data, running the physical robot during training would be prohibitively slow. Toolshed provides mock robot tools that mirror the real robot's API but always simulate successful actions when given valid arguments. This allows the teaching data generation phase (where Claude Sonnet 4.5 calls robot tools) and the spatial reasoning RL training to proceed without requiring the physical robot in the loop. The real robot is only used during the final evaluation experiments in Section 5.3.

Why this design matters for the learning contribution. The paper emphasizes that Toolshed is not just an implementation detail — it is what enables the key learning insight that "interactive RL is key to teaching VLMs consistent reasoning over complex tool sequences" (Section 5.4). Without Toolshed's throughput, the number of RL steps per hour would be so low that the exploration phase of DIRL would not converge. The paper's ablation study in Table 4 makes this point indirectly: Tool SFT (which uses Toolshed for data generation but not for interactive training) achieves 39.19 mean score, while the full DIRL method (which uses Toolshed for both SFT data generation and interactive RL rollouts) achieves 52.48. The gap of +13.4 points is only achievable because Toolshed makes interactive RL with real tool feedback computationally tractable.

The Tool API: Vision and Robotic Tools Available to the Model

The paper provides a detailed API specification in Appendix B.5. Understanding what tools are available and what they return is essential for understanding what the model learns to coordinate. I will describe each tool category and its role in spatial reasoning.

Pointing tools (point1 and point2). These are the foundational tools for spatial grounding — they take an image and an object name string and return the normalized 2D image coordinates $(x, y) \in [0,1]^2$ of the detected object. Two implementations are provided: point1 uses RoboRefer, point2 uses Molmo. Both support detect_one (find one instance) and detect_all (find all instances). The output includes not just the coordinates but also an annotated image with red circular markers at detected locations, and the coordinates are stored as named variables (e.g., $coconut_water_detection) for downstream tool calls.

The paper's rationale for including two pointing tools is that they have different failure modes and strengths, and the model can learn to use both strategically — for example, calling point2 if point1's detection seems inaccurate, as shown in the relative depth example in Figure 3 where the model switches pointing tools because "the values are very close, which suggests the tools may have made wrong estimations."

Segmentation tool (sam2). SAM2 performs instance segmentation from one or more 2D click points. Given an image and normalized coordinates, it returns a boolean mask (H×W numpy array) indicating which pixels belong to the object, along with an IoU confidence score and a visualization with a semi-transparent green overlay. The mask is stored as a named variable ($segmentation_mask) for downstream tools like 3D bounding box fitting and grasp generation, which need to know which subset of the point cloud belongs to the target object.

Depth estimation tools (depth_estimator). DepthPro provides monocular depth estimation with two modes: estimate_depth returns a depth map (H×W float array of metric depths in meters) and the camera's focal length in pixels; estimate_depth_with_pointcloud additionally returns a 3D point cloud (N×3 float array in camera coordinates) by back-projecting the depth map. The depth values are in absolute meters, which is critical — the model can compare numeric depth values across objects to determine relative distance, rather than relying on qualitative depth cues.

3D bounding box fitting (3d_bbox). Given a point cloud, a segmentation mask, and the focal length, this tool fits an oriented 3D bounding box to the masked subset of points. It returns the 8 corner points in both 3D camera coordinates (meters) and 2D normalized image coordinates, the box edges (as pairs of corner indices), and the extent (width, height, depth in meters). This enables precise pose estimation — the model can query the object's exact 3D position, orientation, and dimensions.

Grasp generation (compute_grasp). GraspGen generates collision-free 6-DOF grasp poses for a masked object in a point cloud. It takes the point cloud, segmentation mask, image, and focal length, and returns a $4 \times 4$ transformation matrix representing the grasp pose in the camera frame (OpenCV convention), along with a confidence score and a visualization of the gripper axes. In the text output, it also provides the projected 2D gripper points (grasp center, left/right finger bases and tips) in normalized coordinates. The paper notes that the tool reports statistics about grasp generation: how many total grasps were generated and what percentage were collision-free, giving the model information about scene clutter.

Image operations (image_ops). Basic image manipulation tools: point_crop extracts a pixel value from a numpy array at given normalized coordinates; point_crop (overloaded for images) crops an image to the bounding box of given points; mask_crop crops to a mask's bounding box and sets outside pixels to white.

Code executor (code_executor). The VLM can execute small Python snippets (imports limited to math and numpy) for orchestrating multi-tool workflows or performing calculations on tool outputs. The tool returns the execution result, captured stdout, and stderr. The exec mode runs multi-line code blocks; the eval mode evaluates single expressions. Results can be cached as named variables for reuse.

Robotic tools. For real-world manipulation, the robot exposes: capture_image (returns current RGB image from the robot's onboard camera), get_depth and get_depth_with_pointcloud (returns depth map and optionally point cloud in the robot frame), execute_grasp (moves the end-effector to a specified $4 \times 4$ grasp pose via a pre-grasp point and closes the gripper), and place_object (moves to a specified 2D or 3D location and opens the gripper). All motions are executed with the CuRobo motion planner on a Kinova Jaco arm with a ZED2 RGB-D camera.

Why this specific tool set? The tools are designed to form a composable perception pipeline: pointing localizes objects in 2D → segmentation extracts pixel masks → depth estimation provides 3D geometry → 3D bounding box fitting gives object extent and pose → grasp generation computes manipulation affordances. Each tool's output serves as input to downstream tools, and the model learns to chain them appropriately. The paper emphasizes that the tool set is modular — a better depth estimator or grasp generator could be swapped in without retraining the VLM, since the model learns tool-agnostic coordination patterns.

Double Interactive RL (DIRL): The Teaching Phase

The teaching phase of DIRL addresses the fundamental challenge identified in the paper: naïve interactive RL with 10+ tools creates a combinatorial action space so large that random exploration cannot discover effective policies. The solution is to provide the model with a strong initialization through supervised fine-tuning on high-quality demonstrations that cover both single-tool grounding and multi-tool coordination.

The teacher models. The paper uses two complementary teachers, each providing different kinds of expertise:

The IRL-trained teacher (single-tool specialist). The base model (Qwen2.5-VL-3B-Instruct) is first trained via interactive RL to use only the pointing tool for spatial reasoning tasks — specifically, spatial relationship questions, spatial compatibility questions, and relative depth reasoning. Because the action space is constrained (one tool, binary or coordinate answers, clean reward signals based on distance metrics), IRL converges reliably. The resulting model achieves 72.3% overall accuracy on RoboSpatial (Table 8, Appendix E.1). This teacher generates 2,000 trajectories of grounded reasoning — showing the student model how to use pointing to locate objects before answering spatial questions.

Why a pointing specialist? The paper argues that pointing is the "common first step before querying other vision and robotic tools in spatial reasoning" (Section 5, Dataset paragraph). To segment an object, you first need to know where it is. To estimate its depth, you need its 2D location to query the depth map. To compute a grasp, you need to locate the object and segment it. Teaching the model to reliably ground objects in 2D is therefore a prerequisite for all downstream multi-tool reasoning.

The universal teacher (frontier model). Claude Sonnet 4.5, integrated with Toolshed and given access to all tools (pointing, segmentation, depth estimation, 3D bbox, grasp generation, code execution, mock robot tools), generates solutions for spatial reasoning and robot manipulation tasks. Crucially, only trajectories that lead to correct solutions are retained — incorrect trajectories are filtered out. This teacher generates 6,000 trajectories.

Why Claude Sonnet 4.5? The paper does not claim Claude Sonnet 4.5 is inherently good at spatial reasoning — Table 2 shows its raw spatial reasoning scores are often lower than specialized models. The point is that Claude Sonnet 4.5, as a frontier model, has strong general reasoning and instruction-following capabilities. When given access to precise tools via Toolshed and provided with "additional instructional prompts that encourage careful interpretation of tool outputs and help understanding of image coordinate systems" (Appendix D.1), it can produce coherent multi-tool trajectories that demonstrate useful coordination patterns — even if the underlying model would not have produced those patterns without tools.

The teaching dataset composition. The final SFT dataset consists of 8,000 high-quality tool-use trajectories: 2,000 from the IRL-trained teacher and 6,000 from the universal teacher — a 1:3 ratio. This ratio is not arbitrary: the IRL teacher provides depth in grounding (many examples of careful pointing and spatial reasoning with a single tool), while the universal teacher provides breadth in tool coordination (examples of chaining multiple tools for pose estimation, grasp prediction, and robot manipulation). The image-question pairs are sampled from three datasets: RoboSpatial, RefSpatial, and BOP-ASK. For robot manipulation, the HOPE dataset is augmented with grasping and pick-and-place control tasks, and mock robot tools are used during data generation to avoid requiring the physical robot.

Trajectory structure. Each trajectory is a multi-turn dialogue. A simplified example from the RoboSpatial spatial compatibility task would look like:

User: "Can the tissue box fit left of the bowl?" + image
Assistant (turn 1): thinking I need to locate both objects and assess available space. response <tool_call> point1.detect_one("tissue box", image) and point1.detect_one("bowl", image) </tool_call>
Tool response: tissue_box_detection = (0.306, 0.531), bowl_detection = (0.452, 0.92)
Assistant (turn 2): thinking The tissue box is at left-center, the bowl is lower-center. Looking at the image, there appears to be adequate floor space to the left. The tissue box is relatively small. response <answer> Yes </answer>

For more complex tasks like pose estimation, the trajectory would include multiple tool calls across several turns (point → segment → depth → 3D bbox → answer with corner coordinates), with the model reasoning about each tool output before deciding what to call next.

Supervised fine-tuning details. The SFT step trains the base model (Qwen2.5-VL-3B-Instruct) on this teaching dataset using standard next-token prediction loss. Key training configurations from Table 6:

Batch size: 8
Learning rate: $1 \times 10^{-5}$
Epochs: 2
Warmup ratio: 0.1
LR schedule: cosine
Max prompt length: 8192 tokens
Max response length: 8192 tokens
Trainable parameters: language model only (2.55B parameters); vision encoder and projector are frozen
Training framework: LLaMA-Factory

The loss is computed only over the assistant's turns in each multi-turn dialogue — the model learns to predict the teacher's reasoning, tool calls, and final answers conditioned on the conversation history. Tool outputs (which are deterministic given the input) are not included in the loss computation.

Why freeze the vision encoder? The paper's rationale (implicit in the training setup) is that the visual perception capabilities should come from the tools, not from retraining the VLM's visual backbone. The VLM's vision encoder provides a general visual understanding that is sufficient for deciding which tools to call and interpreting their textual/numerical outputs. Freezing the vision encoder also reduces training cost and prevents catastrophic forgetting of the base model's general visual capabilities.

What the teaching phase does not teach. Critically, the SFT phase only exposes the model to successful trajectories — it never sees tool failures, ambiguous outputs, or error recovery. The model learns what correct tool use looks like but does not learn how to handle the stochastic, sometimes-noisy reality of real tool execution. This limitation is why the exploration phase is necessary.

Double Interactive RL (DIRL): The Exploration Phase

The exploration phase takes the SFT-initialized model and runs interactive RL with the full toolset, allowing the model to discover tool-use strategies that go beyond the teacher demonstrations. This phase is what makes DIRL "double" — it is the second application of interactive RL in the pipeline (the first being the training of the IRL teacher that contributes to the SFT dataset).

Policy optimization algorithm: Group Relative Policy Optimization (GRPO). The paper uses GRPO rather than the more common PPO for several practical reasons. GRPO is a simplified variant that eliminates the need for a separate value function (critic) model, which would double the memory requirements during training — a significant concern when training a 3B-parameter VLM alongside a full tool infrastructure.

The GRPO procedure works as follows. For each input query $I$ , the current policy $\pi_\theta$ generates $N = 5$ independent rollout trajectories $O_1, O_2, \ldots, O_N$ . The paper uses $N = 5$ rollouts per input (Table 6: "Rollout Number: 5"). Each rollout is a complete multi-turn interaction with Toolshed — the model calls real tools, receives real (not precomputed) outputs, and produces a final answer. Each rollout receives a scalar reward $r_i$ based on the correctness of its answer (reward functions are detailed in the next section).

Advantage computation. Rather than using raw rewards directly, GRPO normalizes rewards within each group of $N$ rollouts to compute relative advantages:

$A_i = \frac{r_i - \text{mean}(\{r_1, r_2, \ldots, r_N\})}{\text{std}(\{r_1, r_2, \ldots, r_N\})}$

where $\text{mean}(\{r_1, \ldots, r_N\})$ is the average reward across the $N$ rollouts for input $I$ , and $\text{std}(\{r_1, \ldots, r_N\})$ is their standard deviation. $A_i$ is the standardized advantage for rollout $i$ .

What this computes: For each rollout, the advantage $A_i$ measures how much better or worse that rollout's answer was compared to the average answer quality for the same input. A positive $A_i$ means this rollout found a better-than-average answer; a negative $A_i$ means it found a worse-than-average answer. The standardization (subtracting mean, dividing by standard deviation) ensures that advantages are zero-centered with unit variance, making the optimization scale-invariant to the absolute magnitude of rewards.

Why group-relative? The key property of group-relative advantage is that it automatically adapts to question difficulty. On a very hard question where all rollouts get near-zero reward, the advantages are all near zero — the model is not penalized for failing on genuinely impossible problems, nor is it rewarded for lucky guesses. On an easy question where most rollouts succeed, the one rollout that fails gets a strongly negative advantage. This difficulty-adaptive property is crucial for multi-task training where question hardness varies dramatically (compare easy spatial compatibility questions to hard grasp estimation).

Policy gradient loss. The policy is updated by minimizing the GRPO objective:

$\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E}_i \left[ -\min\left( \rho_i A_i, \text{clip}(\rho_i, 1 - \epsilon, 1 + \epsilon) A_i \right) + \beta \, \text{KL}\left( \pi_\theta \| \pi_{\text{ref}} \right) \right]$

where $\rho_i = \frac{\pi_\theta(O_i | I)}{\pi_{\text{ref}}(O_i | I)}$ is the importance sampling ratio — the probability of generating rollout $O_i$ under the current policy $\pi_\theta$ divided by its probability under the reference policy $\pi_{\text{ref}}$ . The reference policy is the VLM checkpoint from the end of Phase 1 SFT. $\epsilon$ is the clipping threshold (the paper uses $\epsilon = 0.2$ , the standard PPO value, though this is not explicitly stated in the paper — it follows from using the standard GRPO implementation). $\beta$ is the KL penalty coefficient; the paper uses $\beta = 1 \times 10^{-4}$ (Table 6).

What this loss computes, term by term:

Clipped policy gradient: $-\min(\rho_i A_i, \text{clip}(\rho_i, 1-\epsilon, 1+\epsilon) A_i)$ . When the advantage $A_i$ is positive, the loss encourages increasing the probability of rollout $O_i$ , but the update is clipped so that $\rho_i$ cannot exceed $1 + \epsilon$ — preventing the policy from changing too drastically in a single update. When $A_i$ is negative, the loss encourages decreasing the probability, but clipping prevents $\rho_i$ from dropping below $1 - \epsilon$ . This is the standard PPO clipping mechanism that stabilizes training by preventing destructively large policy updates.
KL regularization: $\beta \, \text{KL}(\pi_\theta \| \pi_{\text{ref}})$ . This term penalizes the current policy for diverging too far from the Phase 1 SFT checkpoint. It is essential because without KL regularization, the RL process can cause the model to "forget" its basic tool-use competence while chasing higher rewards — a form of catastrophic forgetting in RL. The paper notes that "a relatively small KL value is necessary to encourage sufficient exploration during RL. However, this introduces a trade-off in training stability — specifically, we observe an initial drop in rewards during Phase-1 IRL when using a smaller KL coefficient" (Appendix D.1).

Why this form? The GRPO objective is the standard clipped surrogate objective from PPO applied in a group-relative setting. The paper chose GRPO over PPO because eliminating the critic network reduces memory requirements (important when training alongside Toolshed) and simplifies the training setup. The group-relative advantage normalization provides automatic reward scaling that is particularly well-suited to the mixed-difficulty, multi-task nature of the training data.

Training hyperparameters for Phase 2 IRL (Table 6):

Batch size: 64
Learning rate: $1 \times 10^{-6}$
Epochs: 2
Warmup ratio: 0.0
KL coefficient: $1 \times 10^{-4}$
Entropy coefficient: 0.0 (no explicit entropy bonus — the KL term already prevents policy collapse)
Temperature: 1.0 (during rollout generation)
Max prompt length: 8192 tokens
Max response length: 8192 tokens
GPUs for VLM: 8
GPUs for Tools: 8 (separate from VLM GPUs)
Trainable parameters: language model only (2.55B), same as Phase 1

The search space challenge and why DIRL's initialization solves it. To appreciate why the teaching phase is necessary, consider what the exploration phase would look like without it. The model starts from a base VLM checkpoint that has never used tools. In each rollout, it must simultaneously decide: (a) whether to call a tool or answer directly; (b) which of 10+ tools to call; (c) what arguments to pass (object names, coordinates, variable references); (d) in what sequence to chain tools; (e) how to interpret tool outputs; (f) when to stop and answer. The reward signal is sparse — it arrives only at the very end of a potentially long trajectory, and it measures only the final answer correctness, not intermediate reasoning quality.

In this setting, random exploration would almost never produce a correct trajectory. The probability of guessing the right tool, the right arguments, the right sequence, and the right final answer by chance is effectively zero. The optimization signal vanishes, and the policy either collapses to producing no tool calls (achieving the base model's poor spatial reasoning accuracy) or produces random tool calls that never lead to correct answers.

The Phase 1 SFT initialization circumvents this: it provides the model with a policy that already produces coherent tool-use trajectories with non-trivial probability. The exploration phase then needs only to refine this policy — discovering, for example, that switching from point1 to point2 when the first detection seems inaccurate improves accuracy, or that when the grasp generator fails, the model can fall back to visually estimating the grasp pose. These refinements are within the reach of RL exploration because the base policy is already functioning.

The paper's ablation in Table 4 quantifies the importance of both teachers. Removing the IRL-trained teacher (keeping only the universal teacher's trajectories for SFT, then running Phase 2 IRL) drops mean performance from 52.48 to 41.68 — a loss of 10.8 points, with particularly severe degradation on RefSpatial (29.60 vs. 53.07). This confirms that the grounding expertise from the single-tool IRL teacher is critical. Removing the universal teacher (keeping only the IRL teacher's pointing trajectories, then Phase 2 IRL) drops mean performance to 42.86, with pose estimation collapsing to 8.92. This confirms that the multi-tool coordination patterns from the universal teacher are essential for learning to chain tools for complex tasks.

Reward Functions: Task-Specific Metrics for RL Optimization

The reward functions translate the correctness of the model's final answer into a scalar signal that GRPO optimizes. Because spatial reasoning encompasses qualitatively different answer types — multiple-choice selections, 2D coordinates, 3D pose estimates, grasp configurations — the paper designs distinct, normalized reward functions for each task category. All rewards are designed to fall in the range $[0, 1]$ to provide consistent scaling across tasks.

Multiple choice reward (binary). For yes/no questions and multiple-choice spatial reasoning:

$R_B = \begin{cases} 1, & \text{if } A_{\text{final}} \text{ is correct} \\ 0, & \text{otherwise} \end{cases}$

What it computes: a simple binary indicator of answer correctness. The model either gets the answer right (1.0) or wrong (0.0), with no partial credit.

Why this form: multiple-choice questions have a discrete, verifiable ground truth. Partial credit is not meaningful because being "almost right" (choosing the wrong option) is no better than being completely wrong.

2D bounding box reward (Mean IoU). For tasks requiring the model to predict one or more 2D bounding boxes:

$R_{\text{MIoU}} = \frac{1}{N} \sum_{i=1}^N \max_j \text{IoU}(\hat{B}_i, B_j)$

where $N$ is the number of predicted boxes, $\hat{B}_i$ is the $i$ -th predicted box, $B_j$ is the $j$ -th ground-truth box, and $\text{IoU}(\hat{B}_i, B_j)$ is the Intersection-over-Union between the two boxes — the area of intersection divided by the area of union, ranging from 0 (no overlap) to 1 (perfect match).

What it computes: For each predicted box, it finds the ground-truth box with maximum IoU (to handle cases where the order of box predictions does not match the ground-truth ordering), then averages these maximum IoU scores across all predictions. The result is a value in $[0, 1]$ that measures how well the predicted box locations and extents match the ground truth.

Why this form: IoU is the standard metric for bounding box quality in object detection because it jointly penalizes errors in position, scale, and aspect ratio. The $\max_j$ operation makes the reward invariant to the ordering of predictions, which is necessary because the VLM might list boxes in an arbitrary order.

Pointing reward (Normalized Negative Distance to Centroid, NNDC). For tasks requiring the model to predict a single 2D point (e.g., "point to the vacant area"):

$R_{\text{NNDC}} = \frac{\exp(-5d) - \exp(-5\sqrt{2})}{1 - \exp(-5\sqrt{2})}$

where $d$ is the Euclidean distance from the predicted point to the centroid of the ground-truth target region. The distance $d$ is measured in normalized image coordinates (range $[0, 1] \times [0, 1]$ ), so the maximum possible distance is $\sqrt{2}$ (diagonal of the unit square).

What it computes: This formula maps distance $d$ to a score in $[0, 1]$ through an exponential transformation. When the predicted point is exactly at the target centroid ( $d = 0$ ), the numerator becomes $1 - \exp(-5\sqrt{2})$ , so $R_{\text{NNDC}} = 1$ . When the predicted point is at the maximum possible distance ( $d = \sqrt{2}$ ), the numerator is zero, so $R_{\text{NNDC}} = 0$ . The factor 5 in the exponent controls how sharply the reward decays with distance.

To emphasize precision, the paper clips this reward with the binary accuracy term:

$R = \max(R_{\text{NNDC}}, R_B)$

where $R_B$ is 1 if the predicted point lies within the ground-truth convex hull (the polygon defined by the target region's annotated points) and 0 otherwise. This clipping ensures that points inside the target region always receive a reward of at least 1.0, while points outside still receive partial credit based on their distance to the centroid.

Why the exponential transformation? A linear distance-based reward (e.g., $1 - d/\sqrt{2}$ ) would give too much credit to points that are moderately far from the target but still somewhat close. The exponential transformation creates a steeper gradient near the target, encouraging the model to be precise. The clipping with binary accuracy ensures that once the model gets inside the target region, it receives full credit — there is no incentive to optimize for the exact centroid at the expense of other behaviors.

Why NNDC over alternatives? The paper's ablation in Table 10 (Appendix E.2) compares NNDC against several alternative pointing rewards:

NSDH (Normalized Signed Distance to Hull): scores based on signed distance to the convex hull boundary, with points inside the hull receiving higher scores. Achieves only 21.31% accuracy vs. 35.25% for NNDC.
NAC (Normalized Area Change): scores based on how much the convex hull area increases when adding the predicted point, with smaller increases (points inside or near the hull) receiving higher scores. Achieves 22.95%.
Binary: simple inside/outside check. Achieves only 15.57%.

NNDC substantially outperforms alternatives because it provides a smooth reward gradient that guides the model toward the target region even when it starts far away, while the binary clipping ensures that precise localization within the region is rewarded.

Pose estimation reward (Convex Hull IoU). For tasks requiring 3D pose prediction (8 cuboid corners):

$R_{\text{IoU}} = \begin{cases} \text{IoU}(\hat{C}, C), & \text{if } |\hat{C}| = |C| = 8 \\ 0, & \text{otherwise} \end{cases}$

where $\hat{C}$ and $C$ are the sets of predicted and ground-truth 2D projected corners (8 points each, in normalized image coordinates), and $\text{IoU}(\hat{C}, C)$ is the Intersection-over-Union of the convex hulls of these two point sets. A convex hull of 8 3D box corners projected to 2D is typically an octagon (or fewer vertices if some corners are occluded or collinear in projection).

What it computes: The 3D pose is converted to a set of 8 2D projected corners, convex hulls are computed for both predicted and ground-truth corner sets, and the IoU of these convex hulls is computed. The result measures how well the predicted 2D projection of the 3D bounding box overlaps with the ground-truth projection.

Why convex hull IoU rather than corner-wise distance? Direct corner-to-corner distance would require establishing correspondence between predicted and ground-truth corners, which is ambiguous (which predicted corner corresponds to which ground-truth corner?). The convex hull formulation is permutation-invariant and captures the overall quality of the pose estimate in image space. However, it is important to note that convex hull IoU does not fully capture 3D pose accuracy — two different 3D poses can project to identical or very similar 2D convex hulls, so this metric is a necessary but not sufficient measure of 3D pose correctness.

Grasp estimation reward (Normalized Negative Coordinate Error, NNCE). For tasks requiring the model to predict grasp contact points:

$R_{\text{NNCE}} = 1 - \frac{1}{\delta_{\max}} \min\left( \delta_{\max}, \frac{1}{N} \sum_{i=1}^N \frac{\|\hat{p}_i - p_i\|_2}{w} \right)$

where $\hat{p}_i$ and $p_i$ are the predicted and ground-truth $i$ -th grasp contact points, $w$ is the gripper width (used for scale normalization), $N = 5$ is the number of reference points (grasp center, left/right finger bases, left/right finger tips), and $\delta_{\max} = 10$ caps extreme errors.

What it computes: For each of the 5 grasp keypoints, it computes the Euclidean distance between the predicted and ground-truth locations, normalizes by the gripper width $w$ (so that errors are measured in units of the gripper's physical scale), and averages across all 5 points. This average normalized error is then capped at $\delta_{\max}$ to prevent extreme outliers from dominating the reward, and the result is mapped to $[0, 1]$ by the linear transformation $1 - \text{error}/\delta_{\max}$ . A perfect prediction (all 5 points exactly correct) gives $R_{\text{NNCE}} = 1$ ; predictions with average normalized error at or above $\delta_{\max}$ give $R_{\text{NNCE}} \leq 0$ .

Why gripper-width normalization? The absolute distance error in pixels depends on the image resolution and the object's distance from the camera. Normalizing by gripper width makes the error physically meaningful — an error of 0.1 gripper widths is small regardless of image scale. This normalization allows the reward to be consistent across images taken at different distances.

Why cap extreme errors? Without capping, a single rollout with a catastrophically bad grasp prediction (e.g., predicting points on the wrong side of the image) could have such a large error that it dominates the group statistics in GRPO, distorting the advantage computation and destabilizing training.

Format score (not used in final training). The paper experimented with a format reward $R_{\text{format}} \in \{0, 1\}$ that verifies structural correctness of the model's output: that every <tool_call> tag is preceded by a thinking tag, that exactly one <answer> tag appears at the end of the response, and that the output follows the required XML structure. The final reward would be $R_{\text{final}} = R_{\text{acc}} + \lambda R_{\text{format}}$ with $\lambda = 0.3$ . However, the paper found that this format reward "provided no measurable improvement and excluded it from final training" (Section 4.3) — the model learned the correct format from SFT alone, and adding a format reward did not improve task accuracy.

Why task-specific reward normalization matters. All rewards are normalized to $[0, 1]$ to ensure that tasks with different inherent difficulty levels contribute equally to the policy gradient. Without normalization, a task with high-magnitude rewards (e.g., grasp estimation with NNCE that can reach large negative values when uncapped) would dominate the gradient updates, causing the model to overfit to that task at the expense of others. The paper's ablation in Table 10 (Appendix E.2) demonstrates that removing normalization from the NNDC reward causes accuracy to collapse to 0% — the model receives no useful gradient signal when rewards are on an ill-conditioned scale.

Summary of Key Design Choices and Their Justifications

Two-phase training (SFT then RL) rather than RL from scratch: prevents exploration collapse in the large multi-tool action space; the SFT phase provides a policy that already produces coherent tool-use trajectories with non-trivial success probability, giving RL a tractable starting point for refinement.
Two complementary teachers rather than a single teacher: the IRL-trained pointing specialist provides depth in spatial grounding (critical for pointing-dependent tasks like RefSpatial), while the frontier model provides breadth in multi-tool coordination patterns (critical for tool-chaining tasks like pose estimation). Removing either teacher causes substantial performance degradation on the tasks that depend on its expertise.
GRPO rather than PPO: eliminates the need for a critic network, reducing memory requirements by approximately 30–40% — a critical consideration when training alongside a distributed tool infrastructure on separate GPUs. The group-relative advantage computation also provides automatic reward scaling across tasks of varying difficulty.
Freezing the vision encoder: the visual perception capabilities are delegated to external tools; the VLM only needs to interpret tool outputs and decide which tools to call. Freezing the vision encoder reduces training cost, prevents catastrophic forgetting, and reinforces the architectural separation between general visual understanding (in the VLM) and precise geometric perception (in the tools).
Toolshed as decoupled infrastructure: running tools synchronously within the training loop would make interactive RL impractically slow due to blocking tool calls and GPU memory contention. Decoupling tool execution onto separate GPU resources with asynchronous, parallel workers enables the high throughput of RL rollouts needed for convergence.
Normalized, task-specific rewards: different answer types (binary, continuous coordinates, pose configurations) require different reward formulations that capture the geometric nature of the error. Normalizing all rewards to $[0, 1]$ ensures consistent gradient scales across tasks and prevents any single task from dominating the multi-task RL optimization.

4. Key Insights and Innovations

Innovation 1: Two-Stage Double Interactive RL as a Decomposition of the Combinatorial Multi-Tool Learning Problem

The field of tool-augmented VLMs faced a sharp tension before this work. On one hand, single-tool interactive RL had been demonstrated as viable — ViGoRL showed that a VLM could learn grounded reasoning with a cropping tool through trial-and-error with reward feedback. On the other hand, training-free tool orchestration approaches (Visual Programming, Set-of-Mark prompting, SpatialPIN, APC) could coordinate multiple tools but relied entirely on the model's pre-existing reasoning capabilities, with no mechanism to learn from tool failures or discover improved strategies. The gap was clear: no one had shown how to make a VLM learn to coordinate many tools through interaction, because the combinatorial action space made naive RL exploration collapse.

This paper's fundamental conceptual move is recognizing that the multi-tool learning problem can be factorized into progressive phases where each phase has a tractable exploration problem. This is not merely a curriculum learning trick — it is a structural insight about what makes different aspects of tool coordination learnable through different mechanisms.

The decomposition works because different teachers provide orthogonal forms of supervision that address distinct sub-problems:

The single-tool IRL teacher addresses spatial grounding. Learning to point at objects in response to natural language queries is itself a non-trivial RL problem, but it is tractable because the action space is small (one tool, binary or coordinate answers), the reward signal is dense (distance-based metrics provide partial credit, not just binary success/failure), and the behavior is composable — pointing is a prerequisite for nearly all downstream tool use. The paper demonstrates in Table 8 (Appendix E.1) that this single-tool IRL training not only achieves 72.3% on RoboSpatial but generalizes to unseen tasks — reaching 34.3% on RefSpatial while all other fine-tuning approaches score zero. This generalization is striking: it means the model has learned something transferable about spatial reasoning through grounding, not just memorized task-specific patterns.
The frontier model teacher addresses multi-tool coordination patterns. Claude Sonnet 4.5 does not need to be good at spatial reasoning per se — it needs to be good at following instructions and producing coherent multi-step plans. When given access to precise tools through Toolshed, it can generate trajectories showing how to chain pointing → segmentation → depth estimation → 3D bounding box fitting → grasp generation for a complex query. The paper retains only correct trajectories, so the student sees what successful multi-tool coordination looks like without being exposed to the frontier model's spatial reasoning failures.
The second IRL phase addresses robustness and strategy refinement. With a strong initialization from the teaching phase, RL exploration can discover behaviors that neither teacher demonstrated: error recovery (switching from point1 to point2 when the first detection seems inaccurate, as shown in Figure 3's relative depth example), tool fallback (estimating grasp poses manually when the grasp generator fails, as shown in Figure 3's grasp example), and adaptive tool selection (using simpler tools for easy queries and reserving complex tool chains for hard ones).

Why this is fundamental rather than incremental: Prior work treated tool-use learning as a monolithic problem — either you supervise it (SFT on traces) or you explore it (RL), but not both in a staged decomposition. The ablation in Table 4 demonstrates that neither approach alone works: pure SFT (Tool SFT) achieves 39.19 mean score; pure RL from scratch (Direct IRL All., Table 9) achieves 19.79 mean score and essentially fails to learn anything. The combination through DIRL achieves 52.48. The gap is not additive — it is synergistic, because the SFT phase provides what RL cannot discover (global coordination patterns) and the RL phase provides what SFT cannot teach (robustness to real tool behavior).

The paper also provides negative evidence that reinforces this decomposition insight. The ReST^EM experiment mentioned in the paper's discussion of revision models (from the reference example — not part of this paper) shows that attempting to further optimize with RL-style training can backfire when the base policy is not sufficiently robust. Similarly, the failure of Direct IRL All. in Table 9 shows that starting RL from an untrained policy in a large action space does not just converge slowly — it collapses, achieving near-zero performance. These results jointly argue that the teaching-exploration decomposition is not just beneficial but necessary for multi-tool RL to work at all.

The intellectual contribution is the identification of which sub-problems are learnable through which mechanisms. Single-tool grounding is learnable through RL because the action space is small and the reward is informative. Multi-tool coordination patterns are learnable through imitation because they can be demonstrated by a capable (if spatially imprecise) frontier model. Robustness to tool noise and adaptive strategy selection are learnable through RL once the model has basic competence. This triage of learning mechanisms is the paper's key conceptual contribution to the tool-augmented VLM literature.

Innovation 2: Toolshed as a Systems Contribution That Enables a Learning Contribution

It is tempting to dismiss Toolshed as engineering infrastructure — necessary for the experiments but not intellectually distinctive. This would be a mistake. Toolshed embodies a specific architectural thesis about the relationship between VLM training and tool execution that changes how one should think about building tool-augmented learning systems.

The architectural thesis: tool execution must be decoupled from policy inference at the systems level to enable interactive learning at the algorithmic level. This is a stronger claim than "we built a distributed system to make things faster." In prior work, the tight coupling between VLM inference and tool execution was not just an implementation inconvenience — it was a binding constraint on what learning algorithms were possible. ViGoRL could use interactive RL with a cropping tool precisely because cropping is fast enough to run synchronously within a training loop without stalling the batch. Scaling to SAM2 segmentation (~500ms per call on a high-end GPU), DepthPro depth estimation (similar latency), or GraspGen grasp generation (which involves iterative optimization over point clouds) would be impossible in a synchronous setup — the VLM training GPU would spend the majority of its time idle waiting for tool results.

The paper recognized that this systems constraint had shaped the research agenda: the field had converged on simple, fast tools (cropping, web search, calculator calls) not because those were the most useful tools for spatial reasoning, but because they were the only tools that could be integrated into interactive training loops. Toolshed breaks this constraint, and in doing so expands the space of learnable tool-use behaviors.

The paper provides concrete evidence for this thesis in the tool API design (Appendix B.5). Toolshed supports tools that return not just text but structured data — segmentation masks as boolean arrays, depth maps as floating-point arrays, point clouds as 3D coordinate sets, and grasp poses as 4×4 transformation matrices. These are dense, high-dimensional outputs that the VLM cannot directly ingest as text tokens. Instead, Toolshed stores them as named variables ($segmentation_mask, $point_cloud, $focal_length_px) that subsequent tool calls can reference. This variable-passing mechanism creates a dataflow graph across tool calls that the VLM learns to orchestrate — the model decides what information to extract from each tool output and how to route it to downstream tools.

Why this is more than engineering: The variable-passing design choice reflects a deeper hypothesis about what the VLM should learn. The model is not learning to interpret raw depth maps or point clouds (which would require architectural modifications to the VLM's visual encoder). Instead, it is learning to coordinate tools that exchange structured spatial representations — treating the tools as a distributed perception pipeline where information flows through typed variables rather than through the VLM's internal representations. This is a fundamentally different model of tool use than the text-in/text-out paradigm that dominates LLM tool-use research.

The paper validates this design through the robot manipulation experiments (Section 5.3, Table 3). When GPT-5 and Claude Sonnet 4.5 are connected to the same tools through Toolshed, they achieve non-trivial pick-and-place performance (65% and 79% partial success, respectively), but the paper reports qualitative failure modes that reveal the limits of training-free tool use: "GPT-5 fails to chain tools coherently, sometimes inventing grasp poses or camera intrinsics instead of reusing computed values." SpaceTools, trained with real tool interactions during RL, achieves 86% partial success and does not exhibit these hallucination failures — it has learned the dataflow dependencies between tools through experience.

The significance beyond this paper: Toolshed establishes a design pattern for future multi-tool VLM training systems: (1) host tools as independent services with isolated resources and environments; (2) support asynchronous, parallel execution to maintain training throughput; (3) provide typed variable passing between tools to enable compositional perception pipelines; (4) expose real tool behavior (including failures) during RL to teach robustness. This pattern is likely to generalize beyond spatial reasoning to any domain where tools are compute-intensive and produce structured outputs — medical image analysis, scientific simulation, CAD manipulation, and others.

Innovation 3: The Pointing-First Heuristic as an Emergent Decomposition of Spatial Reasoning

The paper's decision to use a pointing-specialist as one of the two teachers — and the empirical finding that pointing skill generalizes across tasks — reveals a structural property of spatial reasoning that the paper identifies but does not fully theorize: most multi-step spatial reasoning tasks decompose into a locate-then-analyze pattern, where precise 2D grounding is the critical prerequisite for all downstream geometric operations.

This is not stated as an innovation in the paper's contribution list, but it is the empirical discovery that makes the two-teacher approach work. The paper shows that a model trained via IRL only on pointing tasks (using the RoboSpatial dataset) generalizes to RefSpatial at 34.3% accuracy — while all tool-free fine-tuning approaches score 0% on RefSpatial (Table 8, Appendix E.1). This cross-task transfer is remarkable because RoboSpatial and RefSpatial test different spatial reasoning skills (spatial VQA and vacant space pointing vs. referring expression comprehension and placement reasoning), yet the pointing skill transfers.

Why this generalization occurs: The paper's qualitative examples (Figure 3) reveal the mechanism. In the spatial compatibility task ("Can the tissue box fit left of the bowl?"), the model's reasoning is: (1) locate the tissue box → point1 returns (0.306, 0.531); (2) locate the bowl → point1 returns (0.452, 0.92); (3) reason about spatial relationships using these grounded locations. In the relative depth task, the model again starts by locating both annotated points before querying depth values at those locations. In the pose estimation task, the model locates the target object, segments it, and then fits a 3D bounding box — the initial pointing step is always first.

The locate-then-analyze pattern is not hardcoded — it emerges from the training process because it is reward-optimal. On pointing tasks, the model learns that precise localization produces higher rewards. On multi-tool tasks, the model learns that calling pointing first produces better segmentations, better depth queries, and ultimately better answers. The behavior is reinforced at both the single-tool and multi-tool levels, creating a consistent strategy.

Contrast with prior work: Previous spatial reasoning approaches either (a) bypassed explicit grounding entirely, relying on the VLM's internal spatial representations (tool-free SFT and RL baselines, which achieve 0% on RefSpatial's pointing-dependent tasks), or (b) used hand-crafted pipelines that hardcoded the pointing-first step (SpatialPIN, APC), preventing the model from learning when pointing is unnecessary or when alternative localization strategies would work better.

The paper's approach is distinctive because the pointing-first behavior is learned rather than programmed. SpaceTools sometimes deviates from it — for example, when it detects that a pointing tool's output is unreliable and switches to an alternative tool, or when it estimates a grasp pose manually after the grasp generator fails. These deviations are only possible because the pointing-first strategy is soft (a learned preference reinforced by reward) rather than hard (a fixed pipeline step). The model has learned that pointing is usually the right first step, but also when to override that heuristic.

Why this matters beyond the paper: The locate-then-analyze decomposition suggests a general principle for tool-augmented perception systems: train specialized models for foundational perceptual operations (localization, segmentation) that serve as building blocks for more complex reasoning, then train composition models to chain these building blocks. The fact that pointing skill transfers across tasks implies that improvements to the pointing tool — or to the model's ability to use it — would improve performance across all downstream spatial reasoning tasks without retraining the downstream reasoning components. This is the modularity argument that the paper makes in its conclusion, supported by empirical evidence of cross-task transfer.

Innovation 4: Verifier-Free RL with Geometric Reward Functions as an Alternative to Outcome Verification

A dominant paradigm in RL for reasoning is to use a verifier — a learned model or a symbolic checker that evaluates the correctness of intermediate reasoning steps or final answers. DeepSeek-R1 and related works use outcome reward models or formal verifiers to provide dense reward signals during RL training. This paper takes a different approach: it uses hand-designed, task-specific geometric reward functions that directly measure the spatial accuracy of the model's output without requiring a separate learned verifier.

The reward functions are the verifier. The NNDC pointing reward, the convex hull IoU pose reward, the NNCE grasp reward, and the MIoU bounding box reward are not learned — they are mathematical formulas that compare predicted spatial coordinates to ground-truth annotations. They provide dense, continuous reward signals (not just binary correct/incorrect) that capture the geometric nature of the error: a pointing prediction that is 5 pixels from the target gets higher reward than one that is 50 pixels away, even if both are "technically" wrong by a binary criterion.

Why this is distinctive: In text-based reasoning domains (math, coding), verifying correctness often requires either a formal proof system (limited to formal domains) or a learned reward model (which introduces its own errors and over-optimization risks). Spatial reasoning has a unique property that the paper exploits: correctness is geometrically grounded and can be measured through coordinate comparison. This means the reward signal is inherently reliable — it comes from the dataset annotations, not from a fallible neural network — and naturally provides partial credit, which smooths the RL optimization landscape.

The paper's ablation in Table 10 (Appendix E.2) demonstrates the importance of this continuous reward design. When the pointing reward is replaced with a binary inside/outside check, accuracy drops from 35.25% (NNDC) to 15.57% (Binary). The continuous reward provides a gradient that guides the model toward the target region even when it starts far away; the binary reward provides no signal until the model happens to land inside the target by chance. In a large action space, that chance is effectively zero, and RL fails to learn.

The trade-off the paper makes explicit: Hand-designed geometric rewards are powerful because they are reliable and informative, but they are task-specific — the NNDC formula works for pointing but not for pose estimation; the convex hull IoU works for pose but not for grasp prediction. This means scaling to new task types requires designing new reward functions, which requires domain expertise. The paper does not claim this approach generalizes to arbitrary spatial reasoning tasks — it demonstrates it on a specific, well-defined set of task types with known geometric structure.

This stands in contrast to learned verifiers, which can generalize across task types but introduce their own training complexity and over-optimization risks. The paper's choice reflects a pragmatic judgment: for the spatial reasoning benchmarks studied, geometric reward functions are more reliable than learned verifiers, and the task diversity is small enough that designing multiple reward functions is tractable. This is a methodological contribution — it identifies a regime where hand-designed rewards outperform learned verifiers — not a claim about universal superiority.

Innovation 5: Tool-Augmented Spatial Reasoning as a Scalable Alternative to Capability-Specific Fine-Tuning

Perhaps the paper's most ambitious conceptual claim is architectural rather than algorithmic: VLMs should not learn to perceive; they should learn to orchestrate perception tools. This reframes the entire spatial VLM research agenda from "how do we make VLMs better at spatial reasoning?" to "what spatial reasoning capabilities should live in the VLM versus in external tools?"

The conventional approach the paper rejects: SpatialVLM, RoboPoint, SpatialRGPT, and similar works take a base VLM and fine-tune it on spatial reasoning datasets — sometimes with architectural modifications to output dense predictions (depth maps, segmentation masks, bounding boxes) from language model heads. This approach bakes perceptual capabilities directly into the model's weights. Each new capability requires new training data, and improvements to the underlying perception models (e.g., a better depth estimator) require retraining the VLM to benefit from them.

The alternative the paper demonstrates: Keep the VLM as a reasoning and coordination engine, and delegate precise perception to specialized external tools. The VLM learns when depth information is useful and how to query it, but not how to estimate depth from pixels. This means that if DepthPro is replaced with a better monocular depth estimator, Toolshed can swap it in without retraining SpaceTools — the model's tool-use strategies remain valid as long as the tool API is consistent.

Evidence for scalability: The paper's ablation in Table 4 provides indirect support for this claim. When the universal teacher is removed from the training data (the "w/o Univ. Teacher" variant), performance on tasks requiring multi-tool composition (pose estimation) collapses from 34.37 to 8.92 — but performance on pointing-centric tasks (RefSpatial) remains high at 54.51 vs. 53.07. This suggests that the model's competence separates into two components: grounding skill (learned from the IRL teacher, transfers across tasks) and tool coordination patterns (learned from the universal teacher, specific to tool chains). The grounding skill would persist even if the downstream tools changed; only the coordination patterns would need updating.

Why this is a reframing rather than just a method: The paper is arguing for a division of labor in embodied AI systems that mirrors the division in human cognition: we do not compute depth maps in our brains to judge distances — we use visual cues, but we can also use tools (rulers, laser rangefinders) when precision matters. VLMs, the paper suggests, should similarly operate at the level of deciding when precision matters and which tool to invoke, rather than attempting to internalize all perceptual capabilities.

This reframing has implications beyond spatial reasoning. If the tool-orchestration model extends to other domains — medical imaging (where specialized segmentation models outperform general VLMs), scientific computation (where numerical solvers are more reliable than neural networks), or code execution — then the research agenda shifts from "how do we make larger, more capable foundation models?" to "how do we make foundation models better at using specialized tools?" The paper provides a concrete training methodology (DIRL) and infrastructure (Toolshed) for pursuing this agenda, but the architectural insight is broader than either component.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses a combination of existing benchmarks and a custom-augmented dataset. For spatial reasoning evaluation, the benchmarks are: RoboSpatial-Home (spatial VQA and vacant space pointing, from Song et al., 2025), BLINK (relative depth, from Fu et al., 2024), RefSpatial (placement, location, and unseen referring expressions, from Zhou et al., 2025), CVBench (2D relations and 3D relative depth, from the CVBench team, 2025), and BOP-ASK (object pose estimation and grasp affordance prediction, from Bhat et al., 2025). For robot manipulation evaluation, the authors augment the HOPE dataset (Tyree et al., 2022) with grasping and pick-and-place control tasks. The training SFT dataset is constructed from 8k high-quality tool-use trajectories: 6k from the universal teacher (Claude Sonnet 4.5) and 2k from the IRL-trained teacher, with image-question pairs sampled from RoboSpatial, RefSpatial, and BOP-ASK. The same spatial reasoning image-question pairs used in the SFT dataset are also used in the Phase 2 IRL. Exact test set sizes per benchmark are not specified, though the RoboSpatial test set is described as having multiple question types (VQA and vacant space localization).
Base model. All experiments use Qwen2.5-VL-3B-Instruct (Bai et al., 2025) as the base VLM. The paper argues this model is representative of current open-source multimodal foundations and operates in a useful regime — capable of general visual understanding but lacking spatial reasoning specialization, with sufficient capacity (3B parameters) to learn tool coordination while being small enough that tool-augmented training is computationally tractable. The vision encoder and projector are frozen throughout all training phases; only the language model component (2.55B parameters) is trainable. For the FLOPs-matched comparisons in Table 2, the paper also evaluates against Qwen2.5-VL-32B and several proprietary models at larger scales, but the core training experiments are conducted solely on the 3B model.
Metrics. The paper uses task-specific normalized accuracy metrics, all mapped to a [0, 1] or [0, 100] range: (1) Answer accuracy for multiple-choice and pointing questions (binary correct/incorrect). (2) Mean IoU (MIoU) for 2D bounding box prediction: the average across predicted boxes of the maximum Intersection-over-Union with any ground-truth box, expressed as a percentage. (3) Normalized Negative Distance to Centroid (NNDC) for single-point spatial prediction: an exponential function mapping Euclidean distance from the predicted point to the target region centroid into [0, 1], clipped with binary accuracy so that points inside the ground-truth convex hull always receive a reward of at least 1.0. (4) Convex Hull IoU for pose estimation: the Intersection-over-Union between convex hulls of the predicted and ground-truth 8 projected 2D corners, normalized to [0, 100]. (5) Mean Angular Coordinate Error (MACE) for grasp estimation: a composite metric that jointly scores grasp center location error (normalized by gripper width) and finger orientation error (cosine similarity between predicted and ground-truth direction vectors), normalized to [0, 100], plus a Success Rate (SR) reporting the percentage of grasps achieving MACE > 40. For robot manipulation, the metric is success rate for Pick and Relational Pick tasks and partial success rate for Pick & Place (1 point each for correct pick and correct place).
Baselines. The paper compares against four categories: (1) Proprietary models: Claude Sonnet 4.5 (Anthropic, 2025), GPT-4o (OpenAI, 2024), GPT-5 (OpenAI, 2025), and Gemini-ER 1.5 (Gemini Robotics Team, 2025). (2) General open-source models: LLaVA-NeXT-8B (Liu et al., 2024) and Qwen2.5-VL-32B (Bai et al., 2025). (3) Spatial VLMs: SpaceLLaVA-13B (Chen et al., 2024), RoboPoint-13B (Yuan et al., 2024), Molmo-7B (Deitke et al., 2025), RoboBrain2.0-7B (Team, 2025), and RoboRefer-8B-SFT (Zhou et al., 2025). (4) Tool-free fine-tuning baselines using the same base model (Qwen2.5-VL-3B): a Tool-free SFT variant trained on the same 8k source question-answer pairs from DIRL's Stage 1 but without tool calls, and a Tool-free RL variant applying reasoning RL à la DeepSeek-R1 (DeepSeek-AI, 2025) without tool use. For robot manipulation, the paper adds π0.5 (Black et al., 2025) as a vision-language-action baseline and evaluates GPT-5 and Claude Sonnet 4.5 with Toolshed connected for zero-shot tool use.
Generation budget / compute accounting. The paper does not report a standardized "generation budget" in the way that inference-scaling papers typically do (e.g., number of sampled solutions). Instead, compute is implicitly measured through the training recipe: the number of tool-use trajectories in the SFT dataset (8k), the number of RL rollouts per input (N=5 for GRPO), and the training infrastructure (8 GPUs for VLM + 8 GPUs for tools, specified in Table 6). For inference-time evaluation, all models are evaluated with the same structured conversation format and tool access (when applicable), making comparisons at equal tool availability. The paper does not, however, account for inference-time tool execution cost in its benchmark comparisons — a model calling SAM2, DepthPro, and GraspGen in sequence incurs substantially more wall-clock computation than a model answering directly from internal representations. All reported accuracy numbers are at equal tool access, not at equal FLOPs or latency.
Cross-validation / statistical protocol. For robotic manipulation experiments, the paper reports per-task success rates with exact counts (e.g., "86 (6/7)" for pick tasks, Table 3) across 7 trials per Pick task, 6 trials per Relational Pick task, and 14 trials per Pick & Place task. For benchmark evaluations, the paper does not describe cross-validation — models are evaluated once on the standard test splits. The ablation study in Table 4 reports mean scores across three benchmarks (RoboSpatial, RefSpatial, Pose) without confidence intervals or standard deviations, meaning the statistical reliability of the observed differences (e.g., 52.48 vs. 50.99 between full DIRL and w/o Stage 2 IRL) cannot be assessed. The pointing reward ablation in Table 10 (Appendix E.2) is conducted on a subset of 1k vacant-space questions, and the data composition ablation in Table 11 (Appendix E.2) reports single-run accuracy numbers without statistical quantification. This is a notable limitation — with test sets of unspecified size (though RoboSpatial likely has hundreds of questions based on the paper's description), differences of 1–2 percentage points may not be statistically significant.

Main Quantitative Results

Spatial Reasoning Benchmark Performance

The headline result is that SpaceTools achieves state-of-the-art performance across nearly all spatial reasoning benchmarks, outperforming the strongest proprietary and specialized spatial VLM baselines (Table 2). The key comparisons are:

Versus proprietary models. SpaceTools-3B outperforms the best proprietary baseline (Gemini-ER 1.5) by +7.5% on RoboSpatial overall accuracy (70.00% vs. 62.50%). On pose estimation, it exceeds Claude Sonnet 4.5 by +24.4 percentage points (34.37% vs. 7.49%). On grasp estimation, it surpasses GPT-5 by +8.3 percentage points on MACE (43.06% vs. 39.59%) and by +8.33 points on Success Rate (50.00% vs. 41.67%). On RefSpatial, it achieves 53.07% compared to GPT-5's 23.10% — a +30.0 point gap. The only benchmark where SpaceTools does not lead is BLINK, where Gemini-ER 1.5 achieves 31.10% vs. SpaceTools' 52.46% (SpaceTools leads by 21.36 points). In fact, SpaceTools leads on every single benchmark in Table 2.

Versus spatial VLMs. Compared to RoboRefer-8B-SFT (the strongest spatial VLM baseline), SpaceTools achieves higher accuracy on RoboSpatial (70.00% vs. 59.43%, +10.57 points) and on pose estimation (34.37% vs. 48.37% — RoboRefer leads by 14 points). Note: RoboRefer-8B-SFT achieves 48.37% on pose while SpaceTools achieves 34.37%, making this the one benchmark where a specialized spatial VLM outperforms SpaceTools. On RefSpatial, RoboRefer achieves 88.71% vs. SpaceTools' 90.32% (SpaceTools leads by +1.61 points). On CVBench, RoboRefer achieves 96.31% vs. 94.92% (RoboRefer leads by 1.39 points on Depth). The comparison is nuanced: SpaceTools matches or exceeds RoboRefer on some dimensions but trails on CVBench depth and pose.

Versus tool-free fine-tuning baselines (same base model, same training data). This is the most controlled comparison because it isolates the effect of tool-augmented training. SpaceTools-3B achieves +12% higher accuracy on RoboSpatial than Tool-free SFT (70.00% vs. 58.00%) and +16% higher than Tool-free RL (70.00% vs. 54.00%). On RefSpatial, SpaceTools achieves 53.07% while both tool-free baselines score near zero (2.44% for SFT, 12.00% for RL). On grasp estimation, SpaceTools achieves 43.06% MACE and 50.00% SR, while tool-free baselines score 39.47%/35.00% (SFT) and 38.79%/36.67% (RL). The pattern is consistent: tool-augmented training yields substantially stronger spatial reasoning than fine-tuning the same base model on the same data without tools, regardless of whether the fine-tuning uses SFT or RL.

Proprietary models with vs. without Toolshed (Table 5). Connecting Toolshed to GPT-5 and Claude Sonnet 4.5 in a zero-shot setting (no additional training) produces mixed results that reveal both the promise and limitations of training-free tool use:

Gains on precise geometric tasks: GPT-5 with Toolshed improves RefSpatial from 23.10% to 36.10% (+13.0 points), pose estimation from 9.03% to 15.00% (+5.97 points), and grasp MACE from 39.59% to 41.49% (+1.90 points). Claude Sonnet 4.5 with Toolshed improves RefSpatial from 7.49% to 27.80% (+20.31 points), pose from 1.67% to 25.00% (+23.33 points), and grasp from 40.12% to 44.19% (+4.07 points). These gains confirm that tool feedback mitigates limitations in spatial grounding and 3D understanding for both models.
Declines on high-level reasoning: Both models show performance degradation on RoboSpatial when tools are added — GPT-5 drops from 58.39% to 55.14% (-3.25 points), and Claude drops from 57.43% to 52.86% (-4.57 points). On BLINK, GPT-5 improves from 66.13% to 90.32% (+24.19 points), but Claude drops from 78.23% to 75.00% (-3.23 points). The paper attributes these declines to models that "tend to overuse tools and struggle to correctly interpret nuanced tool outputs" (Section 6).

These results establish that tool augmentation alone is insufficient — it helps on tasks requiring explicit geometric measurement (RefSpatial, pose, grasp) but can hurt on tasks requiring holistic reasoning about tool outputs (RoboSpatial, BLINK). SpaceTools' training with real tool interactions during RL teaches the model when to trust tools, when to fall back to self-estimation, and how to interpret ambiguous outputs — skills that zero-shot tool use cannot provide.

Real-World Robot Manipulation Results

SpaceTools achieves the highest success rates across all manipulation task categories when compared to both a specialized VLA model and frontier VLMs with zero-shot tool access (Table 3).

On Pick tasks (grasping a specified object), SpaceTools achieves 86% (6/7) success, matching Claude Sonnet 4.5 + Toolshed (86%, 6/7) and substantially outperforming GPT-5 + Toolshed (71%, 5/7) and π0.5 (0%, 0/7). The π0.5 result is striking — a specialized vision-language-action model trained for robotic manipulation completely fails on this task suite, likely because spatial understanding is insufficiently grounded for the specific objects and scenes tested.

On Relational Pick tasks (e.g., "pick up the far coconut water" or "pick up the left pineapple juice can"), SpaceTools achieves 83% (5/6) — dramatically higher than Claude Sonnet 4.5 (50%, 3/6) and GPT-5 (33%, 2/6). This task category tests the model's ability to resolve spatial relations (left of, closer to camera, far) before executing a grasp. The paper reports that GPT-5 and Claude showed specific failure modes here: misidentifying which object satisfies the spatial relation, or failing to chain the relation-finding step with the grasp step.

On Pick & Place tasks (pick up an object and place it in/at a target location), evaluated by partial success rate (1 point for correct pick, 1 point for correct place, out of 2 maximum), SpaceTools achieves 86% partial success (12/14). Claude Sonnet 4.5 achieves 79% (11/14) and GPT-5 achieves 65% (9/14). π0.5 again scores 0%. SpaceTools' lead here comes primarily from Relational Pick tasks where spatial reasoning is required before action.

Time-to-First-Movement (TTFM) reveals a substantial efficiency difference: SpaceTools takes 10 seconds on average to begin moving after receiving the instruction, compared to 30 seconds for Claude Sonnet 4.5 and 36 seconds for GPT-5. The paper does not provide a breakdown of what contributes to this latency difference, but the likely explanation is that SpaceTools makes fewer redundant or corrective tool calls — having learned through RL when additional perception is needed versus when existing information is sufficient. The π0.5 model's 1-second TTFM reflects its direct perception-to-action architecture but is irrelevant given its 0% success rate.

Qualitative failure analysis (Appendix E.3). The paper provides concrete examples of failure modes. SpaceTools correctly localizes a vacant area for placement but selects a point too close to the bin boundary, causing the robot to place the object on the edge (Figure 9). This reveals a precision limitation in the model's point selection strategy — the pointing reward (NNDC) optimizes for distance to the target region centroid but does not encode boundary avoidance, and the model has not learned this through RL. On grasp estimation (Figure 12), failures arise from wrong object localization in cluttered scenes and inaccurate pose estimation, highlighting that while SpaceTools coordinates tools effectively, it remains limited by the quality of the underlying perception tools themselves.

Ablation Studies and Robustness Checks

Removing the IRL-trained teacher (single-tool specialist): Dropping the IRL teacher from the SFT dataset while keeping the universal teacher and Phase 2 IRL causes mean performance to drop from 52.48 to 41.68 (–10.8 points, Table 4). The degradation is most severe on RefSpatial (29.60% vs. 53.07%, –23.47 points), confirming that the pointing specialist's grounding expertise is critical for tasks requiring precise object localization. RoboSpatial drops from 70.00% to 61.14% (–8.86 points). Interestingly, pose estimation is essentially unaffected (34.29% vs. 34.37%), suggesting that the universal teacher's multi-tool coordination patterns are sufficient for learning pose estimation even without the IRL teacher's grounding demonstrations.

Removing the universal teacher (frontier model): Dropping the universal teacher while keeping the IRL teacher and Phase 2 IRL causes mean performance to drop from 52.48 to 42.86 (–9.62 points, Table 4). The degradation is most severe on pose estimation (8.92% vs. 34.37%, –25.45 points), confirming that the universal teacher's multi-tool coordination demonstrations are essential for learning to chain pointing → segmentation → depth → 3D bbox for pose tasks. RefSpatial actually improves slightly (54.51% vs. 53.07%, +1.44 points), suggesting that the IRL teacher's pointing expertise alone is sufficient — perhaps even more focused — for pointing-centric referring expression tasks. RoboSpatial drops from 70.00% to 65.14% (–4.86 points).

Removing Stage 2 IRL (SFT only): Training with SFT on both teachers but skipping the exploration phase drops mean performance from 52.48 to 50.99 (–1.49 points, Table 4). While this seems like a small gap, the paper notes that Stage 2 IRL provides "the final boost of tool-augmented reasoning" and that the improvement is consistent across all three benchmarks (RoboSpatial: 67.71% → 70.00%, RefSpatial: 51.98% → 53.07%, Pose: 33.28% → 34.37%). The modest magnitude of improvement raises an important question about how much the interactive RL phase contributes beyond what SFT alone achieves — the gap is substantially smaller than the gaps from removing either teacher. This suggests that while interactive RL refines tool coordination, the majority of the performance comes from the quality and diversity of the teaching demonstrations, not from RL-driven strategy discovery.

Tool SFT baseline (non-interactive): Training with SFT on the universal teacher's traces only (without the IRL teacher and without any RL) achieves a mean score of 39.19 (Table 4). This is 13.29 points below full DIRL and even 11.80 points below the "w/o Stage 2 IRL" variant that includes both teachers. This confirms that the IRL teacher's demonstrations contribute substantially even in an SFT-only setting, and that pure SFT on multi-turn tool-use traces is insufficient — the model needs either the IRL teacher's grounding depth or interactive RL experience (or both) to achieve strong performance.

Tool NIRL baseline (non-interactive RL): Training with RL where tool calls are verified against ground-truth traces rather than executed interactively achieves a mean score of 38.06 (Table 4) — even worse than Tool SFT. This is a notable negative result: non-interactive RL with ground-truth tool call supervision performs worse than simple SFT on the same teacher traces. The paper does not deeply analyze why, but the likely explanation is that the reward signal in NIRL (binary correctness of tool name and arguments) is too sparse and fails to capture the nuanced differences between functionally similar but textually different tool calls, whereas SFT provides dense token-level supervision that better preserves the teacher's reasoning patterns.

Direct IRL on all tasks with all tools (Table 9, Appendix E.2): Applying interactive RL from scratch with all tools on all tasks, without the teaching phase, achieves a mean score of only 19.79 — a catastrophic collapse compared to DIRL's 52.48. On RefSpatial, it scores 3.25%. On pose estimation, 3.26%. This is the paper's strongest evidence that naive multi-tool RL exploration fails: the model cannot discover effective tool-use strategies through random exploration in a combinatorial action space of 10+ tools, and the teaching phase is not just beneficial but necessary for learning to occur at all.

Pointing reward design (Table 10, Appendix E.2): Comparing alternative pointing reward formulations on a 1k-question subset:

NNDC (the paper's chosen reward): 35.25% accuracy
NNDC without clipping: 14.8% (binary clipping to inside-target-hull is critical — without it, the model optimizes for proximity to centroid but does not learn to stay within the target region)
NNDC without normalization: 0.00% (normalization to [0, 1] is essential — without it, the reward scale is ill-conditioned and RL receives no useful gradient signal)
NNDC with format reward: 33.61% (adding a format score slightly reduces accuracy, supporting the paper's decision to exclude it from final training)
NNDC without tool-use examples in prompt: 17.21% (few-shot examples in the prompt are essential for the model to understand the tool-calling format)

Alternative reward functions (NSDH at 21.31%, NAC at 22.95%, Binary at 15.57%) all substantially underperform NNDC, confirming that the exponential distance-to-centroid formulation provides the best gradient signal for pointing tasks. The 0% accuracy for unnormalized variants is particularly informative — it shows that RL for spatial reasoning collapses completely without careful reward scaling, even when the reward function correctly identifies better vs. worse answers.

Dataset composition for IRL (Table 11, Appendix E.2): Varying the mix of data types in the RoboSpatial training set reveals that including grounding data (2D bounding box prediction) improves performance on other task types — the "w/o Ground." variant achieves only 56.90% on RoboSpatial-Home, compared to 69.70% for variants that include grounding. This cross-task transfer suggests that learning to predict precise 2D locations improves the model's general spatial reasoning, even for tasks (like spatial VQA) that do not require explicit coordinate outputs. Increasing the total dataset size from 2k to 6k samples (All-v1 to All-v3) yields negligible additional improvement (69.70% → 69.10%), indicating that data diversity and label balance contribute more to IRL effectiveness than raw quantity.

Critical Assessment

The experiments demonstrate that SpaceTools achieves strong performance on spatial reasoning benchmarks and robotic manipulation, but the paper's central claims require careful scrutiny against what was actually tested.

Claim: "DIRL provides substantial improvements over the vanilla SFT (+12% on RoboSpatial) and RL (+16% on RoboSpatial) baselines." This claim is supported by Table 2, but the comparison requires careful interpretation. The "vanilla SFT" and "vanilla RL" baselines are tool-free — they are trained on the same 8k question-answer pairs without tool access. This means the comparison conflates two factors: (1) the effect of tool augmentation (having access to depth, segmentation, pointing during inference) and (2) the effect of DIRL training (the staged curriculum). The +12% and +16% improvements are measuring the combined effect of "tools + DIRL" versus "no tools," not DIRL versus alternative tool-training methods. A more precise claim would be: "tool-augmented spatial reasoning trained with DIRL outperforms tool-free reasoning by 12–16%." The paper's ablation in Table 4 provides the cleaner comparison: DIRL (52.48 mean) vs. Tool SFT (39.19) shows a +13.29 improvement attributable to the staged training approach (including the IRL teacher and Phase 2 RL), not to tool access alone.

Claim: "SpaceTools achieves state-of-the-art performance on spatial understanding benchmarks." Table 2 supports this claim with the caveat that SpaceTools leads on 9 out of 10 metric columns (the exception being CVBench Depth, where RoboRefer-8B-SFT leads 96.31% vs. 94.92%). However, several important qualifications apply: (1) The comparisons are not at equal compute — SpaceTools calls multiple heavy vision models (SAM2, DepthPro, GraspGen) during inference, while models like RoboRefer-8B-SFT produce answers directly from internal representations. Accuracy-per-FLOPs or accuracy-per-second comparisons are not reported. (2) The proprietary model baselines (GPT-5, Claude Sonnet 4.5, Gemini-ER 1.5) are evaluated zero-shot without tool-specific fine-tuning, while SpaceTools is trained on the same data distribution. This is a reasonable comparison for demonstrating tool-augmented training effectiveness, but it is not an apples-to-apples assessment of "which model is better at spatial reasoning." (3) Some baselines score 0% on certain benchmarks (e.g., open-source models on BOP-ASK Grasp tasks), which may reflect format mismatch rather than complete inability — the models might output answers in wrong formats that the grading function cannot parse.

Claim: "Tool-augmented training yields substantially stronger results on spatial reasoning than tool-free fine-tuning of the same base model on the same 8k VQA pairs regardless of learning technique." This claim is supported in Table 2 (tool-free SFT: 58.00% on RoboSpatial vs. SpaceTools: 70.00%; tool-free RL: 54.00% vs. 70.00%). However, the paper does not report an ablation where the tool-free models receive the tool outputs pre-computed and embedded in their context — that is, what if the tool-free models were given the same depth maps, segmentations, and point coordinates as text tokens, without learning to call the tools themselves? This ablation would disentangle whether the benefit comes from having access to spatial measurements versus learning to actively coordinate perception tools. The tool-free baselines receive only the raw image and question; they never see depth values or segmentation masks. The +12% gap might narrow substantially if tool-free models received the same spatial information in their input context.

Claim: "SpaceTools completes pick-and-place tasks with an 86% success rate, demonstrating effective transfer from spatial reasoning to embodied control and outperforming frontier models equipped with the same tools." Table 3 supports the 86% number for Pick & Place partial success. The comparison to frontier models is valid — GPT-5 and Claude Sonnet 4.5 have the same tool access as SpaceTools. However, the experiment scale is small: 7 Pick trials, 6 Relational Pick trials, and 14 Pick & Place trials (scored as 28 sub-operations). With such small sample sizes, the reported percentages have wide confidence intervals — the difference between 86% (12/14) and 79% (11/14) for Pick & Place is a single trial. The Relational Pick results (83% vs. 50% vs. 33%) are more convincing because of the larger relative gap, but still based on only 6 trials. Additionally, the π0.5 baseline achieving 0% across all tasks is a strong claim that warrants scrutiny — this might reflect distribution shift between π0.5's training data and the specific objects/scenes tested, or issues with the task specification format, rather than fundamental spatial reasoning incapability.

Missing baselines and ablations that would strengthen the paper:

Single-teacher variants with Phase 2 IRL. The paper ablates removing each teacher, but does not report what happens when you take SFT on a single teacher's data and then run Phase 2 IRL. The "w/o IRL Teacher" and "w/o Univ. Teacher" variants in Table 4 both include Phase 2 IRL — but what would their performance be without Phase 2 IRL? This would isolate whether Phase 2 IRL helps more when the SFT data is weaker.
Equal-compute comparison with specialized spatial VLMs. The paper does not report what happens when SpaceTools is limited to a fixed inference budget (e.g., "you may call at most 1 tool") or constrained to match the latency of RoboRefer or Molmo. This makes it impossible to assess whether SpaceTools' accuracy advantage comes from better spatial reasoning or simply from more computation at inference time.
Scaling the SFT dataset size. The paper uses exactly 8k trajectories (6k + 2k split) without ablating the effect of dataset size. The IRL data composition ablation in Table 11 (Appendix) shows that increasing data from 2k to 6k yields minimal gains for single-tool IRL, but there is no equivalent ablation for the multi-tool SFT setting. Would 4k trajectories (2k + 2k) achieve similar performance? Would 16k trajectories further improve? The fixed 8k choice is not justified.
Effect of freezing the vision encoder. The paper freezes the vision encoder and projector during all training phases, but does not ablate this choice. Would fine-tuning the vision encoder during RL improve or degrade performance? Freezing likely prevents catastrophic forgetting of general visual capabilities, but might also limit the model's ability to learn tool-specific visual representations that improve coordination.
Direct evaluation of tool coordination quality. The paper's metrics all measure final answer correctness, not intermediate tool-use quality. There is no metric for "did the model call the right tools in the right order?" or "did the model correctly interpret tool outputs?" This makes it hard to diagnose whether failures come from poor tool coordination or from tools providing incorrect information. The qualitative examples in Figures 3 and 10–12 provide some insight, but no systematic analysis.

Conditional nature of the claims:

The superiority of SpaceTools over specialized spatial VLMs holds most strongly on tasks requiring compositional multi-tool reasoning (pose estimation: 34.37% vs. RoboRefer's 48.37% — SpaceTools actually trails here) and pointing-based tasks (RefSpatial: 90.32% vs. 88.71%). On tasks solvable with strong 2D understanding alone (CVBench depth), specialized models can match or exceed SpaceTools.
The robot manipulation claims are conditioned on object visibility and tool reliability. The failure case in Figure 9 shows a near-miss placement due to boundary selection; the failure case in Figure 12 shows grasp failures from object misdetection. SpaceTools' 86% success rate should be understood as an upper bound under favorable perceptual conditions, not as a guarantee of robust real-world manipulation.
The benefit of interactive RL (Phase 2) over SFT alone is small in absolute terms (~1.5 mean points in Table 4) compared to the benefit of tool access (~13 points) and the benefit of having both teachers (~10 points each). This suggests that for practitioners, the most impactful decision is choosing good teachers and providing tool access; the RL refinement step provides a modest additional improvement whose cost-effectiveness depends on the computational budget available.

The paper would have been strengthened by: (1) Reporting confidence intervals or standard deviations for benchmark results, particularly given the unspecified test set sizes. (2) Running equal-latency or equal-FLOP comparisons against specialized spatial VLMs to isolate the effect of tool use from the effect of extra computation. (3) Ablating the SFT dataset size and teacher ratio to provide guidance for practitioners on data requirements. (4) Including a baseline where tool-free models receive pre-computed tool outputs as text, to disentangle "having spatial measurements" from "learning to coordinate tools." (5) Evaluating on a larger and more diverse set of robot manipulation trials to establish statistical reliability of the 86% success rate.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in Reported Efficiency

The paper's DIRL framework fundamentally depends on being able to generate high-quality teaching demonstrations from two teacher models: an IRL-trained single-tool specialist and Claude Sonnet 4.5 integrated with Toolshed. The paper does not account for the computational cost of producing these demonstrations in any of its reported performance numbers.

The paper is transparent about the scale: the teaching dataset consists of 8,000 tool-use trajectories — 6,000 from Claude Sonnet 4.5 calling the full toolset through Toolshed, and 2,000 from the IRL-trained teacher. Each trajectory is a multi-turn dialogue where the teacher model calls real vision tools (SAM2 segmentation, DepthPro depth estimation, GraspGen grasp generation), and each tool call involves GPU inference on separate hardware. The IRL teacher itself must first be trained via interactive RL with the pointing tool on 4,000 samples (Table 6), which requires its own RL training loop with Toolshed running in the background.

The consequence is that the reported benchmark numbers — SpaceTools achieving 70.00% on RoboSpatial, 53.07% on RefSpatial, 34.37% on pose estimation — represent performance after a substantial investment in teacher computation that is not amortized in the evaluation. A practitioner seeking to replicate this approach for a new domain would need to: (1) train a single-tool IRL specialist on their domain (requiring RL infrastructure and reward design), (2) pay for Claude Sonnet 4.5 API calls (or an equivalent frontier model) generating thousands of multi-turn tool-use trajectories, and (3) run the actual DIRL training. Only step (3) is counted in the paper's training cost estimates (Table 6: 8 GPUs for VLM + 8 GPUs for tools, 2 epochs).

What evidence exists in the paper: The paper never quantifies the teacher generation cost or reports what fraction of total development compute it represents. Table 6 specifies training configurations for Phase-1 IRL (4,000 samples, 5 rollouts per sample meaning ~20,000 tool-interactive episodes), Phase-1 SFT (8,000 trajectories), and Phase-2 IRL (~8,000 samples × 5 rollouts = ~40,000 interactive episodes), but the Claude Sonnet 4.5 generation cost for 6,000 trajectories — each potentially involving 3–8 tool calls to SAM2, DepthPro, and GraspGen — is not estimated. The paper also does not report how many Claude-generated trajectories were discarded to achieve the final 6,000 correct trajectories (Section 4.1 states "retaining only trajectories that lead to correct solutions"), which could multiply the generation cost if the frontier model's spatial reasoning success rate is low.

Mitigation status: The paper does not address this limitation. Section 6 and Appendix A focus on other limitations (application scope, RL methodology, infrastructure scaling) but do not discuss the teacher computation cost. The paper frames DIRL as a training methodology without considering the full lifecycle compute cost from teacher generation through deployment. A direction for future work — not stated in the paper but implied by the architecture — would be to investigate whether the teaching dataset can be made smaller, whether a single teacher suffices, or whether synthetic data generation techniques can reduce dependence on expensive frontier model API calls.

All Results Are on a Single Model Family with a Single Base Architecture

Every training experiment in the paper — the Phase-1 IRL teacher, the Phase-1 SFT, the Phase-2 IRL, and SpaceTools itself — uses Qwen2.5-VL-3B-Instruct as the base model. The paper does not train or evaluate on any other VLM architecture, size, or family (e.g., LLaVA, InstructBLIP, InternVL, or larger Qwen variants).

The paper argues that Qwen2.5-VL-3B-Instruct is "representative of current open-source multimodal foundations" (Section 5, Dataset paragraph), but this claim is unverified. Several properties of the model could influence the results in ways that do not transfer:

Tool-calling capability after SFT. Qwen2.5-VL-3B-Instruct may have particular instruction-following or JSON-formatting capabilities that make it amenable to learning the structured <tool_call> format. A VLM with weaker instruction following might require substantially more SFT data or fail to learn the XML-tagged output format altogether.
Base spatial reasoning ability. The model's pre-existing spatial understanding (53.07% on RoboSpatial VQA without any tool use, per the Direct Inference row in Table 8) provides a non-trivial starting point. A model with much weaker base spatial reasoning might benefit less from tool augmentation because it cannot effectively reason about tool outputs even when they are correct. Conversely, a model with much stronger base spatial reasoning (e.g., GPT-5 at 76.50% on RoboSpatial without tools) might benefit less from tool augmentation because its internal representations are already sufficient for many tasks.
Vision encoder quality. The paper freezes Qwen2.5-VL-3B-Instruct's vision encoder throughout all training. The quality of the frozen visual representations affects the model's ability to interpret tool outputs (which include annotated images with point markers, segmentation overlays, and depth visualizations) and to decide when tools are needed. A model with a weaker vision encoder might fail to notice that a pointing tool mislocalized an object; a model with a stronger encoder might not need tools as often.

The consequence is that the paper's conclusions — particularly the quantitative improvements from DIRL over baselines — may be specific to Qwen2.5-VL-3B-Instruct's particular capabilities and limitations. The +13.4 point improvement of DIRL over Tool SFT (Table 4) might be larger for models with weaker initial tool-use ability (where SFT alone is insufficient) or smaller for models with stronger initial ability (where SFT already captures most of the gain). The paper cannot distinguish between "DIRL is generally effective for tool-augmented spatial reasoning" and "DIRL is effective for Qwen2.5-VL-3B-Instruct's specific shortcomings."

What evidence exists in the paper: None. The paper provides no cross-model experiments. The only evidence about model-specificity comes from the proprietary model comparisons in Table 2, which show that different models have dramatically different spatial reasoning capabilities (GPT-5 scores 76.50% on RoboSpatial without tools, while Qwen2.5-VL-3B scores 53.07%), suggesting that the base model matters enormously — but these comparisons are at inference time, not training time, so they do not reveal whether DIRL would work differently on different architectures.

Mitigation status: The paper does not address this limitation. Appendix A (Limitations and Future Directions) discusses extending to "more complex, longer-horizon, or multi-stage tasks" and "richer environments," but does not mention cross-model validation. This is a standard limitation of single-model papers — the computational cost of running full DIRL training on multiple model families would be prohibitive — but it means the paper's training methodology claims are validated on exactly one model.

The Improvement from Interactive RL Over SFT Alone Is Modest Relative to the Cost

The paper's central claim is that interactive RL is essential for learning multi-tool coordination — that "interactive RL is key to teaching VLMs consistent reasoning over complex tool sequences" (Section 5.4). However, the ablation study in Table 4 reveals that removing Stage 2 IRL (keeping SFT on both teachers) reduces mean performance from 52.48 to 50.99 — a drop of only 1.49 percentage points. This is substantially smaller than the drop from removing the IRL teacher (10.8 points) or the universal teacher (9.62 points).

The breakdown by benchmark further qualifies the claim:

RoboSpatial: 70.00% with Phase 2 IRL vs. 67.71% without (+2.29 points)
RefSpatial: 53.07% vs. 51.98% (+1.09 points)
Pose: 34.37% vs. 33.28% (+1.09 points)

These are small, consistent improvements. They are directionally positive — interactive RL helps — but the magnitude raises a sharp cost-benefit question: Phase 2 IRL requires approximately 40,000 interactive tool-call episodes (8,000 samples × 5 rollouts, Table 6), each involving real tool execution on separate GPU infrastructure (8 GPUs dedicated to Toolshed). The compute cost of this phase likely exceeds the Phase 1 SFT cost substantially (SFT uses 8k static trajectories with no tool execution during training). Yet the improvement is ~1.5 points across benchmarks.

The consequence for practitioners is that Phase 2 IRL may not be cost-effective. If the goal is to maximize spatial reasoning accuracy per dollar of training compute, the evidence suggests that investing in better or more diverse teaching demonstrations (which improved performance by ~10 points per teacher in Table 4) provides far higher return than running interactive RL refinement. The paper does not establish that the specific interactive nature of Phase 2 is what drives the improvement — a second phase of SFT on additional teacher demonstrations, or even SFT on the model's own Phase 1 outputs filtered for correctness, might achieve similar gains at lower cost.

What evidence exists in the paper: Table 4 provides the direct ablation. The paper notes that "Stage 2 IRL provides the final boost of tool-augmented reasoning" and that "eliminating the Stage 2 IRL phase affects performance across RoboSpatial, RefSpatial, and pose tasks" (Section 5.4), but does not discuss the magnitude of the effect relative to its cost. The paper also does not report an ablation where additional SFT data (e.g., 2,000 more trajectories from the universal teacher) is added instead of Phase 2 IRL, which would help disentangle whether the improvement comes from interactive exploration versus simply more training signal.

Mitigation status: The paper does not address this trade-off. The framing treats Phase 2 IRL as an integral part of DIRL rather than as a separately evaluable component with its own cost-benefit profile. A more nuanced conclusion — supported by the paper's own data — would be that interactive RL provides a statistically detectable but practically small improvement over well-designed SFT, and that the primary value of DIRL is the teaching phase design (two complementary teachers) rather than the exploration phase.

Hardest Spatial Reasoning Tasks Remain Largely Unsolved Despite Tool Access

Across the benchmark results in Table 2, SpaceTools shows dramatically different absolute performance depending on task difficulty. On the easiest tasks — 2D spatial relations (CVBench: 94.92%) and depth reasoning (CVBench Depth: 96.00%) — performance is near ceiling. On tasks of moderate difficulty — RoboSpatial VQA (79.38%) and BLINK relative depth (52.46%) — performance is strong but far from perfect. On the hardest tasks — pose estimation (34.37%) and grasp prediction (MACE: 43.06%, SR: 50.00%) — performance remains low, with grasp success rate at essentially coin-flip levels.

The paper does not analyze these task-specific absolute numbers as a limitation per se, but they reveal a fundamental capability boundary: tool augmentation helps most on tasks where the base model already has some competence and tools provide precise measurements that complement existing reasoning. On tasks requiring 3D geometric reasoning from 2D observations (pose estimation, grasp affordance prediction), even with access to SAM2 segmentation, DepthPro depth estimation, 3D bounding box fitting, and GraspGen, SpaceTools fails more than half the time.

The grasp estimation results are particularly revealing. SpaceTools achieves 43.06% MACE and 50.00% SR — meaning that on half of grasp queries, the predicted grasp is either in the wrong location, wrong orientation, or both. The paper's qualitative failure analysis (Appendix E.3, Figure 12) shows that failures stem from "wrong object localization in cluttered scenes" and "inaccurate pose estimation." These are failures in the perception tools themselves (pointing mislocalizes, depth is noisy in clutter, grasp generation fails to find collision-free poses), and DIRL does not teach the VLM to overcome them — it teaches the VLM to fall back to self-estimation, but self-estimation of 6-DOF grasp poses from a single RGB image is extremely difficult even for humans.

The consequence: There is a hard ceiling on what tool-augmented spatial reasoning can achieve, set not by the VLM's reasoning capabilities but by the quality of the underlying perception tools and the fundamental difficulty of monocular 3D understanding. SpaceTools cannot outperform its tools — it can only decide when to trust them and when to attempt self-estimation. On tasks where all tools fail (cluttered grasp scenes, occluded objects, ambiguous poses), the VLM's self-estimation is unlikely to succeed either, since the base model (Qwen2.5-VL-3B) was not trained for dense 3D prediction. The paper's results suggest that tool augmentation shifts the performance curve upward but does not change its asymptotic behavior — the hardest problems remain hard regardless of how skillfully the model coordinates tools.

What evidence exists in the paper: Table 2 provides the absolute performance numbers. Table 5 shows that even frontier models (GPT-5, Claude Sonnet 4.5) with tool access achieve only 15.00% and 25.00% on pose estimation — tool access alone does not solve hard 3D reasoning. The real-robot failure case in Figure 9 (Appendix E.3) shows a boundary-placement failure that arises not from poor tool coordination but from insufficient precision in the VLM's interpretation of tool outputs. The paper does not provide a systematic analysis of what types of errors SpaceTools makes on hard tasks or whether tool failures versus reasoning failures dominate.

Mitigation status: The paper partially acknowledges this in Section 6, noting that "models tend to overuse tools and struggle to correctly interpret nuanced tool outputs" on tasks like RoboSpatial and BLINK. However, this analysis focuses on high-level tasks where tool overuse hurts performance, not on the hard geometric tasks (pose, grasp) where tools provide necessary information but still fail to achieve high accuracy. Appendix A suggests integrating "real or simulated robot feedback into the training process" and developing "lighter-weight tools, model-side approximators, or memory-optimized deployment strategies," but does not address the fundamental limitation that monocular 3D perception is an under constrained problem that tools alone cannot fully solve.

Real-Robot Evaluation Is Performed at Very Small Scale with No Statistical Rigor

The paper's robot manipulation experiments (Section 5.3, Table 3) provide the most compelling evidence for SpaceTools' practical utility — a VLM controlling a real robot arm through alternating perception and action tool calls. However, the evaluation scale is extremely small: 7 trials for Pick tasks, 6 trials for Relational Pick tasks, and 14 trials for Pick & Place tasks (scored as 28 sub-operations of pick and place). These are single-digit trial counts per task category, and the paper reports results as percentages without confidence intervals, standard deviations, or any statistical test of significance.

The paper does not report whether trials were conducted in a fixed order, whether there was any randomization, or whether the same scene configuration was used across trials. There is no discussion of scene reset between trials, lighting conditions, object pose variation, or any other factors that affect real-robot reproducibility. The Time-to-First-Movement metric (10 seconds for SpaceTools, 30–36 seconds for frontier models) is reported as a single number with no variance.

The consequence is that the reported success rates — 86% for Pick, 83% for Relational Pick, 86% for Pick & Place — have extremely wide confidence intervals. With 7 Pick trials, the 95% confidence interval for an observed 86% success rate (6/7) is approximately 42% to 100% using a standard binomial confidence interval. The difference between SpaceTools' 86% and Claude's 86% on Pick tasks is zero trials — both achieved 6/7. The Relational Pick comparison (83% vs. 50% vs. 33%) is based on 6 trials, where a single additional failure would change SpaceTools' rate to 67% and a single additional success for Claude would change its rate to 67%, eliminating the gap entirely. The Pick & Place partial success comparison (86% vs. 79% vs. 65%) is based on 14 trials (28 sub-operations), where the difference between 86% (12/14) and 79% (11/14) is exactly one sub-operation. None of these differences can be claimed as statistically significant at conventional thresholds.

What evidence exists in the paper: Table 3 reports the per-task success counts in parentheses. Table 7 in Appendix D.3 provides a per-task breakdown showing exactly which specific tasks each model succeeded or failed on. The tables are transparent about the small sample sizes, but the paper text reports percentages without qualifying their uncertainty — "SpaceTools completes pick-and-place tasks with an 86% success rate" (Section 5.3) is stated as a point estimate with no error bounds.

Mitigation status: The paper does not address the statistical reliability of the robot results. This is a common limitation in real-robot papers — physical experiments are time-consuming and expensive to run at scale — but the paper's claims about "outperforming frontier models equipped with the same tools" and "demonstrating effective transfer from spatial reasoning to embodied control" would be substantially strengthened by: (1) explicit reporting of confidence intervals, (2) a larger trial count (even 20–30 trials per task would narrow intervals considerably), (3) reporting of scene variation across trials, and (4) a clear statement that differences from frontier models on Pick and Pick & Place tasks are not statistically significant at the reported sample sizes.

Tool Call Latency and Real-Time Constraints Are Not Evaluated

The paper reports Time-to-First-Movement (TTFM) for the robot experiments: SpaceTools takes 10 seconds, Claude Sonnet 4.5 takes 30 seconds, and GPT-5 takes 36 seconds (Table 3). These numbers reveal that tool-augmented spatial reasoning incurs substantial latency — even the fastest model requires 10 seconds of perception and reasoning before initiating any physical action. The paper presents SpaceTools' 10-second TTFM as an advantage over frontier models, but does not analyze what contributes to this latency, how it scales with the number of tool calls, or whether it is acceptable for real-world deployment scenarios.

What the TTFM includes: Based on the robot manipulation workflow shown in Figure 4, a typical pick-and-place task involves: (1) capture_image (robot camera acquisition), (2) point1 (object detection, requiring GPU inference on RoboRefer), (3) segment (SAM2 inference on GPU), (4) capture_depth with pointcloud (robot depth sensor acquisition + DepthPro inference), (5) compute_grasp (GraspGen iterative optimization on GPU), (6) execute_grasp (robot motion planning and execution), (7) point1 again for placement location, (8) place_object (robot motion planning and execution). Each perception tool call involves network communication to Toolshed, GPU inference, and result transmission back to the VLM. The VLM itself generates multiple turns of reasoning text between tool calls.

The 10-second TTFM means that from the moment the instruction is given to the moment the robot arm begins moving toward the grasp pose, 10 seconds elapse. This does not include the time for the grasp motion itself, the placement motion, or any recovery from failed grasps. Total task completion time is not reported.

The consequence: For applications requiring reactive or low-latency behavior — dynamic environments where objects move, human-robot collaboration where delays degrade fluency, or high-throughput manipulation where cycle time matters — the tool-augmented approach may be fundamentally too slow regardless of its accuracy. The paper compares TTFM across models but does not benchmark against non-tool approaches: a specialized VLA model like π0.5 achieves 1-second TTFM (Table 3) by directly mapping observations to actions without iterative tool use. While π0.5's 0% success rate on these tasks makes the latency comparison moot in this specific evaluation, it illustrates the fundamental trade-off: tool augmentation improves accuracy at the cost of latency, and the paper provides no analysis of this tradeoff curve. At what point does adding more tools (e.g., verifying grasp with a second depth capture, segmenting the placement location before moving) yield diminishing accuracy returns while linearly increasing latency?

What evidence exists in the paper: Table 3 reports TTFM as a single number per model. There is no ablation showing how TTFM varies with the number of tools called, no analysis of which tool calls dominate the latency, and no comparison of SpaceTools with a version of itself constrained to use fewer tools. The 10-second number is presented as an advantage ("SpaceTools is better grounded... as well as being capable of orchestrating multiple tools, whereas other methods, like GPT-5, fail to chain tools coherently") without analysis of whether 10 seconds is fast or slow for the task.

Mitigation status: The paper does not address latency as a design constraint or evaluation dimension. Appendix B describes Toolshed's support for "elastic scaling" and "asynchronous parallel workers" as throughput optimizations, but these affect training throughput (RL steps per second) not inference latency for a single query. The gap between SpaceTools' 10 seconds and π0.5's 1 second — a 10× latency penalty for tool-based reasoning — is not discussed. A practical deployment would need to consider whether the accuracy gains from tool use justify the latency cost for the specific application, and the paper provides no framework for making this assessment.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around spatial reasoning for VLMs from "how do we bake more perceptual capabilities into model weights?" to "how do we teach models to orchestrate specialized perception tools?" — a reframing that changes what problems are considered training problems versus systems design problems, and what evaluation frameworks are appropriate.

Magnitude: a methodological reframing, not a paradigm shift. The paper does not introduce a fundamentally new learning algorithm or architecture. GRPO is adapted from DeepSeek-R1's reasoning RL; the SFT-then-RL curriculum is a standard recipe. Rather, the contribution is identifying that the multi-tool learning problem decomposes into tractable phases when teachers provide complementary supervision — single-tool RL for grounding, frontier-model demonstrations for coordination patterns, and interactive RL for robustness. This is a methodological insight about how to structure the training process, not a new mathematical framework. It is best understood as solving a specific, previously blocking engineering problem — combinatorial action space explosion in multi-tool RL — that had prevented the field from even attempting to train VLMs to coordinate more than one or two vision tools.

What becomes more attractive: The paper makes a strong empirical case that investing in tool infrastructure (Toolshed-style decoupled, asynchronous tool serving) and teacher demonstration quality yields higher returns than investing in more sophisticated RL algorithms. The 10.8-point drop from removing the IRL teacher and the 9.62-point drop from removing the universal teacher (Table 4) dwarf the 1.49-point gain from Phase 2 interactive RL. This suggests that for researchers building tool-augmented VLMs, the priority should be curating diverse, complementary teacher demonstrations and building robust tool-serving infrastructure, not designing novel RL objectives. The paper also makes modular tool ecosystems more attractive — the finding that pointing skill transfers across tasks (Table 8: IRL-trained pointing specialist achieves 34.3% on RefSpatial while other fine-tuning approaches score zero) implies that improvements to individual tools or to the model's grounding ability propagate across all downstream tasks that depend on those tools.

What becomes less attractive: The paper provides negative evidence against several research directions that previously seemed promising. Training-free tool orchestration — equipping frontier models with tool APIs at inference time without any tool-specific training — is shown to produce mixed results (Table 5): it helps on tasks requiring explicit geometric measurement (RefSpatial, pose, grasp) but degrades performance on high-level reasoning tasks (RoboSpatial, BLINK) because "models tend to overuse tools and struggle to correctly interpret nuanced tool outputs" (Section 6). This suggests that tool access without tool training is not a reliable strategy, and the field should shift toward training-based approaches. Non-interactive RL for tool use — training with ground-truth tool call traces rather than real tool execution — is shown to underperform simple SFT (Tool NIRL: 38.06 mean vs. Tool SFT: 39.19, Table 4), contradicting the intuition that RL with verifiable tool calls would provide stronger supervision. The paper's evidence suggests that real tool interaction during training matters, and that synthetic or trace-based supervision is insufficient.

Reconciling prior contradictions: The paper resolves a tension that had been emerging in the spatial reasoning literature. On one hand, ViGoRL showed that interactive RL enables a VLM to learn to use a single tool effectively. On the other hand, training-free tool orchestration approaches (Visual Programming, SpatialPIN, APC) showed that VLMs can coordinate multiple tools when provided with pre-designed pipelines, but fail when pipelines are not fixed. The paper shows that both observations were correct but incomplete: single-tool RL works because the action space is tractable; multi-tool RL from scratch fails because the action space is combinatorially large. The staged DIRL approach bridges this gap — it uses the tractable single-tool problem to bootstrap the intractable multi-tool problem — and in doing so explains why prior work reached seemingly contradictory conclusions. This also reconciles the finding that SFT on tool-use traces (TIGeR's approach) is insufficient: Table 4 shows Tool SFT achieves only 39.19 mean score versus DIRL's 52.48, confirming that static imitation of teacher trajectories does not capture the interactive, state-dependent nature of effective tool coordination.

Follow-Up Research This Work Enables

1. Training a difficulty estimator to dynamically allocate tool-use budget. The paper identifies but does not solve the problem of adaptive tool selection: SpaceTools calls tools based on learned heuristics, but there is no mechanism to decide how many tools to call based on question difficulty. A natural follow-up would be to train a lightweight classifier on top of the frozen VLM's intermediate representations that predicts, after the first turn of reasoning, whether the question is solvable with zero, one, or multiple tool calls — analogous to the difficulty estimation in the compute-optimal test-time scaling literature. The training signal exists: the 8,000 SFT trajectories contain ground-truth tool call sequences for each question, and the Phase 2 IRL rollouts provide data on when tool calls succeeded versus failed. A strong experiment would compare a budget-constrained SpaceTools (allowed at most K tool calls) against the unrestricted version on RoboSpatial and RefSpatial, measuring whether adaptive budget allocation can match unrestricted performance at lower inference cost — directly addressing the unaccounted inference-time computation critique from Section 5.

2. Cross-model validation of DIRL on different VLM architectures and scales. The paper's central training methodology claims are validated on exactly one model: Qwen2.5-VL-3B-Instruct. A critical stress-test would replicate DIRL training on at least two alternative architectures — one smaller (e.g., Qwen2.5-VL-1.5B or LLaVA-NeXT-1B to test whether tool coordination emerges with less capacity) and one with a different vision encoder architecture (e.g., InternVL2-4B or a LLaVA-style model to test whether the vision encoder quality affects tool-use learning). The key measurement would be whether the relative contributions of the two teachers and Phase 2 IRL (Table 4) replicate across architectures, or whether the IRL teacher's grounding contribution is specific to Qwen's visual representations. A negative result — finding that DIRL's gains over baselines shrink or vanish on other architectures — would substantially qualify the paper's generalizability claims. A positive result would establish DIRL as a robust training recipe independent of model choice.

3. Combining tool-augmented spatial reasoning with real-time latency constraints. The paper reports 10-second Time-to-First-Movement for SpaceTools' robot manipulation, compared to 1 second for the π0.5 VLA model (Table 3). A direct follow-up would train a variant of SpaceTools with an explicit latency penalty in the reward function — subtracting a small cost per tool call or per second of inference time — to study the accuracy-latency Pareto frontier. The experiment would sweep the latency penalty coefficient and measure both benchmark accuracy and TTFM, producing a curve showing how much accuracy is sacrificed for each second of latency reduction. This would answer a practical deployment question the paper leaves open: for a given application latency budget, what is the optimal tool-use strategy? A follow-up could also investigate whether the model can learn to prefetch tool calls — calling segmentation and depth estimation in parallel rather than sequentially — and whether such parallel tool-use strategies emerge naturally from RL or require explicit architectural support in Toolshed.

4. Extending DIRL to tasks where spatial reasoning tools produce visual rather than textual outputs. The paper's tool set primarily returns structured text and numerical variables (coordinates, depth values, mask arrays accessed via indexing). The paper acknowledges in Appendix A that "extending the model to reason over visual outputs from tools may unlock more expressive or fine-grained reasoning behaviors." A concrete follow-up would add tools that return annotated images as their primary output — for example, a tool that overlays coordinate axes on detected objects, or a tool that renders the predicted 3D bounding box back onto the 2D image — and study whether the VLM can learn to interpret these visual tool outputs by looking at them (through its frozen vision encoder) rather than only reading textual descriptions. This would test whether DIRL's staged training can teach the model to ground its reasoning in visual feedback from tools, which is a capability that current text-output tools cannot provide. The experiment would compare SpaceTools trained with text-output tools versus visual-output tools on the same benchmark tasks, measuring whether visual tool feedback improves accuracy on fine-grained spatial reasoning tasks that are hard to describe textually (e.g., "is the grasp pose aligned with the object's principal axis?").

5. Investigating whether teacher quality matters more than teacher diversity for DIRL's teaching phase. The paper uses two teachers — an IRL-trained specialist and a frontier model — and shows that removing either hurts performance (Table 4). But a deeper question is: if you could only afford one teacher, which one — and at what data scale — gives the best return? A systematic follow-up would compare: (a) doubling the IRL teacher's data (4,000 trajectories, pointing-only) versus (b) doubling the universal teacher's data (12,000 trajectories, multi-tool) versus (c) using a weaker universal teacher (e.g., GPT-4o instead of Claude Sonnet 4.5) at the same data scale. This would reveal whether the universal teacher's value comes from its multi-tool coordination patterns (which might be reproducible by a weaker model) or from its general reasoning quality (which might be specific to frontier models). A particularly informative negative result would be finding that a weaker universal teacher (e.g., GPT-4o) produces trajectories that are actively harmful — teaching the student incorrect coordination patterns that Phase 2 RL cannot fully unlearn — which would establish a quality threshold below which teacher demonstrations are worse than no demonstrations at all.

6. Ablating the contribution of specific tool types to downstream task performance. The paper provides a rich tool set (pointing, segmentation, depth, 3D bbox, grasp generation, robot control) but never ablates individual tools — we do not know which tools are actually necessary for which tasks. A follow-up would train multiple SpaceTools variants, each with one tool removed from the available set during both SFT and RL, and measure the per-task accuracy drop. This would produce a tool importance matrix showing, for example, that removing the depth estimator hurts relative depth questions but not spatial compatibility, while removing the segmentation tool hurts pose estimation but not pointing-based tasks. Such a matrix would be immediately actionable for practitioners deciding which tools to deploy in resource-constrained settings, and would test the paper's implicit claim that having diverse tools enables compositional spatial reasoning. A surprising result — finding that most tasks can be solved with only the pointing tool — would suggest that the multi-tool complexity of DIRL is unnecessary, while finding strong task–tool dependencies would validate the paper's architectural thesis about modular perception.

Practical Applications and Downstream Use Cases

Robotics assembly and logistics where object pose variation requires flexible perception. In warehouse picking or manufacturing assembly, objects arrive in unknown poses and configurations, requiring the robot to locate, identify, grasp, and place them. SpaceTools' 86% pick-and-place success rate (Table 3) and its ability to chain perception tools to handle novel object arrangements — demonstrated by the Relational Pick tasks where the model must resolve spatial relations like "pick the far coconut water" before grasping — make it suitable for deployments where object variety is too high for hardcoded perception pipelines but tasks are structured enough that 10-second perception latency is acceptable. The key practical benefit is that adding new objects or new spatial relationships does not require retraining the VLM or the perception tools; only the task specification changes. A warehouse deploying this system could add new SKUs by updating the natural language task descriptions without any model retraining, which is a substantial operational advantage over specialized VLA models that require per-object or per-task fine-tuning.

Automated spatial reasoning for accessibility applications. Navigation and object-finding tasks for visually impaired users — "is there space on this shelf for my coffee mug?" or "which elevator button is closest to me?" — require the same locate-then-analyze spatial reasoning that SpaceTools demonstrates on RoboSpatial and BLINK. The paper's finding that SpaceTools outperforms frontier models on spatial VQA (79.38% vs. GPT-5's 76.50% without tools, Table 2) and on relative depth (52.46% vs. GPT-5's 22.17% on BLINK) suggests that tool-augmented spatial reasoning provides measurable improvements over the best available general-purpose models for precisely the types of geometric questions that accessibility applications demand. The practical benefit is that a SpaceTools-like system could run on-device (the 3B model is small enough for mobile deployment) while offloading heavy perception to cloud-based Toolshed instances, providing precise spatial answers without requiring the user to carry specialized sensors.

Data annotation and quality assurance for spatial reasoning datasets. The paper's teaching dataset generation pipeline — using a frontier model with tool access to generate multi-turn reasoning traces, then filtering for correctness — is itself a practical contribution for dataset creation. Organizations building spatial reasoning benchmarks (like RoboSpatial, RefSpatial, or BOP-ASK) could use Toolshed-connected frontier models to automatically generate candidate question-answer pairs with tool-use traces, then use human annotators only to verify correctness rather than to create annotations from scratch. The paper's finding that Claude Sonnet 4.5 with Toolshed achieves non-trivial accuracy on pose estimation (25.00%, Table 5) and grasp prediction (44.19% MACE) — tasks where human annotation is expensive and error-prone — suggests that tool-augmented models can serve as a "first pass" annotation system for geometric reasoning data, reducing human annotation cost by surfacing only the hard cases where tools fail.

Rapid prototyping of robot behaviors for research labs. The paper's demonstration that SpaceTools can control a real robot arm through natural language instructions — alternating between perception tool calls (point, segment, depth, grasp) and action tool calls (execute_grasp, place_object) — provides a template for how research labs can quickly prototype new manipulation behaviors without training specialized VLA models. A lab that wants to test a new task ("pick up the leftmost object and place it behind the rightmost object") can specify it in natural language and connect their robot's perception and control APIs to Toolshed, without collecting demonstration data or running RL on the robot. The paper's finding that SpaceTools' 10-second TTFM is substantially faster than frontier models' 30–36 seconds (Table 3) makes this practical for iterative experimentation, where a researcher might test 20–30 task variations in a single session rather than waiting minutes per attempt.

When to Prefer This Method

The paper itself does not articulate an explicit decision rule for choosing DIRL over alternative approaches — the comparisons in Table 1 and Table 4 are descriptive (what prior work does) rather than prescriptive (when each approach is appropriate). However, several implicit trade-offs emerge from the experimental results that can be stated as conditional guidance:

Favor the DIRL + Toolshed approach when:

The spatial reasoning tasks require compositional use of multiple perception modalities (pointing + segmentation + depth + 3D fitting) where no single tool suffices, as in the pose estimation and grasp prediction tasks where removing the universal teacher — which demonstrated multi-tool chains — caused performance to collapse (34.37% → 8.92% on pose, Table 4).
Teacher models are available to generate teaching demonstrations — either a frontier model API (Claude, GPT) for broad tool coordination patterns, or a specialist model that can be trained via single-tool RL for grounding. The paper shows that both teachers contribute ~10 points each to overall performance, and removing both would make the approach infeasible (Direct IRL All. achieves only 19.79 mean, Table 9).
Inference latency of 10+ seconds is acceptable for the deployment scenario. SpaceTools' 10-second TTFM (Table 3) reflects the accumulated cost of multiple GPU-intensive tool calls; applications requiring sub-second responses would need either lighter-weight tools or a different architecture.
Modular tool upgrades are expected over the system's lifetime. The paper's architecture allows swapping individual tools (e.g., upgrading DepthPro to a better depth estimator) without retraining the VLM, making it suitable for long-lived deployments where perception models improve independently of reasoning capabilities.

Favor specialized spatial VLM fine-tuning (RoboRefer, RoboPoint, SpatialVLM) when:

The target task is dominated by a single perceptual capability that can be effectively baked into model weights — for instance, CVBench depth reasoning (where RoboRefer-8B-SFT achieves 96.31% vs. SpaceTools' 94.92%, Table 2) or tasks requiring only 2D pointing.
Inference latency is critical and the overhead of multi-turn tool calls cannot be tolerated. Specialized VLMs produce answers in a single forward pass without tool interaction.
Training computation for teacher generation is unavailable — the DIRL teaching phase requires generating 8,000+ tool-use trajectories from a frontier model and training a single-tool RL specialist, which together may exceed the cost of collecting task-specific fine-tuning data.

Favor training-free tool orchestration (GPT-5 + Toolshed, Claude + Toolshed) when:

Only a subset of tasks requires precise geometric measurement (RefSpatial, pose, grasp) and high-level reasoning tasks (RoboSpatial, BLINK) can tolerate some degradation, as shown in Table 5 where tool access improves geometric tasks but degrades holistic reasoning.
No training budget is available and the model must be deployed zero-shot. The paper shows that tool access alone provides substantial gains on pointing and 3D tasks for frontier models (Claude's pose estimation improves from 1.67% to 25.00% with Toolshed), making this a viable quick-start option when training is not possible.
The cost of tool overuse is low — if calling unnecessary tools only wastes compute but does not cause incorrect answers, training-free orchestration may be acceptable. The paper's finding that GPT-5's RoboSpatial performance drops from 58.39% to 55.14% with tools suggests this tolerance does not always hold.