Visual Spatial Tuning

ArXiv: 2511.05491

🎯 Pitch

Current vision-language models can describe what’s in an image but fail at basic 3D understanding—not because the information isn't there, but because training never demanded they recover it. This work shows you don’t need extra 3D encoders: simply teaching a standard VLM to estimate depth, object dimensions, and view correspondence across 4 million diverse examples builds genuine spatial perception, and adding a modest reasoning dataset with reinforcement learning pushes it to near-human-level video spatial understanding.

1. Executive Summary

This paper introduces Visual Spatial Tuning (VST), a comprehensive framework for endowing general Vision-Language Models with human-like visuospatial abilities spanning perception and reasoning, without adding specialized 3D encoders. Using Qwen2.5-VL models, the authors construct a large-scale perception dataset (VST-P, 4.1M samples across 19 tasks—e.g., monocular 3D object detection, depth estimation, multi-view correspondence) and a curated reasoning dataset (VST-R, 135K samples with chain-of-thought traces and rule-verifiable answers), then apply a progressive training pipeline—supervised fine-tuning for foundational spatial knowledge followed by reinforcement learning for spatial reasoning. The framework achieves state-of-the-art results on multiple spatial benchmarks, including 34.8% on MMSI-Bench and 61.2% on VSIBench, while preserving general multimodal performance, and further demonstrates that VST-tuned backbones boost Vision-Language-Action models by 8.6% on the LIBERO benchmark, establishing that spatial abilities in foundation models can be systematically scaled without architectural modification—though gains on the hardest spatial reasoning problems remain limited by the base model's intrinsic capability boundaries.

2. Context and Motivation

The Core Problem: VLMs Are "Spatially Blind" Despite Strong Visual Recognition

The fundamental problem this paper tackles is that current Vision-Language Models lack genuine spatial understanding, despite their impressive performance on tasks like object recognition, visual question answering, and document parsing. The distinction is between seeing and understanding space: existing VLMs can correctly identify that "there is a chair next to a table" (2D object recognition in pixel space), but they struggle when asked to reason about the chair's 3D position relative to the camera, whether it's closer than the table, what direction it faces, or how the scene would appear from a different viewpoint.

The authors anchor this distinction in cognitive science terminology (citing Piaget, 2013): spatial perception (the ability to discern spatial relationships between objects — "where is it in 3D space?") versus spatial reasoning (the ability to build and mentally manipulate an internal model of an environment — "if I move here, what will I see?"). Current VLMs exhibit competent perceptual recognition (they can put a 2D bounding box around the chair) but deficient spatial perception (they don't know the chair's depth, dimensions, or 3D orientation) and even weaker spatial reasoning (they can't reconstruct a scene layout from multiple viewpoints and infer directional relationships).

The paper's core evidence for this gap comes from the benchmarks it cites (Section 1 and Table 2–3): prior state-of-the-art models like GPT-4o score only 30.3% on MMSI-Bench (a multi-image spatial reasoning benchmark) and 34.0% on VSIBench (video spatial understanding), while performing far better on general multimodal benchmarks (e.g., 65.1% on MMStar, 84.3% on MMBench). This gap — a ≥30-percentage-point discrepancy — reveals that spatial competence is not an emergent byproduct of scaling general visual-linguistic pretraining; it requires targeted intervention.

Why Spatial Understanding Matters: From Robotics to AR/VR

The paper motivates the importance of spatial understanding through concrete downstream applications (Section 1, paragraph 1):

• Robotics and embodied AI: A robot that can recognize a "cup" and a "table" but cannot determine the cup's 3D position relative to its gripper, or predict how objects will shift when it moves its camera, is effectively useless for manipulation tasks. The authors explicitly connect to Vision-Language-Action (VLA) models for robotic control and demonstrate that their spatial tuning yields an 8.6% success rate improvement on the LIBERO benchmark (Table 14).

• Autonomous driving: Vehicles must reason about the 3D positions, velocities, and spatial relationships of surrounding objects from 2D sensor inputs. Deficiencies in spatial reasoning translate directly to safety-critical failures.

• Augmented/Virtual Reality (AR/VR): Systems that overlay digital content onto the physical world must build accurate internal 3D representations of the user's environment. A VLM that can describe what it sees but not reconstruct spatial layouts cannot serve as the intelligence backbone for such applications.

• General physical world interaction: The paper frames spatial ability as "a foundational component of general intelligence, present across a broad spectrum of animals, including humans" (Section 1). This is not merely an engineering feature — it is argued to be a necessary condition for AI systems that operate in and reason about the physical world.

The theoretical significance is equally important: spatial understanding represents a capability axis orthogonal to linguistic fluency or visual recognition. Scaling parameters and data along conventional axes improves these latter abilities, but the paper's evidence shows that spatial competence plateaus or even regresses without explicit targeted training. This suggests that current pretraining paradigms have a fundamental blind spot that cannot be addressed by "more of the same."

Prior Approaches and Their Limitations

The paper groups prior work into three categories and identifies specific shortcomings in each:

1. Expert Encoder Augmentation (Adding 3D-Specific Backbones)

Several studies (e.g., VLM-3R, Chen et al., 2024; Perception Tokens, Bigverdi et al., 2025) attempted to enhance spatial awareness by adding specialized 3D encoders or perception modules to standard VLMs. These modules might ingest depth maps, point clouds, or voxel grids alongside RGB images. The paper identifies two key problems with this approach (Section 1, paragraph 1):

• Extra architectural complexity: Each additional encoder introduces new parameters, new training procedures, and new inference overhead. This complicates deployment and makes the system less general.

• Degradation of general capabilities: Adding specialized components can interfere with the original model's pretrained representations, causing performance regressions on non-spatial tasks. The paper explicitly claims that VST avoids "the side-effect to general capabilities" (Section 1), positioning itself as a method that enhances spatial ability within the standard VLM architecture rather than bolting on external modules.

This is a significant claim that the paper backs with data: VST-7B-RL achieves 61.2% on VSIBench while maintaining 63.5% on MMStar, 83.0% on MMBench, and 68.5% on RealworldQA (Table 2), demonstrating that spatial enhancement did not cannibalize general multimodal performance.

2. Dataset-Centric Approaches (Curating Spatial Training Data)

Several prior works constructed specialized datasets to teach VLMs spatial understanding through supervised fine-tuning. The authors provide Table 1 (Section 1) cataloging these efforts:

The key limitations the paper identifies in these prior works:

• Narrow scope: Most prior datasets focus exclusively on single-image scenarios (SpatialVLM, SAT, MM-Spatial) or exclusively on video (Space-R, VLM-3R). Real-world spatial understanding requires reasoning across all three modalities — a single image for depth and layout estimation, multiple images for viewpoint integration, and video for spatiotemporal relationships.

• Perception-only focus: Most prior works stop at the SFT stage, teaching models to perceive spatial relationships but not to reason about them. Space-R is the exception — it uses RL to enhance video spatial reasoning — but it lacks the multi-image and single-image perception foundation.

• No principled progression from perception to reasoning: The paper's core pedagogical insight is that spatial understanding develops hierarchically: foundational perception skills must precede complex reasoning (mirroring Piaget's developmental theory). Prior works treat these as separate problems or only address one level.

3. CoT-Based Spatial Reasoning Approaches

Another line of work (e.g., MVoT, Wu et al., 2024; SpaceR; MindCube, Yin et al., 2025; SpatialReasoner, Ma et al., 2025; VILASR, Wu et al., 2025) focuses on enhancing spatial reasoning through chain-of-thought mechanisms. These approaches teach models to externalize spatial thinking — through multimodal reasoning traces (MVoT), textual cognition maps (SpaceR, MindCube), or intermediate 3D predictions (SpatialReasoner).

The paper identifies two limitations in these approaches:

• They assume the model already has spatial perception skills to inform its reasoning. If the base model cannot accurately estimate depth or object positions from images, no amount of reasoning chain structure will compensate. The paper argues for building perception first, then reasoning on top — which none of these prior works does.

• The reasoning traces are limited by the teacher model's own spatial understanding. If a large VLM is used to generate CoT data (as in the paper's own Stage 2), its spatial reasoning may be flawed, producing training data with incorrect layout descriptions or faulty inference steps. The paper's novel contribution here is prompting with Bird's-Eye View (BEV) annotation (Section 2.2): it provides the teacher model with a top-down visualization of the scene alongside RGB images, giving it ground-truth spatial context to generate more accurate reasoning chains.

How This Paper Positions Itself

The paper positions VST as a unified, progressive, and architecture-preserving framework that addresses all the gaps identified above:

1. Unified scope: Unlike prior datasets that focus on single-image OR video OR multi-image, VST-P (Section 2.1) spans all three modalities — 64.8% single-image data for foundational spatial perception, 33.1% multi-image data for multi-view integration, and 2.1% video data for spatiotemporal relationships. This is the first dataset to comprehensively cover the spatial understanding modality spectrum.

2. Progressive training: The three-stage pipeline (SFT → CoT Cold Start → RL) explicitly mirrors the cognitive science model of spatial development: build perceptual foundations first (Stage 1: "what is where in 3D space?"), then teach the model to explicitly reason about space through language (Stage 2: "how do I think through a spatial problem?"), then optimize for correct reasoning outcomes (Stage 3: "can I reliably get the right answer?"). No prior work combines all three stages.

3. Architecture-agnostic: VST operates entirely through data and training — it adds no new parameters, no specialized encoders, and no architectural modifications to the base VLM. The model remains a standard "ViT-MLP-LLM" architecture (Section 3). This makes the approach directly applicable to any existing VLM and avoids the "side-effect to general capabilities" that expert encoders introduce.

4. Vision-Language-Action bridging: The paper goes beyond spatial benchmarks to demonstrate that spatial tuning improves real-world robotic manipulation (Table 14), connecting the perceptual/reasoning capabilities to embodied action. This positions VST not just as a benchmark-improvement method but as infrastructure for physically grounded AI.

5. Scalability emphasis: The paper explicitly studies how spatial ability scales with model size and data volume (Tables 8–9), showing consistent improvements from 3B → 7B → 32B parameters and from 1× → 3× data. This positions VST as a scalable paradigm rather than a one-off technique.

The paper's central thesis, implicit throughout Section 1, is that spatial understanding is not a special-case capability requiring specialized architectures — it is a general competency that can be acquired through properly designed data and training curricula applied to standard VLM architectures. This directly challenges the expert-encoder approach and shifts the research question from "how do we add spatial sensing to VLMs?" to "how do we teach VLMs to understand space from the visual data they already process?"

The FLOPs-matched counterfactual is also implicitly argued: adding specialized 3D encoders increases inference cost and complexity per query, while VST's approach leaves the inference architecture unchanged and simply improves the model's ability to extract spatial information from standard RGB inputs. The spatial understanding is "compiled into" the model weights through training rather than computed at runtime through additional modules.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a data engineering and training methodology paper whose core idea is that spatial understanding can be "compiled into" standard Vision-Language Model weights through carefully constructed multi-modal training data applied across a progressive three-stage pipeline, without any architectural modification.

The system being built is a spatially-aware VLM that takes standard visual inputs (single images, multiple images, or video) and natural language questions, and produces text answers that demonstrate both spatial perception (knowing 3D positions, depths, and dimensions of objects) and spatial reasoning (mentally reconstructing scenes from multiple viewpoints and inferring directional/camera-relative relationships). The solution takes the shape of: first, construct massive perception data (~4.1M samples) to teach the model to extract 3D knowledge from 2D inputs; second, construct curated reasoning data (~135K samples) with chain-of-thought traces to teach the model to think spatially; third, apply reinforcement learning to optimize reasoning correctness.

3.2 Big-Picture Architecture (Diagram in Words)

The VST framework has four major components, connected sequentially:

1. Data Engines (Section 2) — A set of automated pipelines that generate training samples from raw data sources (RGB-D scans, synthetic simulators, web images). Each engine is specialized for a data type: single-image 2.5D/3D perception, multi-image correspondence/relationship, video spatiotemporal, and chain-of-thought reasoning. Output: structured instruction-following samples (question-answer pairs, optionally with reasoning traces).

2. VST-Perception Dataset (VST-P, 4.1M samples) — The output of the single-image, multi-image, and video data engines, spanning 19 task types. Used in Stage 1 training to inject foundational spatial knowledge.

3. VST-Reasoning Dataset (VST-R, 135K samples) — The output of the CoT engine (prompting with BEV annotation) plus general reasoning data with rule-verifiable answers. Split into 105K CoT samples (77.8%) and 30K RL verification samples (22.2%). Used in Stages 2 and 3.

4. Three-Stage Training Pipeline (Section 3.1) — Takes a pretrained VLM (Qwen2.5-VL-3B/7B/32B) and sequentially applies:

   • Stage 1 (SFT): Train on VST-P + general multimodal data to build spatial perception
   • Stage 2 (CoT Cold Start): Train on VST-R CoT samples to teach explicit spatial reasoning patterns
   • Stage 3 (RL with GRPO): Optimize on VST-R verification samples using rule-based rewards to strengthen reasoning accuracy

Information flows: raw data (scans, simulators, images) → data engines apply standardization, prompting, and annotation → VST-P and VST-R datasets → base VLM is fine-tuned in three progressive stages → spatially-aware VLM that can perceive and reason about space from standard visual inputs, and can be further adapted into a VLA model for robotic control.

3.3 Roadmap for the Deep Dive

• First, the FoV unification strategy (Algorithm 1), since it is the cornerstone of making heterogeneous 3D data sources compatible for training a single model.
• Second, the VST-P data engines — how each engine constructs training samples from raw data for single-image, multi-image, and video modalities, including the specific task types, coordinate system conventions, and design choices that distinguish VST from prior work.
• Third, the VST-R data engine with BEV prompting — the novel mechanism for generating accurate spatial CoT data when teacher models have limited spatial understanding.
• Fourth, the three-stage training pipeline in detail, including the SFT objective (Eq. 1), the CoT cold start procedure, the GRPO reinforcement learning formulation (Eqs. 2–3), and the specific hyperparameter choices at each stage.
• Fifth, the VLA adaptation — how the spatially-tuned VLM is converted into a robotic action model through action discretization and fine-tuning.

3.4 Detailed, Sentence-Based Technical Breakdown

Field of View (FoV) Unification: Making Heterogeneous Camera Data Compatible

The foundational challenge in training a single VLM to perform 3D object detection across multiple datasets is that different datasets are captured with different cameras having different intrinsic parameters. The 3D bounding box annotations in ScanNet, ARKitScenes, Hypersim, SUN-RGBD, Matterport3D, Objectron, and synthetic data from Isaac Sim all assume different camera models — different focal lengths, different fields of view, different sensor sizes. If a VLM is trained on raw images from all these sources without standardization, it must implicitly learn to account for these varying camera intrinsics when predicting 3D bounding box coordinates. The paper argues this is a "significant challenge" that "can hinder model generalization and scalability" (Section 2.1, Single-image paragraph).

The solution is FoV unification, described in Algorithm 1 (Appendix A.3). The core idea is to resample all input images so they appear to have been captured by a single virtual camera with a fixed, predefined focal length $f_{\text{new}}$. This is NOT a crop or a naive resize — it is a geometrically-principled reprojection that preserves the angular field of view while changing the image resolution to match a target focal length.

The procedure works as follows:

Step 1: Compute the original camera's horizontal and vertical fields of view. Given an input image of width $W$ and height $H$ captured with focal length $f$ (in pixel units), the horizontal and vertical angular fields of view are:

$hfov = 2 \cdot \text{atan}(W / (2 \cdot f))$

$wfov = 2 \cdot \text{atan}(H / (2 \cdot f))$

where $hfov$ is the horizontal field of view in radians, $wfov$ is the vertical field of view in radians, $W$ and $H$ are the original image dimensions in pixels, and $f$ is the original focal length in pixels.

What it computes: the angular width and height of the camera's view frustum from the original camera parameters. For a pinhole camera model, the relationship between image width $W$, focal length $f$, and horizontal field of view $hfov$ is $\tan(hfov/2) = W/(2f)$, so $hfov = 2 \cdot \text{atan}(W/(2f))$. The same applies vertically.

Why this form: this extracts the camera-agnostic property — the angular field of view — which is what actually determines what portion of the 3D scene is visible. Two cameras with different focal lengths but the same field of view capture identical visual content (just at different resolutions). By computing $hfov$ and $wfov$, we decouple the visual content from the sensor-specific parameters.

Step 2: Compute new image dimensions for the target focal length. Given a unified focal length $f_{\text{new}}$ (chosen by the authors, though the specific value is not stated), compute new image dimensions that would produce the same field of view:

$W_{\text{new}} = 2 \cdot f_{\text{new}} \cdot \tan(hfov / 2)$

$H_{\text{new}} = 2 \cdot f_{\text{new}} \cdot \tan(wfov / 2)$

where $W_{\text{new}}$ and $H_{\text{new}}$ are the target image dimensions in pixels, and $f_{\text{new}}$ is the unified focal length.

What it computes: the image resolution that a camera with focal length $f_{\text{new}}$ would need to capture the same angular field of view as the original camera.

Why this form: this ensures that after resizing, the relationship between pixel coordinates and angular directions is identical across all images. A point at pixel coordinate $(u, v)$ in the resized image corresponds to the same 3D ray direction (relative to the camera's optical axis) as a point at a corresponding pixel coordinate in any other resized image. This means the VLM can learn a single, consistent mapping from pixel positions to 3D directions, rather than having to implicitly infer and correct for each dataset's camera parameters.

Step 3: Resize the original image to the new dimensions. A standard image resize operation is applied:

$I_{\text{new}} = \text{resize}(I, (W_{\text{new}}, H_{\text{new}}))$

where $I$ is the original image, $I_{\text{new}}$ is the resized image, and $(W_{\text{new}}, H_{\text{new}})$ are the target dimensions.

Implications and design rationale. After FoV unification, all images fed to the VLM for 3D object detection share a uniform focal length. The 3D bounding box annotations are preserved in the camera coordinate system — the resizing does not change the 3D geometry, only the pixel sampling of the 2D projection. The paper's ablation in Table 12 shows that removing FoV unification (training on raw heterogeneous images) causes a 2.5 AP drop on ARKitScenes (from 29.4 to 26.9 AP), while the effect on SUN RGB-D is negligible (18.5 vs. 18.6 AP). This asymmetry makes sense: ARKitScenes is collected from mobile devices with widely varying cameras, so the intrinsic variation is high; SUN RGB-D is collected with a consistent setup, so unification matters less.

A subtle point: the VLM processes images through a Vision Transformer (ViT) with a fixed input resolution determined by its patch size and sequence length. FoV unification changes the raw image dimensions, but after the ViT tokenization (which may involve further resizing or adaptive pooling), the effective resolution at the patch level is determined by the VLM architecture. The authors do not discuss how FoV-unified images interact with the ViT's own resolution handling — this is a potential source of geometric inconsistency if the ViT applies aggressive resizing that partially undoes the FoV unification. However, given the positive ablation results, the practical effect appears to be net beneficial.

The VST-Perception Data Engines: Building 4.1M Spatial Training Samples

The VST-P dataset is not manually annotated — it is generated through automated data engines that transform raw sensor data (RGB-D scans, simulator outputs, web images) into instruction-following training samples with diverse spatial tasks. The paper describes three engines (visualized in Figure 3, left column), corresponding to single-image, multi-image/video, and the CoT engine (described later for VST-R).

Single-Image 2.5D Engine (Depth-Based Tasks)

Input data sources. This engine ingests:

1. Real-world RGB-D scans from ScanNet++ (captured with professional 3D scanners) providing ground-truth depth maps
2. Synthetic RGB-D data from Hypersim, a photorealistic simulator that renders indoor scenes with perfect depth
3. Web images from COCO, which lack depth annotations

For COCO images, the engine uses Depth Anything V2, a pretrained monocular depth estimation model, as a "depth expert" to generate pseudo-label depth maps. The authors frame this as increasing data diversity: COCO provides "in-the-wild" scenes that the indoor-focused ScanNet++ and Hypersim datasets do not cover.

Depth map standardization. After obtaining depth maps from all three sources, the engine converts them to a common coordinate system. The specific standardization (e.g., metric scale alignment, near-far clipping, invalid pixel handling) is not detailed in the paper, but the intent is clear: all depth values must represent the same physical quantity (distance from camera plane in meters) for the VLM to learn a consistent concept of depth.

Instruction sample generation. The engine produces depth-related samples in four representation formats (examples in Table 20–21, Appendix C):

1. Text-based: The question references objects by name (e.g., "Tell me the depth relationship of the objects of A.sink, B.shelves, C.blinds, D.lamp"). The model must order them from near to far based on its depth estimation.

2. Point-based: Objects are indicated by points overlaid on the image (e.g., "red point-A and point-B. Please decide which point is closer to the camera"). The point locations come from mask centers of the objects — ensuring the point actually sits on the object rather than in background space.

3. Box-based: Objects are indicated by 2D bounding boxes (e.g., "You are given several 2D bounding boxes in the image: [3, 132, 98, 468] [322, 83, 475, 481]"). The model must sort the boxes by depth.

4. Visual-prompt-based: Colored points or boxes are rendered directly onto the image as visual prompts, rather than being specified in text coordinates. This tests the model's ability to interpret visual indicators as object referents.

Task types within depth data. The paper mentions "relative depth estimation (2.5D)" and "depth comparison" as distinct tasks. The term "2.5D" is worth clarifying: it refers to the ability to infer relative depth ordering (which object is closer/further) without necessarily producing metric depth values. This is distinct from full 3D reconstruction, hence "2.5D" — it captures the perspective projection of 3D onto a 2D image plane with an ordering along the depth axis but not full metric 3D coordinates.

Design choice: multiple representation formats. Why provide the same underlying task (depth comparison) in different formats? The authors' implicit reasoning is that spatial concepts must be abstracted from specific input modalities. A model that learns to compare depths only when objects are referenced by names may fail when they are referenced by 2D boxes, even though the underlying spatial relationship is identical. By training across formats, the model learns that "depth" is a geometric property independent of how objects are referenced — exactly the kind of abstraction that general spatial understanding requires. This is a concrete instance of the paper's cognitive-science-inspired philosophy: just as humans understand spatial relationships regardless of whether objects are named, pointed at, or described, a VLMs should too.

Single-Image 3D Engine (3D Bounding Box Tasks)

Input data sources. This engine collects 3D bounding box annotations from multiple sources:

1. Open-source datasets: ScanNet, ARKitScenes, Hypersim, SUN-RGBD, Matterport3D, and Objectron
2. Synthetic data from Isaac Sim: using scenes from GRUTopia, a synthetic environment for robotics simulation

Dataset-specific corrections. The engine applies corrections for known issues in specific datasets:

• ScanNet and Matterport3D provide axis-aligned 3D bounding boxes (boxes whose edges are parallel to the world coordinate axes, not the camera axes). The engine uses "corrected versions from EmbodiedScan" — EmbodiedScan reprojects these axis-aligned boxes into camera-relative coordinate systems with proper rotations, ensuring geometric consistency.

3D bounding box standardization. The engine standardizes all annotations to a unified camera coordinate system defined as:

• Origin: camera center
• X-axis: rightward (parallel to image plane)
• Y-axis: downward (parallel to image plane)
• Z-axis: forward (along optical axis)

This is a right-hand coordinate system. The bounding box representation is a 9-degree-of-freedom (9-DoF) parameterization:

$[x_{\text{center}}, y_{\text{center}}, z_{\text{center}}, x_{\text{size}}, y_{\text{size}}, z_{\text{size}}, \text{pitch}, \text{yaw}, \text{roll}]$

where $(x_{\text{center}}, y_{\text{center}}, z_{\text{center}})$ is the box center in the camera coordinate system in meters; $(x_{\text{size}}, y_{\text{size}}, z_{\text{size}})$ are the dimensions along the X, Y, Z axes in meters (when rotation angles are zero, X-size is front-back extent, Y-size is vertical height, Z-size is lateral/side extent); and $(\text{pitch}, \text{yaw}, \text{roll})$ are Euler angles in degrees normalized by $\pi$ (i.e., divided by 180°) representing rotations around the X, Y, and Z axes respectively.

The Euler angle normalization deserves explanation: instead of representing angles in raw degrees (e.g., 45°), the engine stores them as values in $[-1, 1]$ by dividing by 180. This is a common trick in numerical regression: it keeps angle values in a bounded range that is well-behaved for neural network training (similar to how many architectures normalize coordinates to $[0, 1]$ or $[-1, 1]$). The paper's ablation (Table 12) also tests replacing Euler angles with quaternions, which perform worse — likely because quaternions, while avoiding gimbal lock, have more complex numerical relationships with visible orientation that are harder for a language model to learn from limited data.

Why this coordinate system and parameterization? The choice of camera coordinate system (rather than world coordinates) is crucial: it means the model learns object positions relative to the viewer, which is the natural format for egocentric spatial understanding. A world-coordinate representation would require the model to infer camera extrinsics (camera position and orientation in a global frame) before locating objects — a harder sub-problem that the paper avoids by working in camera space. The 9-DoF format is chosen over alternatives (e.g., 8 corners, 6-DoF pose + dimensions) because it decomposes the bounding box into semantically meaningful components (center, size, orientation) that a language model can reason about in natural language (e.g., "the chair is 1.5 meters in front of the camera and rotated 30° to the right").

Multi-turn formatting for layout learning. An important design choice: 3D object detection data is formatted in both single-turn and multi-turn formats. In multi-turn format, the model is asked to detect multiple objects sequentially, and each subsequent detection can reference the previous ones in context. The paper argues this "allows each subsequent box to reference the previous one during training, helping the model learn the layout information" (Section 2.1, Single-image paragraph). The reasoning: when detecting a chair after already detecting a table, the model can learn that chairs are often near tables, that their sizes are correlated, and that they share a common floor plane. This contextual learning of object co-occurrence and spatial priors is analogous to how humans learn typical scene layouts (e.g., "ceiling lights are above tables"), and the multi-turn format makes these statistical regularities explicit in the training signal.

Scene caption generation. The paper notes that training exclusively on template-based 3D detection data risks the model "overfitting to the specific numerical values and failing to generalize spatial understanding." To mitigate this, the engine generates scene captions (examples in Table 25–26): natural language descriptions that focus on layout and spatial relationships rather than object identities. The generation process:

1. Extract object information (category, 3D center, size) from the scene
2. Extract object relationships (binary and multi-object spatial relations) from a scene graph
3. Prompt a large VLM (specified as "a large VLM" in Section 2.1, identified as Seed1.5-VL in Appendix C, Table 26) with the object data and relationship information
4. The prompt (Table 26) instructs the VLM to produce 4–20 sentence descriptions using specific formatting rules (e.g., "Strictly preserve object names," "Camera to nearest object to depth progression," "Prohibited phrases: 'according to the coordinates'").

The scene captions serve a dual purpose: they teach the model to verbalize spatial relationships in natural language, bridging the gap between numerical 3D predictions and language-based spatial understanding, and they provide dense supervision for spatial layout learning that is less brittle than exact 3D coordinate prediction.

Task types from 3D engine. The engine produces four task categories:

1. 3D Object Detection: Predict the 9-DoF bounding box for specified objects (Table 23)
2. 3D Grounding: Given a 3D bounding box description, identify which object it refers to (the inverse of detection — teaches the model to interpret spatial descriptions)
3. Attribute Measurement: Predict specific dimensions (e.g., "Provide the height of the table in centimeters," Table 24)
4. Distance Estimation: Compare distances between objects (e.g., "Identify which object is nearest to the counter," Table 22)

Multi-Image and Video Engine

Input data sources. This engine samples multi-image sequences from RGB-D scans in ScanNet, ScanNet++, and ARKitScenes. These datasets provide sequences of RGB-D frames captured as a camera moves through a scene, with known camera poses (position and orientation for each frame). The engine samples pairs of frames (for multi-image data) or sequences (for video data) from these trajectories.

Coordinate system unification. All 3D object annotations are transformed into the camera coordinate system of the first image in the sequence. This is a critical design choice: it means the model learns spatial relationships from a fixed reference frame, with subsequent images providing additional viewpoints from known relative camera poses. The alternative — using each image's own camera coordinate system — would require the model to infer camera motion before relating objects across views, which is a harder sub-problem. By establishing the first frame as the canonical reference, the model can directly compare object positions across images using a consistent coordinate system.

Task types from multi-image data:

1. Correspondence (Table 27): Given a reference point in the first image, identify the matching point in the second image. The correspondence points are generated by projecting 3D points onto both images, with occlusion checking (a point is "visible" only if it is within image bounds, in front of the camera, and not occluded — determined by comparing the projected depth to the depth map at that pixel with a 5% relative threshold). This teaches the model to recognize that the same 3D point projects to different 2D locations in different views.

2. Multi-Image 3D Object Detection (multi-image variant of the single-image task): Detect objects in the first frame's camera coordinate system using information from multiple views. This is more powerful than monocular detection because the model can use parallax — the apparent shift of objects relative to the background when the camera moves — to resolve depth ambiguities.

3. Object-Object Relationship (Table 28): Questions about spatial relationships between objects visible in different images. These take the form: "If object A is north of object B, what direction is object C from object B?" The model must reconstruct the 3D layout from limited views, establish a reference direction, and compute relative bearings. The engine selects pairs of images that share at least one common object (to establish correspondence) and contain objects unique to each image (to create non-trivial relational queries). Anchor objects are defined as those present exclusively in one image with center height difference less than 1 meter (ensuring they are on the same floor plane).

4. Camera-Camera Relationship (Table 29): Questions about the relative pose of the camera itself. For example: "When positioned at the second photo spot, how is the first camera placed relative to me?" The model must infer the camera's motion (forward, left, rotated, etc.) from the visual changes between images.

5. Camera Motion (Table 30): Explicitly classify the camera's movement type. The rotation categories include pan left/right, tilt up/down, roll clockwise/counter-clockwise. Translation categories include leftward/rightward/upward/downward/forward/backward movement. These motion types are defined in the prompts used to generate reasoning data (Tables 34–35) with detailed descriptions of how each motion manifests visually (e.g., "Moving Rightward: Foreground objects appear to shift significantly to the LEFT relative to the background").

6. Scene Caption (multi-image variant): Generate a textual description of the scene layout across multiple images, similar to the single-image version but integrating information from multiple viewpoints.

Task types from video data (2.1% of VST-P):

1. Object-Object Relationship (Table 31): Same as the multi-image version but across video frames, with multi-turn formatting for efficiency.
2. Object Counting (Table 32): Count instances of object classes across the video (e.g., "How many chairs can you spot in this part of the video?").
3. Spatiotemporal (Table 33): Identify the order of appearance of objects, requiring temporal reasoning about when objects enter the field of view.
4. Route Plan: Navigation-related spatial reasoning (details sparse in the paper).
5. Measurement: Similar to single-image measurement but using video context.

The video dataset is augmented by sampling two-thirds of the data from VLM-3R, a prior video spatial dataset, and reorganizing it from single-turn to multi-turn format. The multi-turn format is emphasized for video because loading video data is computationally expensive — asking multiple questions about the same video in one training example amortizes the I/O cost.

Why 2.1% video? The video proportion is notably small compared to single-image (64.8%) and multi-image (33.1%). The authors do not explicitly justify this ratio, but several factors likely contribute: video data is more expensive to collect, process, and train on (each sample requires loading multiple frames); the multi-image data already provides many of the same spatial reasoning challenges (multi-view integration, parallax, camera motion); and the primary goal of video data is to add temporal understanding (appearance order, counting across frames) that neither single-image nor multi-image data provides. The small proportion reflects a cost-benefit tradeoff: the marginal spatial learning from video beyond what multi-image already teaches is limited, but the cost is high.

The VST-Reasoning Data Engine: Generating Accurate Spatial CoT with BEV Prompting

The core challenge. To train the VLM to reason spatially, the paper needs chain-of-thought (CoT) data — examples showing the step-by-step reasoning process for solving spatial problems (e.g., "First, I identify the common objects between the two images to establish correspondence. Then, I reconstruct the room layout..."). Generating such data requires a teacher model that can produce accurate spatial reasoning traces. However, the authors acknowledge a critical limitation: "the multi-view spatial understanding of the current large VLMs remains limited relative to their general multi-modal abilities" (Section 2.2). In other words, even the best available VLMs (the paper uses Seed1.5-VL as the teacher) are themselves spatially deficient — if prompted with only RGB images, they would produce reasoning traces containing spatial errors, hallucinations, or inconsistent layout descriptions.

The solution: prompting with Bird's-Eye View (BEV) annotation. The key insight is to provide the teacher model with ground-truth spatial information rendered as a visual aid, rather than expecting it to infer spatial relationships from RGB images alone. Specifically, the engine:

1. Takes the ground-truth 3D bounding boxes for all objects in the scene
2. Projects them onto a top-down (bird's-eye view) visualization — a 2D view looking down from above the scene, where each object is represented by its footprint (projection of the 3D box onto the ground plane)
3. Provides this BEV image as an additional input alongside the RGB images when prompting the teacher model

The BEV image serves as "an auxiliary spatial prompt, allowing the teacher model to better capture spatial relationships compared to using only RGB images" (Section 2.2). It explicitly visualizes:

• The relative positions of objects in the horizontal plane (X-Z plane in the coordinate system)
• The spatial relationships (object A is to the left of object B, object C is behind object D)
• The camera positions for each image (represented as frustum indicators or camera icons)

The prompt structure for CoT generation (Table 19, Appendix C) provides:

• Coordinate system definition: X rightward, Y downward (perpendicular to ground), Z forward
• Object information: category, 3D center position, dimensions, and a list indicating which frames each object appears in
• Camera information: relative poses between frames
• Reference calculation process: the numerical computation for the answer (provided for the teacher to use as ground-truth)
• Explicit instructions: "Always use the TOP VIEW image to ensure your spatial descriptions are consistent with the overall scene layout, but do not explicitly reference the TOP VIEW image in your wording"

The reasoning trace is structured in four parts (Table 18 example shows this pattern):

1. Describe the first image's visual content (what objects are visible, where they are)
2. Describe the second image's visual content with emphasis on correspondences and changes from the first image
3. Summarize the overall scene layout (inferred from both RGB images and the BEV), then apply logical reasoning to answer the question
4. State the final answer

Why BEV prompting works. The BEV image provides the teacher model with information it cannot reliably extract from RGB images alone: accurate metric spatial relationships. However, the teacher still performs valuable work that the BEV does not directly provide: (1) connecting the BEV's top-down abstraction to the first-person visual appearance in the RGB images, (2) generating natural language descriptions of spatial relationships that sound like human reasoning, and (3) structuring the reasoning process into coherent steps. The BEV ensures spatial accuracy; the teacher model provides linguistic fluency and reasoning structure.

The paper's ablation (Table 10) validates this design. Comparing two CoT generation methods:

• RT-CoT (text-based layout reconstruction without BEV): achieves 30.0 on MMSI-Bench
• RT-CoT_BEV (with BEV prompting): achieves 31.1, a 1.1% improvement

The BEV condition particularly helps on the Positional Relationship subsets (OO: object-object: 36.2 vs. 31.9; CO: camera-object: 40.0 vs. 36.5), which are exactly the categories that require accurate spatial layout understanding. On Attribute subsets, the effect is smaller or mixed, consistent with the intuition that attribute questions (size, measurement) depend less on multi-object spatial relationships.

Alternative considered: numerical CoT (Num-CoT). The paper also experiments with representing spatial layouts using 3D bounding boxes and having the model reason through numerical computation (e.g., "Object A is at coordinates (1.2, 0.5, 3.1), Object B is at (2.1, 0.5, 1.8), so the vector from A to B is..."). This achieves 29.2 on MMSI-Bench, outperforming the baseline (26.4) but underperforming text-based reconstruction (30.0 for RT-CoT, 31.1 for RT-CoT_BEV). The authors attribute this to two factors: "estimating camera poses across multiple images can be difficult, especially when viewpoints vary greatly, which can reduce the accuracy of 3D object detection," and "human beings do not reason about space through direct numerical calculation" — suggesting that text-based layout reasoning aligns better with how VLMs (trained primarily on language) represent and manipulate spatial concepts.

VST-R dataset composition (Figure 2b):

• 105K CoT samples (77.8%): Used for Stage 2 (CoT Cold Start)
  • Spatial Reasoning (SR): 42K across single-image (15K), camera motion (6K), object-object (15K), camera-camera (6K)
  • General Reasoning (GR): 64K across math (31K), OCR (9K), knowledge (24K)
• 30K RL verification samples (22.2%): Used for Stage 3 (RL)
  • Spatial Reasoning (SR): 10K
  • General Reasoning (GR): 20K

The inclusion of general reasoning data (math, OCR, knowledge) in both stages is deliberately designed "to preserve the model's reasoning ability on general tasks." This prevents catastrophic forgetting: without general data, a model fine-tuned exclusively on spatial CoT might lose its ability to reason about non-spatial problems. The roughly 60/40 split (60% general, 40% spatial) balances spatial enhancement with capability preservation.

Stage 1: Supervised Fine-Tuning for Spatial Perception

Objective. Stage 1 trains the base VLM to acquire foundational spatial perception abilities — the capacity to extract 3D information (depth, object positions, dimensions, orientations) from 2D visual inputs. The model learns to answer questions that require spatial perception but not complex multi-step reasoning.

Training data. A mixture of VST-P (4.1M samples) and general multimodal data. For the ablation studies using Qwen2.5-VL-3B, the mixture is one-third VST data (~1.37M samples) + 800K general samples from LLaVA-OneVision. For final models, the full VST dataset is combined with 2.4M general multimodal samples (a ~1.7:1 ratio of VST to general data, or roughly 63% VST, 37% general).

The general data serves as a regularizer against catastrophic forgetting: without it, the model might overwrite its pretrained multimodal capabilities (VQA, captioning, document understanding) while learning spatial perception. The LLaVA-OneVision dataset is chosen because it covers diverse visual tasks, providing broad regularization.

Training objective. Standard autoregressive language modeling loss applied to text tokens:

$\mathcal{L}_{\theta}(x) = -\sum_{i=2, x_i \in \text{text}}^{L} w_i \log p_{\theta}(x_i | x_1, \ldots, x_{i-1})$

where $x = [x_1, \ldots, x_L]$ is a training sample of length $L$ (containing both visual tokens and text tokens), $\theta$ are the model parameters, $p_{\theta}(x_i | x_{<i})$ is the model's predicted probability for token $x_i$ given all preceding tokens, $w_i$ is a per-token weight (used to mask visual tokens — the loss is computed only on text tokens, as indicated by the condition $x_i \in \text{text}$), and the summation runs from $i = 2$ to $L$.

What it computes: the negative log-likelihood of the text portion of each training sequence, summed over all text tokens. For a training sample like "Q: Detect the 3D bounding box of the chair. A: [0.19, 0.56, 1.78, ...]", the loss penalizes the model for predicting the wrong text token at each position in the answer, conditioned on the question tokens and the visual tokens from the image. The sum over all samples in a batch, averaged, gives the SFT loss.

Why this form: this is the standard causal language modeling objective used for pretraining and instruction tuning of autoregressive models. It is chosen because (1) it is the objective the base model (Qwen2.5-VL) was originally trained with, so fine-tuning with the same objective causes minimal distribution shift in the optimization landscape, and (2) it naturally handles the mixed visual-text input through the transformer's self-attention mechanism — the model learns to attend from answer tokens to relevant visual patches and question tokens to extract spatial information. Alternative objectives (e.g., a separate regression head for 3D coordinates) would require architectural modifications and break the end-to-end text generation paradigm.

Hyperparameters (Table 15):

• Optimizer: AdamW
• Base learning rate: $5 \times 10^{-5}$
• Vision encoder learning rate: $5 \times 10^{-6}$ (10× lower than the LLM learning rate, preserving pretrained visual features while allowing the LLM to adapt to spatial tasks)
• Sequence length: 16,384 tokens
• Global batch size: 128
• Warmup ratio: 0.03 (3% of training steps for linear learning rate warmup)
• Epochs: 1
• Dynamic data packing: enabled (multiple short samples are packed into a single sequence to minimize padding waste)

The differential learning rate (5e-5 for LLM, 5e-6 for vision encoder) is a standard practice in VLM fine-tuning. The reasoning: the vision encoder was pretrained on massive image data (likely billions of images) and already produces good visual features for object recognition. Fine-tuning it aggressively on a relatively small spatial dataset (4.1M images) risks destroying these general features. The LLM, by contrast, was pretrained on text and needs significant adaptation to learn the new spatial vocabulary (3D coordinates, depth relationships, measurement units). Setting the vision encoder LR an order of magnitude lower preserves visual feature quality while allowing the LLM to learn to extract spatial information from those features.

Expected outcomes. After Stage 1, the model (referred to as VST-*-SFT) should demonstrate:

• Improved performance on spatial perception benchmarks (CVBench-3D, 3DSRBench, BLINK, VSI-Bench)
• Sustained or only slightly reduced performance on general benchmarks (MMStar, MMBench, etc.)
• The ability to answer questions like depth comparison, distance estimation, 3D object detection, and multi-view correspondence from standard RGB inputs — tasks the base model could not perform reliably

The ablation in Table 5 shows progressive improvement as each data type is added: the baseline (general data only) achieves 49.9 S-AVG, which climbs to 56.4 after adding all single-image spatial data, then to 57.7 after adding multi-image data (Table 6), then to 60.6 after adding video data (Table 7) — a total S-AVG improvement of 10.7 percentage points from end-to-end Stage 1 training.

Stage 2: Chain-of-Thought Cold Start

Objective. Stage 2 teaches the Stage 1 model to reason spatially using explicit step-by-step thinking. The model learns to structure its responses with a thinking ... response ... format, where the thinking section reconstructs the spatial layout in text and applies logical inference, and the response section provides the final answer.

Training data. A mixture of the 105K CoT samples from VST-R (77.8% of VST-R) and general reasoning data. The spatial CoT data includes:

• Spatial Reasoning Single-Image (SR-SI): 15K samples teaching the model to reason about spatial relationships in a single image
• SR-Camera Motion: 6K samples analyzing how the camera moved between frames
• SR-Object-Object: 15K samples reasoning about directional/relational queries between objects across images
• SR-Camera-Camera: 6K samples inferring camera-to-camera spatial relationships
• General Reasoning Math: 31K samples with math reasoning traces
• General Reasoning OCR: 9K samples with OCR-related reasoning
• General Reasoning Knowledge: 24K samples with knowledge-based reasoning

Why "cold start"? The term "cold start" (borrowed from the DeepSeek-R1 training pipeline) refers to using supervised data to initialize the model's reasoning behavior before reinforcement learning. Without this stage, the model would be asked to produce CoT reasoning during RL from scratch, and the initial outputs would likely be poorly structured, making RL exploration inefficient. The CoT cold start provides a "warm" initialization where the model already produces reasonably formatted and structured reasoning traces, allowing RL to focus on improving reasoning correctness rather than reasoning format.

CoT format (Appendix A.2). Every CoT sample follows this structure:

<|im_start|>system
You are a helpful assistant. You should first think about the reasoning process 
in the mind and then provide the user with the answer. The reasoning process is 
enclosed within  thinking  response tags, i.e.,  thinking reasoning process 
here  response answer here.<|im_end|>
<|im_start|>user
<|vision_start|>image.jpg<|vision_end|>{question}<|im_end|>
<|im_start|>assistant
 thinking{thinking content} response {answer}<|im_end|>

The thinking ... response ... tags are special tokens that the model learns to emit to delineate the reasoning section from the answer section. This is a design choice borrowed from DeepSeek-R1: by explicitly separating reasoning and answering, the model learns to perform "internal" reasoning that can be lengthy and exploratory without polluting the final answer, and the reasoning trace can be examined for debugging.

Training objective. Same autoregressive loss as Stage 1 (Equation 1), but now the model is learning to predict long reasoning traces in addition to answers. The loss is computed on all text tokens within both thinking and response sections.

Hyperparameters (Table 16):

• Optimizer: AdamW
• Base learning rate: $1 \times 10^{-5}$ (5× lower than Stage 1, to avoid catastrophic forgetting of Stage 1 spatial perception skills)
• Vision encoder learning rate: $1 \times 10^{-6}$ (10× lower than LLM, consistent with differential LR strategy)
• Sequence length: 16,384 tokens
• Global batch size: 128
• Warmup ratio: 0.03
• Epochs: 2 (double Stage 1 epochs)

The authors note a specific observation: "we train the model for 2 epochs, as we observed that smaller-scale models require extended training to effectively master the long-form CoT reasoning process." This is a practical empirical finding: CoT reasoning involves generating dozens or hundreds of tokens of structured reasoning, which is a more complex behavior to learn than the short-answer format of Stage 1. One epoch is insufficient for the model to reliably produce well-formed reasoning chains.

Expected outcomes. After Stage 2, the model should demonstrate basic spatial reasoning capability — it can follow the CoT format and produce plausible (though not always correct) reasoning traces for spatial questions. Table 10 shows the cold-start model achieves 31.7 on MMSI-Bench (with CoT inference), compared to the pre-Stage-2 baseline of 26.4 (without CoT) — a 5.3% improvement. Table 11 shows an interesting phenomenon: after CoT cold start, the model performs better without CoT inference (33.6) than with CoT inference (31.7). This suggests the model has absorbed spatial knowledge from the CoT training data that improves its direct answering ability, but its CoT reasoning chains are still error-prone, leading to worse results when CoT is explicitly used. This motivates Stage 3: reinforcement learning to strengthen the reasoning chains.

Stage 3: Reinforcement Learning with GRPO

Objective. Stage 3 improves the Stage 2 model's spatial reasoning correctness through reinforcement learning, using Group Relative Policy Optimization (GRPO) with rule-based rewards on verification data.

Why RL rather than continued SFT? The paper's motivation mirrors that of DeepSeek-R1: SFT teaches the model to imitate reasoning traces, but those traces (even from a strong teacher) may contain suboptimal reasoning patterns, flawed inferences, or inefficient chains. RL allows the model to explore alternative reasoning paths and be rewarded for those that lead to correct answers, potentially discovering better reasoning strategies than what the teacher demonstrated. Additionally, RL can optimize for outcomes on tasks where generating high-quality SFT data is difficult (e.g., 3D object detection, where the answer format is structured numerical output).

GRPO algorithm. The paper uses Group Relative Policy Optimization (GRPO), adapted from DeepSeekMath and refined in the VeRL framework. GRPO is an actor-only RL algorithm that eliminates the need for a separate value function (critic) model, reducing memory and computational overhead:

In GRPO, for each question, the model generates a group of responses. The advantage of each response is computed relative to the mean reward of the group, rather than using an absolute value function.

The key mechanism: for each training question $q$, the current policy (the Stage 2 model) generates $G$ responses $\{\hat{y}_1, \ldots, \hat{y}_G\}$ (the paper sets $G = 5$ in Table 17, listed as "Rollout number: 5"). Each response receives a scalar reward $R(q, \hat{y}_g)$ from the reward function. The advantage $A_g$ for response $g$ is computed as:

$A_g = \frac{R(q, \hat{y}_g) - \mu_R}{\sigma_R}$

where $\mu_R = \frac{1}{G}\sum_{j=1}^G R(q, \hat{y}_j)$ is the mean reward across the group, and $\sigma_R$ is the standard deviation of rewards in the group.

What it computes: the normalized advantage — how much better or worse this particular response is compared to the average response the model generates for this question. If a response gets a higher reward than average, its advantage is positive, and the policy is updated to make that response more likely. If a response gets a lower reward, its advantage is negative, and the policy is updated to suppress it. The normalization by standard deviation ensures that the advantage scale is consistent across questions with different reward variances.

Why this form: computing advantages relative to the group mean eliminates the need for a value function baseline, which would require training a separate model to predict expected rewards. The group-relative formulation is a form of "self-normalization" that works well when the reward function is well-calibrated (accurate rewards discriminate good from bad responses) and the group size is large enough to provide a stable baseline. The alternative (using an absolute value function) would add training complexity and potential instability. The PPO-style clipping parameters (clip_low=0.2, clip_high=0.28 in Table 17) control how much the policy is allowed to change per update, preventing destructive large updates.

The reward function. The overall reward for a response $\hat{y}$ given ground truth $y$ is:

$R(y, \hat{y}) = R_{\text{acc}}(y, \hat{y}) + R_{\text{format}}(y, \hat{y})$

where $R_{\text{acc}}(\cdot, \cdot)$ is an accuracy reward that scores correctness, and $R_{\text{format}}(\cdot, \cdot)$ is a format reward that incentivizes adherence to the thinking... response ... structure.

What it computes: the sum of two independent reward components. The accuracy reward measures whether the answer is correct according to the task-specific evaluation protocol. The format reward provides a small bonus (likely binary: 1 if properly formatted, 0 otherwise) for producing outputs with the correct CoT structure. The total reward is unbounded (accuracy rewards may sum to different maximum values depending on the task).

Why this form: separating accuracy and format rewards allows the model to receive partial credit for producing well-structured reasoning even when the answer is wrong (which guides exploration toward better-structured chains), while ensuring that correct answers remain the dominant optimization signal. The additive combination is simpler than a multiplicative or hierarchical reward and works well when the two components are independent (format correctness doesn't depend on answer correctness).

Accuracy reward by task type:

For multiple-choice, open-ended, and OCR tasks, the accuracy reward is computed following "standard evaluation protocols" — likely exact match with the ground-truth answer for multiple-choice and open-ended, and standard OCR metrics (character accuracy, word accuracy) for OCR tasks. The paper cites existing benchmark evaluation protocols: VQA accuracy (Goyal et al., 2017), ChartQA relaxed accuracy (Masry et al., 2022), DocVQA ANLS (Mathew et al., 2021), TextVQA accuracy (Singh et al., 2019).

For 3D object detection tasks, the accuracy reward is a linear combination of 3D IoU and F1 score:

$R_{3D}(y, \hat{y}) = \alpha \cdot R_{\text{IoU}}(y, \hat{y}) + (1 - \alpha) \cdot R_{\text{F1}}(y, \hat{y})$

where $\alpha = 0.5$ (default), $R_{\text{IoU}}(\cdot, \cdot)$ is the average 3D Intersection over Union of matched bounding box pairs, and $R_{\text{F1}}(\cdot, \cdot)$ is the F1 score based on an IoU threshold of 0.25 for true positive matches.

What it computes: First, given $N$ predicted 3D bounding boxes and $M$ ground-truth boxes, a bipartite matching (Hungarian algorithm) pairs predictions to ground truth to maximize total IoU. $R_{\text{IoU}}$ is the average IoU across all successfully matched pairs. For $R_{\text{F1}}$, a matched pair is a true positive if its IoU exceeds 0.25; precision = true_positives / N, recall = true_positives / M, and F1 is the harmonic mean. The final reward is the equally-weighted average of IoU and F1.

Why this form: combining IoU and F1 addresses complementary aspects of detection quality. IoU measures spatial precision (how well the predicted boxes align with ground truth), but can be low even when the correct objects are detected (e.g., large boxes with slight misalignment). F1 measures detection completeness (did the model find the right objects?), but ignores precise localization quality. Equal weighting ($\alpha = 0.5$) balances these concerns. The alternative attempted (IoU + recall, Table 13) caused "performance drops markedly because each ground-truth box is matched with too many false-positive predictions" — recall rewards the model for detecting all ground-truth objects but doesn't penalize false positives, leading the model to predict many boxes to inflate recall at the expense of precision. F1 naturally penalizes this through its precision component.

The IoU threshold of 0.25 for F1 computation is relatively lenient (typical object detection uses 0.5 or 0.75), reflecting the difficulty of 3D detection from monocular/multi-view inputs — a 25% overlap is considered a "reasonable detection" in this setting, whereas stricter thresholds would classify most predictions as failures and provide no training signal.

Verification dataset composition. The 30K RL verification samples from VST-R are categorized into four task types: multiple-choice, open-ended, OCR, and 3D detection. The inclusion of 3D detection in RL is notable: it treats structured prediction (regression of 9-DoF bounding boxes) as a text generation task, with the accuracy reward computed by parsing the generated text back into numerical coordinates and computing geometric metrics. This demonstrates that RL with rule-based rewards can optimize for tasks beyond standard classification/QA.

RL hyperparameters (Table 17):

• Optimizer: AdamW
• Learning rate: $1 \times 10^{-6}$ (constant, no warmup or decay, 10× lower than Stage 2 for stable policy updates)
• Vision encoder learning rate: $1 \times 10^{-6}$ (same as LLM in this stage — vision features are allowed to adapt to the RL optimization)
• Max sequence length: 8,192 (half of SFT stages, likely because RL rollouts are shorter — the model generates responses, not full multi-turn dialogues)
• Global batch size: 128
• Rollout number (G): 5 responses per question
• KL penalty: disabled (the authors do not use a KL divergence penalty to keep the policy close to the reference model, unlike some RLHF implementations)
• PPO clipping: clip_low = 0.2, clip_high = 0.28

The disabled KL penalty is a significant design choice. In standard RLHF, a KL divergence term penalizes the policy for deviating too far from a reference (usually the SFT) model, preventing reward hacking. The authors apparently found it unnecessary — the rule-based rewards are well-specified enough that the model doesn't learn degenerate behaviors that exploit reward loopholes, or the small learning rate and PPO clipping provide sufficient regularization.

Expected outcomes. Table 11 shows RL results: the model improves from 31.7 (cold start, with CoT) to 35.3 (RL, with CoT) on MMSI-Bench — a 3.6% absolute improvement. Notably, the RL model without CoT inference scores 34.7, only 0.6% below the CoT version, suggesting that RL strengthens both explicit reasoning chains and the model's implicit spatial knowledge. The RL model with CoT shows the largest gains on Camera-Object (CO) relationships: 48.5 with CoT vs. 36.4 without CoT after RL vs. 22.7 at cold start — a 25.8% absolute gain on this challenging subset, demonstrating that RL particularly benefits tasks requiring explicit spatial thinking.

Expanding to Vision-Language-Action (VLA) Models

Motivation. The paper investigates whether spatial tuning benefits downstream embodied AI tasks by converting the spatially-tuned VLM into a VLA model for robot manipulation. The hypothesis: a model that better understands 3D spatial relationships should more effectively predict robot actions, which are inherently spatial (moving a gripper to a specific 3D position).

Adaptation methodology. Following OpenVLA, the approach formulates action prediction as a vision-language generation task:

1. Input: An observation image (third-person camera view of the robot workspace, 256×256 resolution) and a natural language instruction (e.g., "pick up the black bowl")
2. Output: An action sequence predicted auto-regressively as text tokens

Action discretization. The continuous action space is discretized into 256 bins. Each action dimension (likely 7-DoF: 3D position, 3D orientation, gripper state, following standard robot manipulation action spaces) is mapped to a discrete token in a range of 256 values. These tokens are added as special tokens to the language model's vocabulary (the "action de-tokenizer" in Figure 4b).

The discretization scheme: for each continuous action dimension, the range of possible values is divided into 256 equal-width bins. A continuous value is converted to the nearest bin index, which becomes the token ID. During inference, the model generates a sequence of bin tokens, which are de-tokenized back to continuous actions by mapping each bin index to its center value. This is the standard approach used in RT-2 and OpenVLA, chosen because it allows action prediction to be handled by the same autoregressive text generation mechanism as language — no architectural changes are needed, only vocabulary expansion.

Training objective and hyperparameters. Same autoregressive loss as Equation 1, but with modified hyperparameters to accommodate the shorter action sequences and smaller images:

• Optimizer: AdamW
• Base learning rate: $5 \times 10^{-5}$
• Vision encoder learning rate: $5 \times 10^{-6}$ (differential LR preserved from Stage 1)
• Global batch size: 128
• Max sequence length of data packing: 1,024 (down from 16,384 — action sequences are very short, typically a few dozen tokens)
• Training steps: 2.5K and 10K on LIBERO

The reduced packing length (1,024 vs. 16,384) is "necessary to compensate for the relatively short action sequences and the small resolution of the training images." This likely means that with a very long packing length, each training sequence would contain hundreds of action prediction examples, and the gradient would be dominated by the easy examples (where the action is simple), while the hard examples contribute proportionally less. A shorter packing length ensures each batch contains a more balanced mix of examples.

Training data. Direct fine-tuning on the LIBERO benchmark, without any pretraining on robotic data — in contrast to OpenVLA, which pretrains on a large-scale robotic manipulation dataset before fine-tuning. This is a deliberate choice to isolate the effect of spatial pretuning: any performance difference between Qwen2.5-VL-3B and VST-tuned Qwen2.5-VL-3B can be attributed solely to the spatial knowledge gained during VST training, not to robotic-specific pretraining.

Results interpretation. Table 14 shows:

• At 2.5K steps: VST-tuned model achieves 61.7% average success rate vs. 53.1% for the base model (+8.6%)
• At 10K steps: 70.8% vs. 67.9% (+2.9%)

The diminishing gap at longer training (8.6% → 2.9%) suggests that with enough robot-specific training, the base model eventually learns some spatial understanding from the LIBERO data directly. However, the VST-tuned model achieves equivalent or better performance with 4× fewer robot-specific training steps (2.5K steps with VST ≈ 10K steps without VST on several task suites), demonstrating that spatial pretuning improves sample efficiency — the model doesn't need to learn spatial perception from scratch during robot training because it already possesses it.

The largest improvements at 2.5K steps are on the LIBERO-Spatial (+8.4%) and LIBERO-10 (+10.4%) suites — exactly the tasks most dependent on spatial understanding. LIBERO-Spatial requires reasoning about object positions for rearrangement, while LIBERO-10 is a long-horizon benchmark requiring multiple sequential spatial interactions. The smaller gains on LIBERO-Object (+1.8%) and LIBERO-Goal (+4.0%) reflect that these suites emphasize object recognition and goal understanding more than spatial manipulation.

4. Key Insights and Innovations

Innovation 1: Spatial Understanding as a Capability That Can Be "Compiled Into" Standard VLM Weights Through Data Alone

The paper's most fundamental conceptual move is reframing spatial understanding not as a sensing problem requiring specialized hardware or architectures, but as a learning problem solvable through properly constructed training data applied to existing model architectures. This directly challenges the dominant assumption in prior work—visible in VLM-3R, Perception Tokens, and others—that VLMs need dedicated 3D encoders (depth backbones, point cloud processors, voxel grids) to achieve genuine spatial awareness. Those approaches implicitly assume that the standard "ViT-MLP-LLM" pipeline lacks the representational capacity for 3D understanding and must be augmented.

VST's evidence contradicts this assumption. VST-7B-RL achieves 61.2% on VSIBench (Table 3) and 44.2 AP@15 on SUN RGB-D 3D object detection (Table 4)—both competitive with or exceeding expert-augmented systems like VLM-3R-7B (which uses a specialized 3D reconstruction module)—while operating with a standard vision backbone and no additional parameters at inference. The model learns to extract metric depth, 3D object positions, and multi-view correspondences from the same RGB inputs that the base Qwen2.5-VL processed before tuning. This demonstrates that the ViT's patch-level features, when trained with the right supervisory signal, contain sufficient geometric information for 3D understanding—the bottleneck was the training data, not the architecture.

This insight is significant beyond the raw performance numbers because it changes the research direction: rather than designing new 3D-aware architectures (which fragment the model ecosystem and degrade general capabilities), the field should focus on constructing comprehensive spatial training data and curricula. It is a fundamental reframing rather than an incremental improvement—it shifts the problem from "what architecture do we need for spatial AI?" to "what do we need to teach standard architectures to make them spatially intelligent?"

The evidence that VST preserves general capabilities (Table 2: 83.0 on MMBench, 63.5 on MMStar for VST-7B-RL, comparable to the base model's 83.5 and 63.9) is essential to this argument. If spatial tuning destroyed general multimodal performance, the "compile it in" approach would be practically useless—you'd be trading one capability for another. The fact that both coexist validates the claim that spatial understanding is an additional competency that can be layered onto existing representations without overwriting them.

Innovation 2: The Perception-to-Reasoning Progression as a Principled Curriculum Mirroring Human Spatial Development

The paper's three-stage training pipeline (SFT → CoT Cold Start → RL) is not merely an engineering convenience—it embodies a specific cognitive hypothesis about the structure of spatial intelligence. By explicitly decomposing spatial ability into perception (Stage 1: "where is it in 3D?") and reasoning (Stages 2–3: "how do I think through a spatial problem?") and training them sequentially, the paper argues for a developmental hierarchy where foundational perceptual skills must precede and enable complex reasoning.

Prior work treated these as independent problems: dataset-centric approaches (SpatialVLM, SAT, MM-Spatial, SPAR) focused exclusively on perception through SFT, while CoT-based reasoning approaches (MVoT, SpaceR, MindCube, VILASR) assumed perception as a given and focused on reasoning. No prior work systematically validated that building perception first, then reasoning on top produces better outcomes than training both simultaneously or training reasoning alone. The paper's ablation in Table 10 provides evidence for the dependency: the baseline model (with perception training but no CoT data) scores 26.4 on MMSI-Bench; adding CoT training (Num-CoT) raises this to 29.2; but the gains come primarily from improved object-attribute reasoning, not spatial layout reasoning, suggesting that CoT training without robust perception foundations cannot fully address spatial reasoning deficits.

This is more than a training trick—it is a diagnostic framework for understanding where spatial failures originate. When a model fails on a spatial reasoning benchmark, VST's decomposition tells you whether the failure is perceptual (it can't extract the 3D layout from images) or reasoning-level (it has the layout but can't manipulate it mentally). This diagnostic capability is absent from prior work, which treats "spatial understanding" as a monolithic benchmark score. The paper's finding that the cold-start model performs better without CoT than with CoT (Table 11: 33.6 vs. 31.7 on MMSI-Bench) is a concrete example: the model has learned spatial knowledge from CoT training data but its reasoning chains are error-prone, so explicit CoT degrades performance. This distinguishes reasoning-chain quality from underlying spatial knowledge—a diagnostic that would be invisible if the two were trained jointly.

The connection to Piaget's developmental theory (cited in Section 1) elevates this from an empirical finding to a principled design philosophy. The claim is not just "this three-stage recipe works"—it is that spatial intelligence has an inherent hierarchical structure that training curricula should respect. This positions VST as a cognitive-science-informed approach to capability building, in contrast to the data-scaling philosophy of "throw everything at the model and hope it sorts out the dependencies." Whether the hierarchy is truly necessary (would joint training on perception + reasoning simultaneously achieve the same result?) is not tested, but the paper provides a coherent theoretical framework that makes testable predictions about transfer and interference.

Innovation 3: Bird's-Eye View Prompting as a General Strategy for Bootstrapping Teacher Models with Deficient Spatial Understanding

The paper identifies and solves a specific chicken-and-egg problem in spatial AI: to generate high-quality spatial reasoning training data, you need a teacher model with strong spatial understanding, but the reason you're generating training data in the first place is that current models lack strong spatial understanding. This is a bootstrapping problem: how do you create training data for capability X when no existing model reliably demonstrates capability X?

The solution—prompting with BEV annotation—is conceptually elegant: rather than asking the teacher model to infer spatial relationships from RGB images (which it does poorly), provide the spatial relationships as an explicit visual aid (the top-down BEV image rendered from ground-truth 3D boxes) and ask the teacher to describe and reason about what it sees. This separates spatial sensing (which the BEV provides) from spatial verbalization and reasoning structure (which the teacher provides). The teacher model's value is not in its spatial perception but in its linguistic fluency—it can generate natural-sounding reasoning traces that connect the abstract BEV representation to the first-person visual experience of the RGB images.

Prior CoT generation approaches for spatial reasoning (MVoT, SpaceR, MindCube) relied on the teacher model's own visual understanding to produce reasoning traces, which risks propagating the teacher's spatial errors into the training data. VST's BEV approach breaks this dependency: the spatial facts are guaranteed correct (derived from ground-truth 3D annotations), and the teacher only needs to package them into coherent language. This is a fundamental insight about capability decomposition in teacher-student training—when the teacher is weak on a sub-capability, offload that sub-capability to a reliable external source (in this case, geometric ground truth rendered as an image) rather than expecting the teacher to learn it.

The ablation evidence (Table 10: RT-CoT_BEV at 31.1 vs. RT-CoT at 30.0 on MMSI-Bench) shows that BEV prompting particularly helps on spatial relationship reasoning (object-object: 36.2 vs. 31.9; camera-object: 40.0 vs. 36.5), exactly the categories where accurate spatial layout knowledge matters most. This confirms that the BEV is providing information the teacher model cannot extract from RGB images alone, and that this information translates into measurably better training data. The technique is generalizable: any domain where ground-truth spatial/structural information exists but teacher models are weak could benefit from rendering that ground truth as an auxiliary visual channel during data generation.

Innovation 4: 3D Object Detection as a Trainable Text-Generation Task with a Differentiable Geometric Reward

The paper demonstrates that 3D object detection—a task traditionally handled by specialized computer vision architectures with regression heads, NMS post-processing, and dedicated loss functions—can be treated as a standard text-generation problem and optimized through reinforcement learning with a geometric reward. The VLM generates 9-DoF bounding boxes as JSON text strings, and the RL reward function (Eq. 3) computes the 3D IoU and F1 score between the parsed predictions and ground truth.

This is conceptually significant because it unifies spatial perception with language generation under a single framework. Rather than treating spatial tasks as "special" and requiring architectural exceptions, VST treats them as instances of the same structured prediction problem the VLM already solves for other domains (generating code, formatting JSON, producing structured answers). The model learns the "language" of 3D bounding boxes—the numerical conventions, coordinate system semantics, and relationship between visual appearance and metric spatial coordinates—through the same autoregressive objective and RL optimization used for natural language.

The reward design ablations (Table 13) reveal an important negative result with practical implications: using 3D IoU + recall as the accuracy reward causes "performance drops markedly because each ground-truth box is matched with too many false-positive predictions." This is a classic reward hacking failure mode: recall incentivizes the model to predict many boxes (maximizing the chance that each ground-truth object is matched with something), while the IoU component is too weak to penalize this behavior. Switching to IoU + F1 (which includes precision) fixes this by penalizing false positives. This finding is not specific to 3D detection—it illustrates a general principle for designing composite rewards in RL for structured prediction: ensure that reward components that incentivize coverage (recall) are balanced by components that penalize over-generation (precision). The paper's explicit documentation of this failure mode and its solution is a methodological contribution beyond the specific task.

The FoV unification strategy (Algorithm 1, Table 12 ablation) is another generalizable insight: when training a single model on heterogeneous sensor data, geometric standardization of the input space reduces the burden on the model to implicitly learn sensor-specific corrections. The 2.5 AP improvement on ARKitScenes when FoV is unified demonstrates that this matters in practice for datasets with high camera variation. This principle extends beyond 3D detection to any task where input data comes from sensors with different intrinsic parameters—multispectral imagery, medical imaging across scanner types, or multi-robot systems with heterogeneous cameras.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All spatial understanding experiments use a range of benchmarks spanning three modalities. For single-image spatial understanding, the primary benchmarks are CVBench (specifically its 2D and 3D splits, based on Cambrian-1 [60]) and 3DSRBench [41] (a comprehensive 3D spatial reasoning benchmark). For multi-image spatial understanding, BLINK [23] and MMSI-Bench [75] serve as the main benchmarks. For video spatial understanding, VSIBench [72] is the primary benchmark. General multimodal capability is evaluated using a standard suite: MMStar [12], MMBench [38], RealworldQA [66], MMMU [79], OCRBench [39], and AI2D [30]. The 3D object detection capability is specifically evaluated on the SUN RGB-D [54] validation set (using the Total3D split [46]) and the ARKitScenes [4] test set (using the Omni3D split [7]). The VLA evaluation uses the LIBERO benchmark [35], specifically the LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-10 task suites, with 50 independent rollouts per task.

• Base model(s). The paper uses Qwen2.5-VL [3] as the base architecture, conducting experiments at three scales: 3B, 7B, and 32B parameters. These models follow the standard "ViT-MLP-LLM" paradigm—a pretrained Vision Transformer combined with a large language model via an MLP projector. The 3B variant serves as the primary ablation testbed (Tables 5–7, 12–13), while the 7B and 32B variants are used for scaling experiments and final benchmark comparisons (Tables 2, 8–9). The choice is motivated by Qwen2.5-VL's strong performance on 2D visual recognition tasks coupled with its documented limitations in 3D spatial understanding, making it an ideal substrate for demonstrating the gains from spatial tuning. The authors cite the model's ability to "accurately identify objects and locate them in pixel space" while lacking 3D awareness as the specific rationale (Section 3, opening paragraph).

• Metrics. For spatial understanding benchmarks, accuracy is the primary metric—specifically, the percentage of questions answered correctly according to each benchmark's grading protocol. For CVBench, the paper reports separate 2D and 3D accuracy scores. For MMSI-Bench, overall accuracy is reported alongside fine-grained sub-scores for positional relationships (MSR, CC, OO, RR), attribute relationships (CO, OR, CR), and motion understanding (M, A, C, O). For VSIBench (Table 3), the paper reports overall average accuracy plus sub-scores on eight fine-grained dimensions: Object Count, Absolute Distance, Object Size, Room Size, Relative Distance, Relative Direction, Route Plan, and Appearance Order. For general multimodal benchmarks, standard evaluation protocols are followed: exact match accuracy for multiple-choice benchmarks, VQA accuracy [24] for open-ended visual questions, relaxed accuracy for ChartQA [43], ANLS for DocVQA [44], and word-level accuracy for TextVQA [53]. For 3D object detection, standard detection metrics are used: Average Precision (AP) at IoU thresholds of 0.15, 0.25, and 0.50 (denoted AP@15, AP@25, AP@50), and Average Recall for the top 100 predictions (AR@100). The paper also computes a composite "S-AVG" score as the arithmetic mean across CVBench (2D and 3D), 3DSRBench, MMSI-Bench, BLINK, and VSIBench, and a "MM-AVG" score as the mean across the six general multimodal benchmarks. For the LIBERO VLA evaluation, the metric is task success rate (% of rollouts achieving the goal). For RL accuracy rewards, the 3D detection reward uses a linear combination of mean 3D IoU and F1 score with α = 0.5 (Eq. 3).

• Baselines. The paper compares against a wide range of state-of-the-art models spanning proprietary systems, open-source generalist VLMs, and specialized spatial models. Proprietary baselines include GPT-4o [1], Gemini-2.5-Pro [17] (for MMSI-Bench), Gemini-1.5-Pro [57] (for VSIBench), Gemini-2.0-Pro (for 3D detection), Gemini Robotics-ER [56] (for 3D detection), and Seed1.5-VL [27] (for spatial understanding and 3D detection). Open-source generalist VLMs include LLaVA-OneVision-7B [33], Qwen2.5-VL-3B/7B/32B [3], InternVL3-8B and -38B [84], and MiMo-VL-7B-RL [58]. Specialized spatial baselines include SpatialVLM [11], SAT [49], MM-Spatial [19], SPAR-8B [82], SpaceR-7B [47] (which uses RL for video spatial reasoning), VLM-3R-7B [22] (which augments VLMs with 3D reconstruction modules), VILASR-7B [64] (which uses visual tools and prompting for spatial reasoning), and InternVL-Spatial-8B [21]. For 3D object detection, specialized expert systems Total3DU [46] and Implicit3D [81] serve as baselines. For the VLA experiments, the direct comparison is between Qwen2.5-VL-3B (without VST tuning) and VST-tuned Qwen2.5-VL-3B, both trained from scratch on the LIBERO benchmark without any robotic pretraining data.

• Generation budget / compute accounting. The paper does not explicitly report inference-time compute budgets (e.g., number of generated tokens, FLOPs per query) for benchmark evaluation. Models are evaluated in a standard inference setting: each benchmark question is presented to the VLM, which generates a single answer (optionally with chain-of-thought reasoning for the RL models). For 3D object detection RL, the reward computation involves generating 5 candidate responses per question (rollout number = 5, Table 17) during training, but evaluation uses a single generation. Training compute is reported in terms of hyperparameters (batch sizes, learning rates, sequence lengths, epochs) rather than total FLOPs. The key resource metric is training data scale: VST-P at 4.1M samples, VST-R at 135K samples, with controlled scaling experiments varying data volume by 1× vs. 3×. For the VLA experiments, training budget is measured in steps (2.5K vs. 10K).

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The paper evaluates on fixed benchmark test/validation splits as defined by each benchmark's standard protocol. Strategy selection (e.g., optimal CoT format, reward function design) is based on performance on the same benchmarks used for final evaluation, which technically constitutes a form of test-set optimization, though this is standard practice in the VLM benchmarking literature. The data scaling experiments (Tables 8–9) implicitly serve as a form of robustness check by showing consistent trends across model sizes and data scales. For ablation studies on single-image, multi-image, and video data (Tables 5–7), the incremental addition of data types with monotonic improvements provides a form of internal validation, though no formal statistical tests are applied.

Main Quantitative Results

Spatial Benchmark Performance (Table 2)

The headline result is that VST-7B-RL achieves leading performance on spatial understanding benchmarks while preserving general multimodal capabilities. On CVBench (spatial perception), VST-7B-SFT attains 85.5, surpassing the proprietary Seed1.5-VL (85.2) by 0.3 points, and VST-7B-RL further improves to 86.5. On MMSI-Bench (multi-image spatial reasoning), VST-7B-RL achieves 34.8%, a 2.8-point gain over VST-7B-SFT (32.0%), and approaching Gemini-2.5-Pro at 36.9%—notably, VST operates without any specialized 3D encoder. On VSIBench (video spatial understanding), VST-7B-SFT reaches 60.6%, and VST-3B-SFT already achieves 57.9%, both substantially ahead of GPT-4o at 34.0% (Table 2, and detailed sub-scores in Table 3).

When comparing model scales, the 3B variant achieves 57.9% (SFT) and 57.7% (RL) on VSIBench, while the 7B variant reaches 60.6% (SFT) and 61.2% (RL). The RL stage provides a consistent boost across scales: VST-3B improves from 30.2 to 31.3 on MMSI-Bench, VST-7B from 32.0 to 34.8. On general benchmarks, VST-7B-SFT scores 83.3 on MMBench, 84.9 on AI2D, and 63.1 on MMStar—comparable to the base Qwen2.5-VL-7B (83.5, 83.9, 63.9 respectively), confirming that spatial tuning does not substantially degrade general multimodal performance. The S-AVG metric (spatial benchmark average) shows VST-7B-RL at 63.4 (Table 8, 3× data), versus 56.7 for the base Qwen2.5-VL-32B, a 6.7-point improvement despite using a smaller model.

Fine-Grained VSIBench Analysis (Table 3)

The VSI-Bench results reveal that VST excels on specific fine-grained spatial sub-tasks. On Object Size estimation, VST-7B-RL achieves 75.5%, surpassing VLM-3R-7B at 69.2% (which uses an expert 3D encoder) by 6.3 points. On Room Size, VST-7B-RL reaches 69.2%, again exceeding VLM-3R-7B (67.1%). On Appearance Order (temporal reasoning about when objects enter the video frame), VST-7B-RL scores 69.2%, dramatically outperforming VLM-3R-7B (40.1%) by 29.1 points. On Relative Direction, VST-7B-RL scores 55.6% compared to VLM-3R-7B's 80.5%—a notable weakness that the paper does not explicitly discuss. On Absolute Distance, VST-7B-RL achieves 43.8% vs. VLM-3R-7B's 49.4%. These sub-scores reveal that VST's strengths are concentrated in metric estimation (size, room dimensions) and temporal ordering, while directional reasoning remains a relative weakness compared to the expert-encoder-augmented VLM-3R.

3D Object Detection Results (Table 4)

VST demonstrates strong monocular 3D object detection capability despite having no specialized detection architecture. On the SUN RGB-D benchmark, VST-7B-RL achieves 44.2 AP@15, surpassing the proprietary Gemini-2.0-Pro (32.5 AP@15) by 11.7 points and Seed1.5-VL (33.5 AP@15) by 10.7 points. It also outperforms specialized expert models: Implicit3D (24.1 AP@15) by 20.1 points and Total3DU (14.3 AP@15) by 29.9 points. VST-7B-RL is competitive with Gemini Robotics-ER (48.3 AP@15), a model specifically designed for embodied reasoning, lagging by only 4.1 points despite having no robotic pretraining or specialized 3D modules. The RL stage provides significant gains: VST-3B improves from 37.3 (SFT) to 40.1 (RL) AP@15 (+2.8), VST-7B improves from 41.6 to 44.2 AP@15 (+2.6). Across the full detection metrics on SUN RGB-D and ARKitScenes (Table 9), VST-7B with 3× data achieves 28.0 AP and 42.1 AR@100 on SUN RGB-D, and 39.1 AP and 54.3 AR@100 on ARKitScenes.

Data and Model Scaling (Tables 8–9)

Increasing model size consistently improves spatial benchmark performance. Moving from 3B to 7B with 1× data yields a 1.3-point S-AVG gain (60.6 → 61.9); moving from 7B to 32B yields a 1.7-point gain (61.9 → 63.6). Tripling the data scale (1× → 3×) provides additional gains at each model size: +1.1 points for 3B (60.6 → 61.7), +1.5 points for 7B (61.9 → 63.4), and +1.7 points for 32B (63.6 → 65.3). The interaction between model scale and data scale shows that larger models benefit more from additional data: the 32B model's 1.7-point gain from 3× data exceeds the 3B model's 1.1-point gain, suggesting that spatial understanding capacity scales with model size and that larger models can absorb more spatial training data without saturating.

For 3D object detection, the scaling picture is more nuanced (Table 9). Increasing model size from 3B to 7B with 3× data improves SUN RGB-D AP from 26.5 to 28.0 (+1.5) and ARKitScenes AP from 38.0 to 39.1 (+1.1). However, increasing from 7B to 32B with 3× data actually decreases performance: SUN RGB-D AP drops from 28.0 to 22.5 (-5.5), and ARKitScenes AP drops from 39.1 to 33.6 (-5.5). This is a striking non-monotonic scaling behavior. The authors attribute it to the possibility that "a model with 7B parameters is already sufficient to handle this fundamental perception task" (Section 4.3, scaling paragraph), but the magnitude of the drop suggests a more complex dynamic—possibly optimization difficulties at the larger scale or interference between the 3D detection objective and the 32B model's stronger pretrained priors. Data scaling alone (1× → 3×) provides substantial benefits: for 3B, SUN RGB-D AP improves from 20.2 to 26.5 (+6.3) and ARKitScenes AP from 31.5 to 38.0 (+6.5). For 7B, the corresponding improvements are 24.2 to 28.0 (+3.8) and 35.0 to 39.1 (+4.1).

Reinforcement Learning for Spatial Reasoning (Table 11)

The RL stage provides a 3.6-point improvement on MMSI-Bench when using chain-of-thought inference: 31.7 (cold start, with CoT) → 35.3 (RL, with CoT). The most dramatic gains are on Camera-Object (CO) relationships, which improve from 22.7 (cold start, with CoT) to 48.5 (RL, with CoT)—a 25.8-point gain. The Camera-Camera (CC) subset also shows substantial improvement: 44.2 → 54.6 (+10.4 points). On attribute-based subsets, the RL model without CoT (34.7 overall) slightly underperforms the cold-start model without CoT (33.6 overall) on positional relationships (MSR: 34.4 vs. 43.0), suggesting that while RL strengthens explicit reasoning chains, it may cause some regression on implicit spatial knowledge for specific sub-tasks when CoT is not used. The fact that the RL model with CoT (35.3) outperforms both the RL model without CoT (34.7) and the cold-start model with CoT (31.7) confirms that RL specifically improves the quality of the CoT reasoning chains, not just general spatial knowledge.

VLA Transfer Results (Table 14)

The VST-tuned VLM, when adapted to a VLA model and trained on the LIBERO benchmark, substantially outperforms the base Qwen2.5-VL-3B. At 2.5K training steps, the VST-tuned model achieves 61.7% average success rate versus 53.1% for the base model—an 8.6-point absolute improvement (16.2% relative). The gains are largest on LIBERO-Spatial (+8.4) and LIBERO-10 (+10.4), the task suites most dependent on spatial understanding. At 10K training steps, the advantage narrows but persists: 70.8% vs. 67.9%, a 2.9-point gain (4.3% relative). The diminishing gap with more training steps suggests that the base model eventually acquires spatial understanding from the robot-specific data directly, but the VST-tuned model achieves comparable or better performance with substantially fewer robot-specific training steps—the VST model at 2.5K steps achieves 65.0% on LIBERO-Spatial, exceeding the base model at 10K steps (76.0% vs. 78.4% at the same step count, but the 2.5K VST result is competitive with the 10K base result on certain suites).

Ablation Studies and Robustness Checks

Single-image data progression (Table 5): Incrementally adding spatial data types to the base model (Qwen2.5-VL-3B fine-tuned on 800K general samples) reveals the contribution of each data category. Starting from a baseline S-AVG of 49.9, adding 3D object detection data solely improves CVBench-3D by 5.7 points (72.6 → 78.3) but provides only 1.0 point overall S-AVG gain, indicating that 3D detection ability alone does not transfer to broader spatial tasks. Adding scene captions (single-image) provides a 2.3-point S-AVG boost (50.9 → 52.8), with particularly large gains on CVBench-3D (+5.5) and VSIBench (+2.2). Adding measurement data yields small but consistent gains across 3DSRBench (+0.8) and VSIBench (+2.4). The largest single addition is depth and distance data, which boosts S-AVG by 3.1 points (53.3 → 56.4), with CVBench-3D improving by 10.4 points and VSIBench by 3.2 points. The cumulative effect of all single-image data moves S-AVG from 49.9 to 56.4 (+6.5), with CV-3D nearly doubling from 72.6 to 93.4.

Multi-image data progression (Table 6): Building on the single-image baseline (S-AVG 56.4), adding multi-image data types yields incremental gains. Correspondence data improves BLINK by 1.8 points (50.6 → 52.4). Multi-image 3D object detection contributes 2.1 points on MMSI-Bench (28.8 → 30.0), validating the hypothesis that multi-view detection helps layout reasoning. Object-object relationship data provides the largest multi-image boost: +3.1 on MMSI-Bench (30.0 → 31.9) and +1.7 on BLINK (52.7 → 53.2 after object-object addition). Camera motion data adds 2.0 points on BLINK (53.0 → 55.0). Multi-image scene captions contribute a modest 0.3 points on MMSI-Bench (31.9 → 32.1). The cumulative multi-image effect raises S-AVG from 56.4 to 57.7 (+1.3), and notably has no impact on single-image benchmarks (CV-2D, CV-3D remain stable), confirming that multi-image data teaches complementary skills.

Video data progression (Table 7): Starting from the single-image + multi-image baseline (S-AVG 57.7 from Table 6's final row but note that the video baseline in Table 7 starts at 54.8—this discrepancy is explained by the video baseline using a different 1× vs. 3× data split), adding general video data from LLaVA-OneVision actually decreases VSI-Bench by 0.3 points (38.4 → 38.1), demonstrating that non-spatial video data does not transfer to spatial video understanding. Adding VST-specific video data provides a dramatic 16.6-point improvement on VSI-Bench (38.1 → 54.7), with stable performance on other benchmarks. This is the largest single-data-type improvement in the paper and confirms that video spatial understanding cannot be bootstrapped from general video understanding—it requires dedicated spatial video training.

FoV unification for 3D detection (Table 12): Removing the FoV unification strategy causes a 2.5 AP drop on ARKitScenes (29.4 → 26.9 using the baseline configuration, though Table 12 shows the baseline at 18.5 AP and w/o FoV at 18.6 AP on SUN RGB-D but drops on ARKitScenes). The asymmetric effect—negligible on SUN RGB-D, substantial on ARKitScenes—aligns with ARKitScenes' greater camera diversity (mobile device captures) versus SUN RGB-D's consistent capture setup.

Angle representation for 3D boxes (Table 12): Replacing Euler angles with quaternions in the 3D bounding box representation reduces SUN RGB-D AP from 18.5 to 18.3 (-0.2) and ARKitScenes AP from 29.4 to 28.6 (-0.8). While the differences are small, Euler angles consistently outperform quaternions, suggesting that the decomposable axis-angle representation is easier for the language model to learn, likely because it aligns better with the token-by-token autoregressive generation process where each angle component is predicted independently.

Multi-turn vs. single-turn formatting (Table 12): Replacing all multi-turn 3D detection data with single-turn data causes a 1.7 AP drop on SUN RGB-D (18.5 → 16.8) and a 1.8 AP drop on ARKitScenes (29.4 → 27.6). This is a substantial relative decrease (9.2% and 6.1% respectively), confirming that contextual learning of object co-occurrence and layout priors from multi-turn training contributes meaningfully to detection accuracy.

CoT generation strategy for spatial reasoning (Table 10): Comparing CoT data generation methods on MMSI-Bench: Numerical CoT (Num-CoT, which represents layouts as 3D boxes and performs numerical computation) achieves 29.2, outperforming the baseline (26.4) by 2.8 points. Text-based reconstruction CoT (RT-CoT) improves to 30.0 (+0.8 over Num-CoT). Adding BEV prompting (RT-CoT_BEV) further improves to 31.1 (+1.1 over RT-CoT). Expanding data diversity (RT-CoT_BEV with mixed data types) reaches 31.7. The largest sub-category gains from BEV prompting are on Object-Object (OO) relationships: 36.2 vs. 31.9 for RT-CoT, and Camera-Object (CO): 40.0 vs. 36.5—exactly the categories where accurate spatial layout information is most critical.

RL accuracy reward for 3D detection (Table 13): Ablating the reward function for 3D object detection RL reveals a critical finding. Using 3D IoU + Recall as the accuracy reward causes a catastrophic performance drop: AP falls from 20.2 (baseline SFT) to 13.8 (-6.4), with AP@15 dropping from 30.3 to 20.9. This negative result demonstrates reward hacking: the recall component incentivizes the model to predict excessive bounding boxes, and the IoU component provides insufficient counterbalancing signal. Replacing recall with F1 score (which includes precision) recovers and improves performance: AP rises to 24.4 (+4.2 over baseline, +10.6 over the recall variant). This ablation is methodologically significant—it demonstrates that the choice of RL reward function is the decisive factor in whether RL helps or catastrophically harms structured prediction tasks.

CoT vs. no-CoT inference after cold start (Table 11): After the CoT cold-start stage, the model performs worse with explicit CoT inference (31.7) than without it (33.6)—a 1.9-point gap. This suggests that the cold-start model has internalized spatial knowledge from the CoT training data but its reasoning chains are error-prone, leading to incorrect answers when the reasoning is made explicit. After RL, the relationship reverses: with CoT (35.3) now outperforms without CoT (34.7), a 0.6-point gap in the expected direction. This confirms that RL specifically improves reasoning chain quality, not just implicit spatial knowledge.

BEV prompting for CoT generation (Table 10, comparing RT-CoT vs. RT-CoT_BEV): The BEV annotation strategy provides a 1.1-point improvement on MMSI-Bench overall (30.0 → 31.1), with the gains concentrated in Positional Relationship sub-categories: OO improves from 31.9 to 36.2 (+4.3), CO from 36.5 to 40.0 (+3.5). On Attribute sub-categories, the effect is smaller or mixed: CR improves from 32.5 to 36.1 (+3.6), but OR drops from 32.8 to 26.6 (-6.2). This suggests that BEV prompting most benefits tasks requiring spatial layout reconstruction, while having inconsistent effects on attribute estimation tasks.

Data scaling for foundational tasks (Figure 5, Table 8 data scaling column): For 3D object detection (Figure 5a), AP@15 on SUN RGB-D and ARKitScenes increases progressively with data volume from 0K to 1600K samples, validating that the VLM can effectively absorb increasing amounts of spatial data. For depth-related tasks (Figure 5b), CV-3D performance improves from approximately 55% with 0K depth data to roughly 95% with 800K+ depth samples, approaching saturation at high data volumes. The S-AVG metric continues improving throughout the data scaling range for both tasks, while MM-AVG (general multimodal performance) remains stable, confirming that spatial data does not interfere with general capabilities even at high volumes.

RL platform choice (Table 11, implicit): The paper uses GRPO with group size 5 (Table 17: "Rollout number: 5") and no KL penalty. No ablation is provided comparing GRPO to alternative RL algorithms (PPO, DPO, rejection sampling), so the specific contribution of the GRPO formulation to the final performance cannot be isolated. The KL penalty disablement suggests the authors found it unnecessary, but no experiment demonstrates that adding a KL penalty would not further improve or stabilize training.

Critical Assessment

Does VST achieve state-of-the-art spatial understanding while preserving general capabilities?

This claim is partially supported with important caveats. On VSIBench, VST-7B-RL (61.2%) is competitive with VLM-3R-7B (60.9%), but VLM-3R uses a specialized 3D encoder, so VST's achievement of comparable performance without architectural modification is genuine evidence for the "spatial ability can be compiled in" thesis. However, VLM-3R substantially outperforms VST on Relative Direction (80.5% vs. 55.6%)—a specific sub-skill where expert 3D modules appear to provide non-trivial advantages that data-only approaches have not yet closed. On MMSI-Bench, VST-7B-RL (34.8%) is competitive with GPT-4o (30.3%) and approaching Gemini-2.5-Pro (36.9%), but absolute performance remains low—the best model is still incorrect on 65% of questions, indicating that spatial reasoning remains an open challenge that VST has not solved, only improved upon.

The "preserving general capabilities" claim is well-supported by the MM-AVG metrics in Table 2, where VST models maintain comparable scores to the base Qwen2.5-VL models. However, a more rigorous test would be to evaluate on benchmarks deliberately excluded from the training mixture to ensure the preservation is not an artifact of including similar data in the general multimodal training set (the paper uses LLaVA-OneVision data, which may overlap with benchmark distributions).

Does VST achieve effective 3D object detection from monocular inputs?

This claim is strongly supported on SUN RGB-D (Table 4). VST-7B-RL (44.2 AP@15) surpasses both proprietary VLMs (Gemini-2.0-Pro at 32.5, Seed1.5-VL at 33.5) and specialized expert systems (Total3DU at 14.3, Implicit3D at 24.1) by wide margins. The claim is weakened by the 32B scaling anomaly (Table 9) where the 32B model underperforms the 7B model on 3D detection, suggesting that the approach may not scale monotonically with model size—a concern for practical deployment at large scale.

A missing experiment: comparison to a dedicated monocular 3D object detector (e.g., Cube R-CNN, DETR3D, or recent transformer-based detectors) trained specifically on SUN RGB-D. The current baselines (Total3DU, Implicit3D) are relatively weak by modern standards (published in 2020–2021), so the comparison likely overstates VST's competitiveness with the computer vision state-of-the-art for 3D detection.

Does spatial tuning improve VLA performance?

This claim is supported by the LIBERO results (Table 14), but the evidence base is narrow: evaluation on a single benchmark (LIBERO) with a single base model (Qwen2.5-VL-3B), trained from scratch without robotic pretraining, for a single VLA adaptation method (action discretization following OpenVLA). The diminishing advantage at longer training (8.6% gap at 2.5K steps → 2.9% at 10K steps) raises the question of whether the benefit would asymptotically approach zero with sufficient robot-specific training. A more rigorous test would evaluate whether VST pretraining reduces the total training budget needed to reach a target success rate (sample efficiency) rather than just measuring performance at fixed step counts. The paper does not report learning curves, which would be necessary to quantify this efficiency gain.

Additionally, the paper does not compare against alternative spatial pretraining approaches for VLA—for instance, does pretraining on the VST data outperform simply pretraining on more robot-specific data (e.g., Open X-Embodiment)? Without this comparison, the claim that VST specifically (rather than additional pretraining of any kind) improves VLA performance is not isolated.

Does difficulty-dependent allocation explain VST's benchmark improvements?

The paper does not analyze difficulty-dependent performance (unlike the reference example paper on compute-optimal test-time scaling, which extensively bins questions by difficulty). Without difficulty-level analysis, we cannot determine whether VST's gains are uniform across easy/medium/hard spatial problems, or concentrated on easier problems while hard problems remain unsolved. The low absolute scores on MMSI-Bench (34.8% best) and certain VSIBench sub-categories (e.g., Absolute Distance at 44.4% for VST-7B-SFT) suggest that spatial reasoning performance is heavily right-skewed—models get easy questions right and hard questions wrong, and VST may primarily improve the easy-to-medium range without addressing fundamental capability limits on the hardest problems. This is a significant gap in the analysis that the reference example paper handled thoroughly.

Missing experiments and methodological concerns

Single model family: All experiments use Qwen2.5-VL variants. Without replication on a different VLM family (e.g., LLaVA, InternVL, Molmo), we cannot know whether VST's effectiveness depends on Qwen2.5-VL-specific architectural properties (e.g., its dynamic resolution handling, its particular ViT pretraining, its coordinate tokenization for 2D grounding).

No confidence intervals or statistical testing: Every result in Tables 2–14 is reported as a point estimate with no variance information. For MMSI-Bench (500 test questions) and the LIBERO benchmark (50 rollouts per task), sampling variance could easily account for 1–3 percentage point differences, making precise model comparisons unreliable. The paper does not report how many evaluation runs were performed or whether results are averaged across runs.

Benchmark overlap with training data: The paper does not analyze potential contamination between VST training data and the evaluation benchmarks. Since VST-P draws from ScanNet, ARKitScenes, SUN-RGBD, and other datasets, and the evaluation benchmarks (CVBench, 3DSRBench, BLINK, MMSI-Bench) may incorporate data from overlapping sources, there is a risk that some performance gains reflect memorization rather than generalization.

The 32B regression on 3D detection is unexplained: The 7B model achieves 28.0 AP on SUN RGB-D with 3× data; the 32B model drops to 22.5 AP (Table 9). This 5.5-point regression is not adequately investigated. Possible explanations (hyperparameter sensitivity at larger scale, interference between spatial and general knowledge, optimization instability) are not tested. This is a concerning result that undermines the "scalable paradigm" framing.

RL algorithm sensitivity: No ablation is provided comparing GRPO to alternative RL algorithms (PPO, DPO, best-of-N rejection sampling) or to simply using more SFT data. The 3.6-point improvement on MMSI-Bench from RL (Table 11) could potentially be achieved through additional SFT with higher-quality data, which would be simpler and more stable. Without this comparison, the value of the RL stage specifically (versus more SFT) is not established.

VLA transfer limited to one benchmark and one base model: Expanding to a second robot manipulation benchmark (e.g., CALVIN, RLBench) and a second base model would substantially strengthen the VLA transfer claim. The current evidence is suggestive but far from conclusive that spatial tuning is a general-purpose enhancer for embodied AI.

Difficulty estimation cost for the teacher model: The CoT data generation process uses Seed1.5-VL (a proprietary large VLM) with BEV prompting. The paper does not report the cost (inference time, monetary API cost, or engineering effort) of generating the 105K CoT samples, nor does it analyze how CoT quality scales with the teacher model's capability. If a weaker (cheaper) teacher model with BEV prompting can generate training data of similar quality, the approach is more practically scalable; if it requires the strongest available teacher, the data generation pipeline may be prohibitively expensive for broader adoption.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For, Making Headline Efficiency Gains an Upper Bound

The assumption or constraint. The paper claims VST achieves state-of-the-art spatial understanding without architectural modification, but the data generation pipeline—particularly the CoT reasoning traces and the BEV-prompted teacher inference—relies on access to ground-truth 3D bounding boxes, camera poses, depth maps, and scene graph annotations to construct training samples. For the CoT data specifically, the paper acknowledges using BEV annotation rendered from ground-truth 3D information as an "auxiliary spatial prompt" (Section 2.2) to compensate for the teacher model's limited spatial understanding. The VST-P dataset requires depth maps from professional 3D scanners (ScanNet++), synthetic simulators (Hypersim), and pseudo-labels from a depth expert model (Depth Anything V2); multi-image data requires calibrated multi-view captures with known camera extrinsics; and 3D detection data requires manually corrected axis-aligned bounding boxes from EmbodiedScan (Section 2.1, Single-image paragraph). This infrastructure is expensive to acquire, process, and standardize—costs that are externalized from the paper's training budget accounting.

The consequence. A practitioner attempting to replicate VST on a new domain or with a different base model would face a data construction bottleneck that the paper does not quantify. The VST-P dataset's 4.1M samples are generated through automated engines, but those engines require as input precisely the kind of annotated 3D data that is scarce outside of the specific academic datasets used (ScanNet, ARKitScenes, Hypersim, SUN-RGBD, Matterport3D, Objectron, and GRUTopia in Isaac Sim). For domains without existing 3D annotations—outdoor navigation, dynamic human environments, manufacturing floors, agricultural settings—constructing equivalent training data would require either prohibitively expensive manual annotation or the development of new domain-specific simulation and annotation pipelines. The paper's claim that VST avoids "extra overhead" (Section 1, paragraph 1) refers to inference-time architectural overhead, not to the data construction overhead, which is substantial and is the practical bottleneck for extending the approach.

Additionally, the CoT data generation with BEV prompting uses a proprietary large VLM (Seed1.5-VL) as the teacher model. The paper does not report the inference cost (FLOPs, wall-clock time, or API expense) of generating the 105K CoT samples, nor does it analyze how CoT quality degrades with cheaper teacher models. If the approach requires a state-of-the-art proprietary teacher to generate useful reasoning traces, the data generation pipeline may be inaccessible to researchers without access to those models, and the cost may dominate the total compute budget.

What evidence exists in the paper. The paper's data engines are described in Section 2.1 and Appendix C, but the cost of data construction is never quantified. The ablation in Table 10 shows that BEV prompting provides a 1.1-point improvement on MMSI-Bench over text-only reconstruction CoT (RT-CoT_BEV at 31.1 vs. RT-CoT at 30.0), so the BEV annotation is demonstrably valuable—but the cost of acquiring the ground-truth 3D annotations to render those BEV images is not reported. The paper notes that scene captions are generated using a "large VLM" (Section 2.1, Scene caption paragraph, identified as "a large VLM [27]"—Seed1.5-VL), and that the CoT engine uses the same teacher model with BEV prompting (Appendix C, Table 19). The paper does not report the number of inference calls, tokens generated, or computational cost of this teacher model usage.

Mitigation status. The paper does not address this limitation. It suggests no method for reducing the dependency on annotated 3D data, no analysis of how much data is "enough" (beyond the 1× vs. 3× scaling in Tables 8–9, which shows continued improvement with more data, implying the optimal data volume is not yet reached), and no strategy for bootstrapping spatial data generation from weaker teacher models or self-generated annotations. The paper's contributions are the data engines and training pipeline; the cost of running those engines is externalized from all reported metrics.

Hard Spatial Reasoning Problems Remain Largely Unsolved, Revealing a Fundamental Capability Ceiling

The assumption or constraint. The paper's VST framework improves spatial understanding by training on data that teaches the model to extract 3D information from 2D inputs and reason about spatial relationships. However, this approach can only amplify capabilities that are latent in the base model's architecture and pretraining—it cannot create fundamentally new representational capacities. On the hardest spatial reasoning tasks, where the base model's initial performance is near zero, VST provides minimal improvement, revealing a hard ceiling determined by the base model's intrinsic limitations rather than by the training data or methodology.

The consequence. For applications requiring robust spatial reasoning on difficult problems—multi-step navigation planning, complex multi-view scene reconstruction with significant occlusions, fine-grained relative direction inference across large viewpoint changes—VST does not provide a path to high reliability. The best VST model (VST-7B-RL) achieves only 34.8% on MMSI-Bench and 55.6% on the Relative Direction sub-task of VSIBench (Table 3). On the hardest sub-categories of MMSI-Bench, even after RL training, performance is low: Object Relationship (OR) achieves 33.8% with CoT inference, Camera Relationship (CR) reaches 31.6%, Attribute (A) is at 31.6%, and Object-only tasks are at 22.7% (Table 11, RL with CoT row). These are absolute performance levels where the model is wrong roughly two-thirds of the time or more—insufficient for safety-critical or autonomy applications. This is analogous to the reference paper's finding that test-time compute provides essentially zero benefit on the hardest difficulty quintile (bin 5): VST's spatial tuning, like compute-optimal test-time scaling, amplifies existing capability but cannot create capability where the base model has none.

This limitation is particularly consequential for the VLA transfer claim. The paper demonstrates that VST improves LIBERO benchmark performance by 8.6% at 2.5K steps (Table 14), but LIBERO tasks are relatively simple pick-and-place scenarios with clean tabletop environments and few distractors. In real-world robotic manipulation—cluttered scenes, deformable objects, transparent or reflective surfaces, dynamic obstacles—the spatial reasoning demands are substantially harder, and VST's inability to solve hard spatial problems in benchmark settings suggests that the VLA benefits may not generalize to more challenging embodied AI settings.

What evidence exists in the paper. The ceiling is visible across multiple benchmarks and sub-categories. On MMSI-Bench, the pre-training baseline (Qwen2.5-VL-7B fine-tuned on perception data) achieves 26.4 (Table 10); the best model after CoT cold start and RL reaches 35.3 with CoT inference (Table 11)—a 8.9-point absolute improvement, but still far below usable accuracy. On VSI-Bench Relative Direction, VST-7B-RL achieves 55.6% compared to VLM-3R-7B's 80.5% (Table 3), a 24.9-point gap despite VST's overall comparable average score (61.2 vs. 60.9). On 3DSRBench, the best VST model achieves 60.1 (Table 2, VST-7B-RL), meaning roughly 40% of spatial reasoning questions are still answered incorrectly. The 32B scaling experiment (Table 8) shows S-AVG of 65.3 with 3× data—a 4.1-point improvement over VST-7B-RL's 61.2 (extrapolating from Table 2, where VST-7B-RL achieves S-AVG component values that average to approximately 61.2)—suggesting that even quadrupling model parameters and tripling data provides diminishing marginal returns, consistent with a capability ceiling rather than a data or compute bottleneck.

Mitigation status. The paper does not acknowledge this ceiling or discuss strategies for overcoming it. The three-stage training pipeline is presented as a scalable paradigm (Section 6, "The generality, scalability, and effectiveness of VST highlight a promising direction toward building physical AI systems that reason and act in space with human-like intelligence"), but the results show that scaling (more parameters, more data) provides modest improvements on spatial benchmarks (Table 8: +1.3 from 3B to 7B, +1.7 from 7B to 32B with 1× data) and regresses on 3D object detection at 32B (Table 9). The paper does not investigate whether architectural innovations (e.g., higher-resolution visual encoders, explicit 3D representations, test-time compute strategies like multi-view rendering) could push through this ceiling, nor does it characterize which types of spatial problems are fundamentally beyond the reach of the current approach.

The 32B Model Regresses on 3D Object Detection, Undermining the Scalability Claim

The assumption or constraint. The paper positions VST as a "scalable paradigm" (Section 6) and presents data showing consistent improvements on spatial understanding benchmarks as model size increases from 3B to 7B to 32B (Table 8). However, this scaling behavior breaks down for 3D object detection, a foundational spatial perception task central to VST's claimed capabilities. The 32B model underperforms the 7B model on both SUN RGB-D and ARKitScenes, and in some configurations underperforms the 3B model as well. This non-monotonic scaling violates the expectation that larger models should absorb spatial training data more effectively, and the paper provides no investigation or explanation for this regression.

The consequence. If VST's spatial perception capabilities do not scale reliably with model size, the framework's applicability to the largest and most capable VLMs is uncertain. A practitioner choosing between deploying VST-7B vs. VST-32B faces a tradeoff the paper does not characterize: the larger model provides better general multimodal performance (Table 2: MM-AVG of 75.9 for 32B with 3× data from Table 8, vs. 73.3 for 7B with 3× data) and better spatial benchmark scores (S-AVG 65.3 vs. 63.4), but worse 3D object detection (SUN RGB-D AP 22.5 vs. 28.0 with 3× data, Table 9). For applications where 3D spatial perception is more important than broad multimodal reasoning—robotic grasping, autonomous navigation, AR object placement—the 32B model may be strictly inferior to the 7B model despite being 4.6× larger and presumably more expensive to serve. The regression also raises concerns about extrapolation: would a 70B or 100B model trained with VST improve 3D detection, plateau, or further regress? Without understanding the mechanism causing the 32B regression, there is no basis for predicting scaling behavior beyond 32B.

Potential explanations (not tested by the paper) include: (1) optimization instability at larger scales—the learning rate, batch size, or training duration that works for 7B may be suboptimal for 32B; (2) interference between the 3D detection objective and the 32B model's stronger pretrained priors—the larger model may have more entrenched 2D visual representations that resist the 3D fine-tuning signal; (3) the fixed training budget (1 epoch for SFT, Table 15) may be insufficient for the larger model to converge on the 3D detection task; or (4) the FoV unification and coordinate standardization strategies may interact differently with the 32B model's resolution handling.

What evidence exists in the paper. Table 9 is explicit: on SUN RGB-D with 3× data, the 3B model achieves 26.5 AP, the 7B model improves to 28.0 AP (+1.5), and the 32B model drops to 22.5 AP (−5.5 from 7B, −4.0 from 3B). On ARKitScenes with 3× data, the 3B model achieves 38.0 AP, 7B improves to 39.1 AP (+1.1), and 32B drops to 33.6 AP (−5.5 from 7B, −4.4 from 3B). The regression is consistent across both datasets and across all detection metrics (AP@15, AP@25, AP@50, AR@100). With 1× data, the same pattern holds but is less severe: 3B at 20.2 AP, 7B at 24.2 AP (+4.0), 32B at 19.6 AP (−4.6 from 7B). The paper notes this anomaly briefly: "increasing the model size from 7B to 32B, the performance does not exhibit a positive correlation as seen in the spatial understanding benchmarks. This may be because a model with 7B parameters is already sufficient to handle this fundamental perception task" (Section 4.3, scaling paragraph). This explanation—that 7B parameters "saturate" the task—is inconsistent with the large magnitude of the drop (a 5.5-point AP regression is not saturation, it is degradation) and with the fact that the 3B model outperforms the 32B model on SUN RGB-D with 3× data (26.5 vs. 22.5), meaning 3B parameters are more "sufficient" than 32B.

Mitigation status. The paper does not investigate the cause of the 32B regression beyond the brief speculative sentence quoted above. No hyperparameter sweep, learning rate tuning, or extended training experiment is reported for the 32B model. The authors do not discuss whether this regression is a fundamental limitation of VST's approach at large scales or an optimization artifact that could be resolved with better hyperparameters. This is a significant gap given that the paper's central thesis is that spatial understanding can be "systematically scaled" (Abstract, Section 6) through the VST framework.

The VLA Transfer Evidence Is Narrow: Single Benchmark, Single Base Model, Single Adaptation Method, and Diminishing Returns

The assumption or constraint. The paper demonstrates that a VST-tuned VLM, when adapted to a VLA model for robotic manipulation, outperforms the base VLM on the LIBERO benchmark (Section 4.4, Table 14). This evidence is used to support the broad claim that VST "paves the way for more physically grounded AI" (Abstract) and that "the Vision-Language-Action models have been proven to enhance visuospatial skills, enabling more grounded interaction with the physical world" (Section 6). However, the evidence is confined to a single benchmark (LIBERO), a single base model (Qwen2.5-VL-3B), a single adaptation recipe (OpenVLA-style action discretization with direct fine-tuning), and a single training setup (no robotic pretraining, only LIBERO fine-tuning). The claim of general VLA enhancement is supported by one data point with a specific experimental configuration that may not generalize.

The consequence. A practitioner considering whether VST pretraining would benefit their VLA system for a different robot, task domain, or base model cannot confidently extrapolate from the LIBERO results. The 8.6% improvement at 2.5K steps is impressive, but the diminishing gap at 10K steps (2.9%, Table 14) raises the question: does VST pretraining provide a permanent capability advantage, or does it merely accelerate early learning that would eventually be matched by robot-specific training alone? If the latter, the value of VST pretraining depends on the cost of robot-specific data collection relative to the cost of VST pretraining—a tradeoff the paper does not quantify. Additionally, the VST-tuned model is compared only to the untuned Qwen2.5-VL-3B, not to alternative spatial pretraining approaches (e.g., pretraining on the Open X-Embodiment dataset, or on domain-randomized simulated manipulation data). A practitioner cannot determine whether VST is specifically beneficial or whether any form of additional visual pretraining would yield similar VLA improvements.

The VLA adaptation also makes a specific design choice that may interact with the spatial tuning in unexamined ways: actions are discretized into 256 bins (Section 3.2) and predicted as text tokens. The VST training teaches the model to associate visual inputs with spatial language (3D coordinates, depth comparisons, layout descriptions); it is possible that the VLA benefit arises not from genuine spatial understanding but from the model being better at the format of structured numerical prediction—the same skill it learned for 3D bounding box regression transfers to action bin prediction, even if the spatial semantics are not genuinely understood. The paper provides no ablation to distinguish these mechanisms (e.g., pretraining on non-spatial structured output tasks vs. VST's spatial tasks).

What evidence exists in the paper. Table 14 reports success rates on four LIBERO task suites at two training budgets. At 2.5K steps: VST-tuned achieves 61.7% average vs. 53.1% for the base model (+8.6), with the largest gains on LIBERO-Spatial (+8.4) and LIBERO-10 (+10.4). At 10K steps: 70.8% vs. 67.9% (+2.9). The paper notes that "In contrast to the approach used by OpenVLA, we do not utilize any pre-trained data on robotic learning" (Section 4.4), meaning the comparison is between a model with spatial pretraining and a model with no additional pretraining at all. This is not a test of VST's specific spatial contribution—it is a test of whether any visual pretraining beyond the base VLM helps VLA fine-tuning. A fairer comparison would be VST-tuned vs. a model pretrained on an equivalent volume of general visual instruction data, which would isolate the spatial component. The paper includes no such control.

Additionally, the paper evaluates only Qwen2.5-VL-3B for the VLA experiment, despite having VST models at 7B and 32B. The larger models' VLA performance is not reported, so we cannot assess whether the VLA benefit scales with model size or is specific to the 3B scale. The LIBERO benchmark's standard evaluation uses 50 rollouts per task (Section 4.4)—with 4 task suites, that is 200 rollouts per configuration, which yields a standard error of roughly 3.5 percentage points for a 70% success rate. The 2.9% gap at 10K steps is within this margin of error, meaning we cannot reject the null hypothesis that VST provides no asymptotic VLA benefit at all.

Mitigation status. The VLA transfer is presented as a secondary contribution (Section 4.4, one table), not as the paper's central claim, so the limited scope is partly justified by the paper's focus on spatial understanding benchmarks. However, the paper's abstract and conclusion make strong claims about VLA enhancement ("paving the way for more physically grounded AI," "VLA models have been proven to enhance visuospatial skills") that are not proportionally supported by the evidence. The paper does not suggest the missing controls (general visual pretraining baseline, alternative base models, additional robot benchmarks) as future work, nor does it acknowledge the diminishing returns at longer training as a limitation of the current evidence.

The Paper Evaluates Only on Indoor, Static, Object-Centric Spatial Understanding; Dynamic, Outdoor, and Egocentric Navigation Scenarios Are Unexplored

The assumption or constraint. The VST dataset and evaluation benchmarks share a common implicit scope: indoor, static, object-centric spatial understanding. The training data comes from indoor RGB-D scans (ScanNet, ARKitScenes, Matterport3D) and indoor simulators (Hypersim, GRUTopia in Isaac Sim). The evaluation benchmarks assess spatial perception and reasoning about rooms, furniture, and household objects from discrete viewpoints. This scope excludes several categories of spatial understanding that are critical for real-world embodied AI: dynamic scenes with moving objects and people, outdoor environments with large-scale navigation and long-range depth, egocentric (first-person) continuous video with smooth camera motion rather than discrete viewpoint jumps, and tasks requiring metric action prediction in physical coordinates (the VLA experiment partially addresses this, but only for tabletop manipulation with a fixed third-person camera).

The consequence. A practitioner deploying VST for an outdoor robot (delivery drone, agricultural robot, autonomous vehicle), an AR/VR system that tracks a moving user through dynamic environments, or a human-robot interaction system where spatial relationships change continuously would encounter a domain gap that the paper does not characterize. The spatial skills VST teaches—estimating depth from a single indoor image, detecting 3D bounding boxes of furniture in camera coordinates, reasoning about static room layouts across discrete viewpoints—may not transfer to outdoor lighting conditions, large depth ranges (tens to hundreds of meters vs. indoor ranges of 1-10 meters), dynamic occlusion, or continuous ego-motion. The paper's emphasis on standardized camera coordinate systems (Section 2.1, FoV unification; Appendix A.1) and static scene reconstruction from discrete viewpoint pairs is well-suited to the indoor scanning paradigm but does not address the challenges of outdoor, dynamic, or large-scale spatial understanding.

This limitation matters because the paper's motivating applications (Section 1, paragraph 1) include autonomous driving and AR/VR, both of which involve outdoor/large-scale and dynamic spatial reasoning. The paper's evidence supports claims about indoor spatial understanding but provides no basis for extrapolating to these motivating applications. The VST framework's heavy reliance on ground-truth 3D annotations from specialized scanning hardware further limits outdoor applicability: the data engines require depth maps, 3D bounding boxes, and camera poses that are far more difficult and expensive to acquire for outdoor environments at scale.

What evidence exists in the paper. The datasets used for VST-P construction (Section 2.1) are exclusively indoor: ScanNet (indoor rooms), ScanNet++ (indoor high-fidelity), ARKitScenes (indoor mobile capture), Hypersim (indoor synthetic), SUN-RGBD (indoor rooms), Matterport3D (indoor buildings), Objectron (indoor object-centric), and GRUTopia in Isaac Sim (indoor robotics simulation). The evaluation benchmarks share this scope: CVBench (primarily indoor), 3DSRBench (indoor spatial reasoning), BLINK (indoor), MMSI-Bench (indoor rooms from ScanNet++), VSIBench (indoor video from ScanNet++), and SUN RGB-D / ARKitScenes for 3D detection (both indoor). The paper never evaluates on an outdoor spatial benchmark (e.g., KITTI for 3D detection, nuScenes for autonomous driving, StreetLearn for navigation) or a dynamic spatial reasoning benchmark. The video data in VST-P (2.1% of the dataset) comes from indoor scans and emphasizes appearance order and counting, not continuous motion prediction. The paper does not acknowledge this indoor/static/object-centric scope limitation.

Mitigation status. The paper does not discuss the indoor/outdoor domain gap or propose strategies for extending VST to outdoor or dynamic scenarios. The data engines are described as general pipelines, but their dependence on depth maps, 3D box annotations, and known camera poses makes outdoor extension non-trivial—outdoor depth estimation from monocular images is less reliable, 3D bounding box annotation for outdoor objects (vehicles, pedestrians, buildings at long range) is more ambiguous, and continuous ego-motion introduces additional coordinate system challenges. The paper's claim that VST "paves the way for more physically grounded AI" (Abstract) implies broader applicability than the indoor evidence supports, and the lack of discussion of this scope limitation could mislead practitioners about VST's readiness for outdoor or dynamic applications.

The 3D Object Detection Evaluation Overstates Competitiveness by Comparing Against Weak or Outdated Baselines

The assumption or constraint. The paper claims that VST achieves strong 3D object detection "even without auxiliary 3D encoders" and presents results (Table 4) showing VST-7B-RL at 44.2 AP@15 on SUN RGB-D, surpassing proprietary VLMs (Gemini-2.0-Pro at 32.5, Seed1.5-VL at 33.5) and expert systems (Total3DU at 14.3, Implicit3D at 24.1). This comparison is used to support the central thesis that VST's data-only approach matches or exceeds specialized 3D architectures. However, the "expert systems" baselines—Total3DU (Nie et al., 2020) and Implicit3D (Zhang et al., 2021)—are 4-5 years old at the time of VST's publication and are far from the state-of-the-art in monocular 3D object detection. Modern dedicated detectors on SUN RGB-D routinely achieve substantially higher performance.

The consequence. A practitioner looking to deploy a monocular 3D object detection system cannot determine from the paper whether VST's 44.2 AP@15 is competitive with, superior to, or substantially worse than a modern dedicated detection architecture. If a Cube R-CNN, DETR3D, or transformer-based monocular 3D detector achieves 55+ AP@15 on SUN RGB-D, then VST's approach—while impressive for a general VLM—is not a practical substitute for dedicated systems in applications where 3D detection accuracy is the primary requirement. The paper's framing (Table 4, "Expert Models" vs. "Proprietary Models" vs. "VST") implies that VST competes at the level of specialized systems, but the comparison is against outdated baselines that a modern detection paper would not use as primary competitors. This weakness is compounded by the fact that the paper evaluates 3D detection only on SUN RGB-D and ARKitScenes—both indoor datasets with limited object diversity and depth ranges. Performance on standard autonomous driving detection benchmarks (KITTI, nuScenes, Waymo Open Dataset) is not reported, so the generalizability of VST's detection capability to the most common 3D detection application domain is unknown.

The 32B regression (discussed in limitation 3 above) further weakens the detection claim: if VST's detection performance peaks at 7B and degrades at 32B, then scaling to larger models—the primary axis for improving VLM capabilities—is counterproductive for detection, suggesting a fundamental tension between spatial perception and general multimodal capability at scale that the paper does not address.

What evidence exists in the paper. Table 4 reports AP@15 comparisons on SUN RGB-D. The specialized expert baselines (Total3DU, Implicit3D) are cited from 2020 and 2021 respectively. The paper does not compare against any monocular 3D detector published after 2021, despite the rapid progress in 3D vision (DETR3D was published in 2021, PETR in 2022, BEVFormer in 2022, and numerous follow-ups). Table 9 reports more detailed detection metrics (AP, AP@15/25/50, AR@100) on both SUN RGB-D and ARKitScenes, showing VST-7B with 3× data achieves 28.0 AP on SUN RGB-D and 39.1 AP on ARKitScenes (using the AP metric without a specific IoU threshold—likely the COCO-style AP averaged over IoU thresholds, though the paper does not specify). These numbers provide a more complete picture but still lack comparison to modern detector performance.

The paper also includes single-image 3D object detection as a training task (Section 2.1, Single-image 3D engine paragraph), and the ablation in Table 5 shows that adding 3D detection data provides a 5.7-point improvement on CVBench-3D (72.6 → 78.3). This suggests that the detection training does transfer to broader spatial perception, but the detection capability itself is not benchmarked against competitively selected baselines.

Mitigation status. The paper does not compare against modern monocular 3D detectors and does not acknowledge the dated nature of the expert baselines. The choice of Total3DU and Implicit3D may be justified by the fact that they are also "general scene understanding" methods rather than pure detectors, making them conceptually closer comparators to a VLM-based approach—but this justification is not made in the paper. The omission of standard detection benchmarks (KITTI, nuScenes) means the detection capability is evaluated only on the same indoor data distribution used for training, providing no evidence of generalization to the outdoor driving domain where 3D detection is most practically impactful.

7. Implications and Future Directions

How This Work Changes the Landscape

The core conceptual shift: spatial understanding is a data problem, not an architecture problem. The paper's primary reframing is the demonstration that general Vision-Language Models can achieve competitive 3D spatial perception and reasoning without specialized depth encoders, point cloud processors, or voxel backbones. VST-7B-RL achieves 44.2 AP@15 on SUN RGB-D 3D object detection (Table 4) and 61.2% on VSIBench (Table 3)—both competitive with or exceeding expert-augmented systems like VLM-3R-7B (which adds a dedicated 3D reconstruction module)—using only a standard ViT-MLP-LLM architecture. This is more than an incremental improvement on benchmarks; it shifts the burden of proof from "VLMs need 3D encoders" to "show that 3D encoders provide capabilities that data-only approaches cannot match."

The magnitude of this shift should be calibrated carefully. This is not a paradigm shift on the order of the Transformer replacing recurrent architectures—the VST framework does not introduce a new model class or learning algorithm, and the base architecture (ViT + MLP + LLM) is identical to the pretrained model. Rather, it is a methodological reframing with significant practical consequences: prior work implicitly assumed that spatial understanding required engineering specialized perception modules; VST demonstrates that the bottleneck was the training data curriculum, not the architectural inductive biases. This redirects research investment from architecture design (how do we add spatial sensing to VLMs?) to data engineering (how do we construct training data that teaches VLMs to extract spatial information from standard visual inputs?).

The paper also reconciles a tension in the literature that was previously unacknowledged. Expert-encoder approaches (VLM-3R, Perception Tokens) demonstrated that adding 3D modules improves spatial benchmarks, suggesting that spatial understanding requires specialized components. Meanwhile, the generalist VLM community found that simply scaling models and data did not reliably improve spatial reasoning (GPT-4o scores 30.3% on MMSI-Bench vs. 65.1% on MMStar, per Table 2, a 35-point gap). These findings appeared contradictory—spatial ability seemed to require architectural priors yet resisted scaling. VST resolves this: spatial ability does not require architectural priors, but it does require explicitly spatial training data, which generalist pretraining pipelines do not provide. The resolution is that both camps had partial truth—the expert-encoder camp correctly identified that spatial understanding needs targeted intervention, while the scaling camp correctly identified that architectural modifications are not theoretically necessary. The missing ingredient was the data curriculum.

Research directions that become more attractive:

• Spatial data scaling laws: If spatial understanding can be "compiled into" standard VLM weights through data, then the natural next step is characterizing how spatial capability scales with data volume, data diversity, and model size. The paper's 1× vs. 3× scaling experiments (Tables 8–9) are a starting point, but a principled scaling law (analogous to Chinchilla for pretraining) for spatial data remains open.
• Cross-architecture transferability studies: The paper's findings are confined to Qwen2.5-VL. Testing VST on other VLM families (LLaVA, InternVL, Molmo) would determine whether the "spatial ability is teachable" claim generalizes or depends on Qwen2.5-VL-specific properties (e.g., dynamic resolution handling).
• Spatial capability probing: VST provides a framework for decomposing spatial understanding into perception and reasoning stages. This enables systematic probing: which sub-skills (depth estimation, multi-view correspondence, layout reconstruction, directional reasoning) transfer across architectures, and which are architecture-dependent?
• Curriculum learning for other "missing capabilities": The perception-to-reasoning progression (SFT → CoT cold start → RL) may apply to other capabilities where current VLMs are deficient but the pretraining data was not targeted—temporal reasoning, causal understanding, physical dynamics prediction. VST provides a template for data-driven capability injection.

Research directions that become less attractive:

• Architecture-first approaches to spatial AI: The paper's evidence that a standard VLM can match or exceed expert-augmented systems makes it harder to justify the additional complexity of dedicated 3D encoders unless those encoders can demonstrate capabilities that data-only approaches demonstrably cannot achieve. The RSIBench Relative Direction sub-task (Table 3), where VLM-3R-7B achieves 80.5% vs. VST-7B-RL's 55.6%, identifies one such capability gap that remains a valid motivation for architectural approaches. But the burden of proof has shifted: expert encoders must now show their value on specific sub-skills rather than claiming general spatial superiority.
• Modality-specific spatial pretraining: Approaches that train separate depth estimation, surface normal prediction, or point cloud completion models and then fuse their outputs with VLMs become less attractive when a single VLM can learn these capabilities end-to-end from appropriately constructed text generation training data. The integration engineering cost of multi-stage pipelines must now be justified against VST's simpler approach.
• Pure scaling as a path to spatial understanding: The paper's evidence that generalist VLMs (GPT-4o, Gemini-1.5-Pro) substantially underperform VST on spatial benchmarks despite being much larger confirms that parameter scaling alone does not close the spatial capability gap. Research programs betting on emergent spatial understanding from scaling general visual-linguistic pretraining should recalibrate expectations.

Follow-Up Research This Work Enables

1. Spatial data scaling laws: characterizing the relationship between spatial training data volume, diversity, and downstream capability. The paper's 1× vs. 3× experiments (Tables 8–9) show continued S-AVG improvement as data triples (+1.1 for 3B, +1.5 for 7B, +1.7 for 32B), but the scaling curves have not saturated. A systematic study would train VST models at 1×, 3×, 10×, and 30× data scales, measuring performance on spatial perception (CVBench, depth estimation, 3D detection) and spatial reasoning (MMSI-Bench, VSIBench) benchmarks. The critical question: does spatial capability follow a power-law relationship with data volume (suggesting continued investment in data construction is the optimal strategy), or does it asymptote at a level determined by the base model's representational capacity (suggesting that data alone hits a ceiling)? The 32B model's regression on 3D detection (Table 9: 7B at 28.0 AP vs. 32B at 22.5 AP on SUN RGB-D with 3× data) is the anomalous data point that a scaling law study must explain—is this an optimization artifact, an indication that larger models require different data mixtures, or evidence of a fundamental interference between spatial and general knowledge that scales with model size?

2. Cross-architecture replication to test the "spatial ability is teachable" claim. The paper's findings depend entirely on Qwen2.5-VL, which has specific properties that may facilitate spatial learning: dynamic resolution handling enables processing images at native resolution rather than fixed-size crops, and the model was pretrained with 2D grounding data (bounding box coordinate prediction) that may provide helpful priors for 3D coordinate prediction. A strong test would replicate VST on three architecturally distinct VLM families:

• LLaVA-OneVision (fixed-resolution ViT, MLP projector, no native grounding pretraining)
• InternVL3 (dynamic resolution, pixel shuffle compression, strong general vision pretraining)
• Molmo (point-based visual grounding pretraining, different coordinate tokenization)

The replication would train each architecture with VST-P + VST-R and evaluate on the full spatial benchmark suite (CVBench, 3DSRBench, MMSI-Bench, BLINK, VSIBench). A positive result (consistent gains across architectures) would establish VST as a general capability-injection technique independent of base model properties. A negative result (gains limited to Qwen2.5-VL or architectures with specific properties) would identify the architectural prerequisites for spatial learning—critical knowledge for practitioners choosing base models. The study should also compare VST against a baseline where each architecture is fine-tuned on an equivalent volume of non-spatial visual instruction data, to isolate the spatial-specific benefit from general fine-tuning effects.

3. The 32B regression root cause analysis: optimization artifact or fundamental scaling limit? The 32B model's 3D detection performance degrades relative to 7B (Table 9)—a 5.5-point AP drop on SUN RGB-D and ARKitScenes. This is the most concerning anomaly in the paper's results and demands investigation before VST can be confidently recommended for large-scale deployment. A targeted study would systematically test hypotheses:

• Optimization artifact: Sweep learning rates (5e-6, 1e-5, 2e-5 base LR for the LLM), train for 2–3 epochs instead of 1, and test whether the 32B model eventually matches or exceeds 7B performance with better-tuned optimization.
• Data mixture interference: Ablate the ratio of spatial to general data for the 32B model. If the larger model has stronger general priors that resist spatial fine-tuning, reducing general data or increasing spatial data proportion may recover performance.
• Resolution interaction: Test whether the FoV unification strategy (Algorithm 1) interacts poorly with the 32B model's dynamic resolution handling. If the 32B model processes images at a different effective resolution than 3B/7B, the unified FoV may produce geometric inconsistencies at larger scales.
• Task interference: Train separate 32B models on 3D detection only (without other spatial tasks) and on the full VST-P mixture. If single-task training recovers performance while multi-task training causes regression, this would indicate negative transfer between spatial tasks at larger scales.

The study's outcome determines whether VST is a "train at 7B, distill to larger models" technique or a genuinely scalable paradigm. If the 32B regression is an optimization artifact, resolving it opens the path to 70B+ spatial VLMs. If it is fundamental, the community needs to understand why spatial understanding interferes with scale and whether alternative training strategies (e.g., post-hoc adapter modules rather than full fine-tuning) can circumvent the issue.

4. Difficulty-binned analysis of VST's spatial reasoning gains. The reference paper on compute-optimal test-time scaling extensively analyzes how their method's effectiveness varies by question difficulty, revealing that gains are concentrated on easy-medium problems while hard problems show near-zero improvement. VST currently evaluates spatial reasoning at aggregate benchmark level, obscuring whether the 8.9-point MMSI-Bench improvement (26.4 → 35.3, Table 11) comes from saturating easy questions, improving medium questions, or making genuine progress on the hardest problems. A difficulty-binned replication would:

• Bin MMSI-Bench and VSIBench questions by the base model's (Qwen2.5-VL without VST) accuracy, creating 3–5 difficulty tiers.
• Report VST gains per tier.
• Characterize which spatial sub-skills (depth estimation, layout reconstruction, directional reasoning, metric estimation) contribute to difficulty—for example, do hard questions require precise metric depth (which VST trains for) or flexible multi-step spatial reasoning (which VST may not sufficiently address)?

The hypothesis based on the paper's ceiling evidence (low absolute scores on MMSI-Bench, 55.6% on VSIBench Relative Direction) is that VST's gains are skewed toward easy-medium questions, with hard questions remaining near baseline. If confirmed, this identifies the capability frontier where fundamentally new approaches (architectural innovations, test-time search, external tool use) are needed. If rejected (substantial gains on hard questions), VST's data-only approach would be shown to be more powerful than the ceiling evidence suggests.

5. Outdoor and dynamic spatial understanding: stress-testing the domain generalization of VST's spatial skills. The paper's training data and evaluation benchmarks are exclusively indoor, static, and object-centric. A critical extension would test whether VST-tuned models transfer spatial skills to outdoor, large-scale, or dynamic scenarios without domain-specific retraining. Concrete experiments:

• Outdoor depth and distance estimation: Evaluate VST models zero-shot on KITTI or nuScenes for monocular depth estimation and metric distance prediction. The indoor training data (depth ranges typically 0.5–10m) may not generalize to outdoor ranges (10–100m for driving), and outdoor lighting, texture, and object categories differ substantially. This tests whether VST teaches general spatial perception or indoor-specific spatial priors.
• Ego-motion and video spatial reasoning: Evaluate VST on Ego4D or EPIC-Kitchens for first-person continuous video spatial reasoning. VST's video data (2.1% of VST-P) comes from indoor scans with discrete viewpoint jumps; continuous ego-motion with motion blur, parallax, and dynamic objects is substantially harder.
• Large-scale navigation spatial reasoning: Construct or adapt a benchmark requiring reasoning about spatial relationships at building- or city-scale (e.g., "If building A is north of building B, and building C is east of building B, what direction is building C from building A?"). VST's training data operates at room-scale (objects within 1–10 meters); large-scale reasoning requires different spatial representations (allocentric rather than egocentric reference frames, metric rather than topological relationships).

A negative result (substantial performance drop from indoor to outdoor/dynamic/large-scale) would not invalidate VST but would establish its domain of applicability, which the paper currently fails to specify. A positive result (transfer to outdoor/dynamic scenarios) would dramatically strengthen the "general spatial intelligence" claim.

6. Self-improving spatial data generation: closing the loop between spatial perception and reasoning training. VST's data engines depend on ground-truth 3D annotations (depth maps, bounding boxes, camera poses) that are expensive to acquire. A natural extension is to use the VST-trained model itself to generate training data for further improvement:

• Stage 1: Train a base VST model on the existing VST-P and VST-R datasets.
• Stage 2: Use the trained model to predict 3D bounding boxes, depth maps, and spatial relationships on unlabeled in-the-wild images (e.g., COCO, web images, egocentric video datasets). Filter predictions using confidence scores or multi-view consistency checks to create pseudo-labeled spatial data.
• Stage 3: Retrain or continue training with the pseudo-labeled data, and measure whether this self-training loop improves spatial benchmarks beyond the initial VST performance.

This is analogous to STaR/ReST for reasoning, but applied to spatial perception. The critical question: does the VST model's spatial predictions on out-of-distribution images have sufficient accuracy to serve as training targets, or does noise in pseudo-labels cause degradation? The COCO-based depth data in VST-P (generated by Depth Anything V2, a specialized depth expert) already demonstrates that pseudo-labels can work, but uses a dedicated expert rather than the VLM itself. Replacing the expert with the model's own predictions would close the data dependency loop and test whether spatial understanding can be bootstrapped from minimal annotated data using self-supervision.

Practical Applications and Downstream Use Cases

1. Sample-efficient robotic manipulation with VLAs. The paper demonstrates that VST pretraining improves LIBERO success rates by 8.6% at 2.5K training steps (Table 14), with VST-tuned models at 2.5K steps achieving 61.7% average success vs. 53.1% for the base model. The largest gains appear on LIBERO-Spatial (+8.4%) and LIBERO-10 (+10.4%)—the task suites most dependent on understanding object positions and spatial relationships. For robotics practitioners, this translates to reduced data collection costs: achieving a target success rate with 4× fewer robot-specific demonstrations. In a production scenario where collecting 10K robot demonstrations costs tens of thousands of dollars in equipment time and human supervision, VST pretraining could reduce the required demonstrations to 2.5K while maintaining performance, directly reducing the cost of deploying a new manipulation skill. The practical integration path is straightforward: take an existing VLA pipeline (OpenVLA or similar), replace the base VLM with a VST-tuned checkpoint, and fine-tune on the target manipulation task with standard action discretization.

2. Indoor scene understanding for AR/VR and digital twins. VST's strong performance on monocular 3D object detection (44.2 AP@15 on SUN RGB-D, Table 4) and metric attribute estimation (Object Size: 75.5%, Room Size: 69.2% on VSIBench, Table 3) enables a VLM to produce structured 3D scene representations from single RGB images. For AR/VR applications, this means a user could point their phone camera at a room and receive a 3D layout with detected objects, their dimensions, and their spatial relationships—without requiring a depth sensor or specialized scanning hardware. VST-7B-RL outperforms the proprietary Gemini-2.0-Pro on 3D detection (44.2 vs. 32.5 AP@15) while running on open-weight models, making it deployable on-device for privacy-sensitive applications. The practical limitation is the indoor domain: VST's training data does not cover outdoor or industrial environments, so AR/VR applications in those settings would require domain-specific fine-tuning or would risk degraded performance.

3. Spatial QA for accessibility and navigation assistance. VST-7B-SFT achieves 60.6% on VSIBench (Table 3) with strong performance on Appearance Order (65.2%) and Object Count (72.0%)—capabilities relevant to assisting visually impaired users with spatial scene understanding from egocentric video. A system built on VST could process a continuous video stream from a wearable camera and answer natural language spatial queries: "How many chairs are in the room?", "Is the doorway to my left or right?", "Did anyone enter the room after I sat down?" The 29.1-point improvement on Appearance Order over VLM-3R-7B (65.2% vs. 40.1%) suggests VST is particularly strong at temporal spatial reasoning—tracking when objects appear or disappear—which is critical for dynamic scene understanding. The practical barrier is latency: VST models are standard transformer VLMs with no architecture-level optimization for streaming video, so real-time deployment would require engineering work to reduce inference time or to process keyframes rather than every frame.

4. 3D asset annotation and spatial data curation at scale. The VST data engines (Section 2.1, Appendix C) demonstrate that a large VLM with BEV prompting can generate high-quality spatial captions and reasoning traces when provided with ground-truth 3D annotations. A trained VST model can then perform the reverse: generate 3D bounding boxes, depth estimates, and spatial relationship descriptions from RGB images without ground-truth annotations. For companies building 3D asset libraries (e.g., for gaming, simulation, or e-commerce), a VST model could process thousands of product or environment images and produce structured spatial metadata—object dimensions, relative positions, camera-relative coordinates—reducing manual 3D annotation costs. The 3D detection AP@15 of 37.3–44.2 (Tables 4, 9) on indoor data is sufficient for coarse annotation where downstream human refinement is planned. The key integration decision is whether to use VST predictions directly as annotations (for applications tolerant of ~60% precision at IoU 0.15) or as pre-annotations for human correction (reducing annotation time by providing initial guesses).

When to Prefer This Method

The paper explicitly positions VST against two alternatives: (1) adding specialized 3D encoders or expert modules to VLMs, and (2) scaling generalist VLM pretraining without spatial-specific data. The decision rules emerge from the evidence:

• Prefer VST's data-only spatial tuning over adding 3D expert encoders when:

  • Preserving the standard VLM architecture is important for deployment simplicity (no additional inference modules, no increased per-query latency, no new dependency management).
  • Preserving general multimodal capabilities is critical—the paper shows VST-7B-RL maintains 83.0 on MMBench and 63.5 on MMStar (Table 2), whereas expert encoders "usually harms general capabilities" (Section 1).
  • The spatial tasks of interest are those where VST matches or exceeds expert-augmented systems: Object Size estimation (VST-7B-RL: 75.5% vs. VLM-3R-7B: 69.2%, Table 3), Room Size estimation (69.2% vs. 67.1%), Appearance Order (69.2% vs. 40.1%), and 3D object detection (44.2 vs. comparable expert systems, Table 4).
  • The target domain is indoor, object-centric spatial understanding—VST's training data distribution.

• Prefer adding 3D expert encoders over VST's data-only approach when:

  • The spatial sub-task of interest is Relative Direction estimation, where VLM-3R-7B achieves 80.5% vs. VST-7B-RL's 55.6% (Table 3)—a 24.9-point gap that suggests expert 3D modules provide genuine advantages for fine-grained directional reasoning.
  • The application requires spatial understanding in outdoor or large-scale environments where VST is not evaluated and where specialized modules pretrained on outdoor 3D data (e.g., from autonomous driving datasets) may provide stronger priors.
  • Inference latency is dominated by the VLM backbone regardless, so the marginal cost of an additional encoder is acceptable relative to the spatial capability gain on sub-tasks where VST underperforms.

• Prefer VST's targeted spatial training over scaling generalist VLM pretraining when:

  • The application domain requires reliable spatial understanding specifically—generalist models like GPT-4o (30.3% on MMSI-Bench, Table 2) and Gemini-1.5-Pro (45.4% on VSIBench, Table 3) substantially underperform VST-7B-RL (34.8% on MMSI-Bench, 61.2% on VSIBench) despite being larger and trained on more data.
  • The model size is constrained (e.g., on-device deployment), making it impossible to use the largest generalist models. VST-3B-SFT achieves 57.9% on VSIBench (Table 3), outperforming GPT-4o (34.0%), demonstrating that spatial capability at small model sizes requires targeted data.

• Prefer scaling generalist VLM pretraining when:

  • The application requires broad multimodal capabilities that VST does not address (document understanding, GUI agents, chart reasoning), and spatial understanding is only occasionally needed. The generalist models' strong average performance may outweigh their spatial weakness if spatial tasks are rare.
  • The inference budget supports using the largest models, and the spatial sub-tasks of interest are those where generalist models are already competitive (e.g., CVBench-2D, where Qwen2.5-VL-32B achieves 81.9 vs. VST-7B-RL's 86.5—a gap that may close with further scaling).