SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning

ArXiv: 2603.22057

🎯 Pitch

Injecting language-based spatial reasoning into a vision encoder like DINOv3 boosts semantic segmentation performance by nearly 4 points without any architectural changes to the downstream task head. This method extracts dense 3D knowledge from single images and converts it into multi-turn QA chains to teach the model hierarchical spatial understanding, which surprising also lifts performance on purely 2D tasks like ImageNet classification.

1. Executive Summary

This paper introduces SpatialBoost, a scalable framework that enhances the spatial awareness of existing pre-trained vision encoders by injecting 3D spatial knowledge expressed in linguistic descriptions through a Large Language Model. The approach uses a multi-turn Chain-of-Thought reasoning process that builds hierarchical spatial understanding across pixel-level, object-level, and scene-level question-answering pairs, then fine-tunes the vision encoder via dual-channel attention layers to acquire this spatial knowledge while preserving pre-trained representations. Applied to DINOv3 on ADE20K semantic segmentation, SpatialBoost improves mIoU from 55.9 to 59.7—a 3.8 percentage point gain that achieves state-of-the-art performance—while also boosting monocular depth estimation (RMSE dropping from 0.31 to 0.25 on NYUd), 3D scene understanding (SQA3D rising from 51.4% to 54.9%), and vision-based robot learning (average CortexBench score climbing from 72.8 to 80.8). The gains extend even to tasks not explicitly requiring spatial understanding—ImageNet linear probing accuracy rises from 88.4% to 90.2%—establishing that linguistic spatial supervision improves general vision capabilities without overfitting, though the hardest problems remain bounded by the base model's fundamental competence and the framework's reliance on vision models for 3D data extraction means performance inherits whatever errors those upstream models produce.

2. Context and Motivation

The fundamental problem SpatialBoost tackles is that large-scale pre-trained vision encoders—models like DINOv2, DINOv3, SigLIP, and OpenCLIP—are trained predominantly on 2D image data and therefore lack robust 3D spatial awareness. The paper observes in Section 1 that despite remarkable success across vision tasks including classification, segmentation, depth prediction, and vision-language understanding, these models "face a fundamental challenge in acquiring 3D spatial awareness capabilities." This manifests concretely in two ways:

Large vision-language models struggle to discern 3D spatial relationships between objects in images (Section 1, citing Fu et al. [30], Cheng et al. [19], Wang et al. [87]), meaning they cannot reliably answer questions like "Is the cup to the left of the book?" or "How far is the chair from the table?"
Vision-based robotic control systems that rely on these encoders demonstrate sub-optimal performance compared to approaches that directly utilize 3D information (Section 1, citing Ke et al. [48], Ze et al. [99], Zhen et al. [104]). A robot trying to grasp an object needs to understand its position in 3D space, not just its 2D pixel location.

This gap exists because the standard self-supervised learning paradigm—whether contrastive (SimCLRv2, MoCov3, DINOv2, iBOT) or mask prediction (BEiT, MAE)—operates entirely on 2D image data. These methods learn rich semantic features by distinguishing between different images or reconstructing masked patches, but neither objective inherently encodes depth, distance, relative position in 3D, or object occlusion relationships. As the paper puts it in Section 2.1, "these methods excel at capturing rich semantic features within 2D images" but "lack mechanisms to effectively encode 3D spatial knowledge."

Why This Matters: The Practical Consequences of Spatial Blindness

The paper's motivation extends beyond academic interest—the spatial awareness gap has concrete downstream impacts across multiple application domains where 3D understanding is critical:

Robotic manipulation and navigation: A disembodied vision encoder that can't distinguish "in front of" from "behind" or estimate object distances will produce representations that are fundamentally impoverished for robot control. The CortexBench results in Table 4 demonstrate this empirically: even state-of-the-art vision encoders like DINOv3 achieve only 72.8% average score across manipulation and locomotion tasks, leaving substantial room for improvement.
3D scene understanding: Tasks requiring spatial reasoning in 3D environments—answering questions about object positions that require integrating multiple viewpoints, localizing objects from textual descriptions, establishing geometric correspondences between partial views—all depend on representations that encode 3D structure. The Lexicon3D benchmark results in Table 3 show baseline encoders struggling on these tasks, with OpenCLIP achieving only 6.9 mIoU on 3D semantic segmentation.
Dense prediction tasks: Even 2D tasks like monocular depth estimation and semantic segmentation benefit from spatial awareness embedded in the representation, since depth prediction inherently requires estimating 3D structure from a single image and segmentation involves understanding object boundaries and occlusion relationships. The baseline results in Tables 1 and 2 show these tasks are far from saturated.

From a theoretical standpoint, the gap points to a fundamental limitation of current representation learning: models learn to see but not to perceive 3D structure. Closing this gap would mean that vision encoders capture a more complete model of the physical world, moving beyond 2D pattern recognition toward true spatial understanding—a prerequisite for robust embodied intelligence.

Prior Attempts and Where They Fall Short

The paper identifies three main families of prior approaches that attempt to address the spatial awareness gap, each with significant limitations.

Approach 1: Multi-View Training on Multiple Viewpoints

The most direct way to inject 3D spatial information into vision models is training on multi-view images that inherently encode spatial structure (Section 1 and Section 2.3, citing works like Dust3r [88], pixelsplat [12], and Monst3r [101]). These approaches have shown promise, particularly in robot control: methods like MV-MWM [74] use multi-view masked prediction, while others employ multi-view contrastive learning [75] to build representations that are viewpoint-invariant or viewpoint-aware.

Where this falls short: The limitation is scalability and data curation. As Section 1 states, these approaches' "broader applicability remains constrained by the need to use carefully curated data [97] or obtain multi-view datasets from simulation environments [73]." MVImgNet, the largest multi-view dataset, requires significant curation effort. Simulation data provides multi-view access but introduces sim-to-real domain gaps. This means multi-view training cannot easily be scaled to the massive, diverse datasets that have driven SSL's success, limiting its adoption as a general-purpose pre-training strategy for spatial understanding.

Approach 2: Explicit 3D Feature Prediction from Image Representations

Another line of work predicts 3D features directly from image representations during training (Section 2.3, citing Act3D [34], 3D Diffuser Actor [48], 3D Diffusion Policy [99]). These approaches train vision encoders to output representations from which 3D properties—depth maps, point clouds, object poses—can be decoded. This creates an explicit learning signal tying the representation to 3D structure.

Where this falls short: Like multi-view training, this approach is limited by the availability of 3D supervision data. More fundamentally, predicting 3D features from 2D representations is an ill-posed problem—infinitely many 3D configurations project to the same 2D image—so the learning signal is inherently ambiguous. The paper notes that these methods showed "significant performance improvements in vision-based robot control" but "such methods are limited by multi-view data, making it difficult to develop them into pre-trained models for general 3D understanding" (Section 2.3). They solve specific tasks well but don't generalize as broadly as desired.

Approach 3: Vision-Language Pre-training (CLIP-style and Beyond)

Multi-modal vision-language models like CLIP [67], SigLIP [100], OpenCLIP [20], and autoregressive variants (M3AE [33], iGPT [15]) learn representations by aligning images with text, providing semantic grounding that general-purpose SSL lacks. These models "successfully enrich visual representations with semantic context derived from natural language descriptions" (Section 2.2).

Where this falls short: Three specific limitations:

Joint pre-training from scratch is computationally prohibitive (Section 2.2): "existing models necessitate joint pre-training of both modalities from scratch, imposing significant computational demands and preventing efficient adaptation of existing pre-trained models." If you already have a strong pre-trained vision encoder, you can't easily add language alignment without retraining from scratch.
Text captions in standard datasets describe semantic content, not 3D spatial structure: A typical image caption says "a cat sitting on a chair" but rarely specifies the cat's 3D position, its distance from the camera, whether it's in front of or behind the chair, or the depth ordering of objects in the scene. So while CLIP-style alignment provides semantic knowledge, it doesn't directly address the spatial awareness gap.
The resulting models still fail at 3D spatial reasoning: Even large vision-language models trained with massive compute budgets struggle with basic spatial reasoning tasks, as documented by recent work cited in the paper's introduction [19,30,54,87]. This suggests that simply scaling up existing vision-language pre-training recipes won't automatically yield spatial understanding—the learning signal needs to explicitly encode spatial relationships.

Approach 4: Language-Based Improvement of Vision Representations

Several very recent works—AIMv2 [29], dino.txt [46], TIPS [61], and PE-Core [11]—have shown that language can serve as a scalable supervision signal for enhancing vision encoders after pre-training. These methods (which the paper includes as baselines) demonstrate that linguistic descriptions can improve visual representations without requiring joint training from scratch.

Where this falls short: These approaches "primarily focus on semantic understanding or object localization" (Section 1) rather than comprehensive 3D spatial understanding. They inject semantic knowledge (what objects are present, where they are in the 2D image) but not the rich 3D spatial relationships—relative depth, 3D distances, occlusion relationships, geometric structure—that SpatialBoost targets. The paper's experimental results in Tables 1–5 consistently show that while these methods improve over vision-only encoders, SpatialBoost achieves substantially larger gains on tasks requiring 3D understanding, suggesting they miss a significant portion of what language-based spatial supervision can provide.

The Missing Ingredient: Converting Explicit 3D Structure into Linguistic Supervision

The key insight that motivates SpatialBoost is a synthesis of two observations that, individually, were known but hadn't been combined:

Specialized vision models already extract 3D information from 2D images: Depth estimation models (like Depth Pro [10]) can produce metric depth maps. Segmentation models (like SAM [49]) can identify objects and their boundaries in 2D. 3D reconstruction models (like VGGT [86]) can build 3D point clouds from multi-view images. These tools exist and are reasonably accurate—they make it "possible to extend spatial information by modeling geometric relationships between objects in a scene" (Section 1).
Language can compose information sequentially and hierarchically: "since language naturally composes information in a sequential and structured form, this property allows the construction of labels that capture dense spatial relationships within a scene" (Section 1). Rather than providing a single spatial label (a depth value, a 3D coordinate), language can express rich, multi-step reasoning chains that build understanding progressively—from individual point depths to object positions to inter-object spatial relationships.

The synthesis is to use existing vision models as spatial knowledge extractors and then convert that extracted 3D information into structured linguistic descriptions that can serve as a training signal for the vision encoder. This addresses the key limitations of prior approaches:

No need for expensive multi-view curation: Single-view 2D images (which are abundant) can have 3D information extracted and expressed linguistically.
No joint pre-training from scratch: The method operates as a post-training fine-tuning step on existing vision encoders using an LLM decoder, preserving pre-trained knowledge.
Explicit spatial supervision: The linguistic descriptions capture 3D relationships directly, unlike standard image captions.
Scalable: The pipeline uses automated models for extraction and GPT-4o for question generation, enabling construction of large-scale spatial reasoning datasets from uncurated image collections.

How SpatialBoost Positions Itself

The paper positions SpatialBoost not as a replacement for existing SSL or vision-language pre-training, but as a complementary enhancement that can be applied to any pre-trained vision encoder (DINOv2, DINOv3, SigLIPv2, OpenCLIP, and even already spatial-aware encoders like TIPS and PE-Core) to inject 3D spatial knowledge without the cost of training from scratch. The experimental strategy demonstrates this explicitly: applying SpatialBoost to each baseline encoder and measuring improvements across a diverse set of tasks.

The three-stage training pipeline (feature alignment → visual instruction tuning → dual-channel attention fine-tuning) reflects this positioning. Stages 1 and 2 adapt a standard LLaVA-style multi-modal architecture to handle multi-view inputs and spatial questions. Stage 3 is where the actual spatial knowledge injection happens, using a carefully constructed multi-turn Chain-of-Thought reasoning dataset that "progressively incorporates dense spatial knowledge and builds hierarchical spatial understanding" (Abstract and Section 3.2).

The dual-channel attention mechanism—which adds a parallel attention branch initialized from pre-trained weights and blended via a learnable mixture parameter—is the key architectural innovation that allows SpatialBoost to add spatial knowledge without catastrophic forgetting. As the paper demonstrates in Figure 6 and Table 17, full fine-tuning or LoRA adaptation to spatial data causes significant drops in ImageNet classification accuracy (from 86.3% to 79.5% for full fine-tuning), while dual-channel attention both preserves and improves classification performance (87.6%).

The paper thus positions itself at the intersection of three research themes:

3D visual understanding: Addressing the long-standing gap between 2D-trained vision models and real-world 3D perception requirements.
Language as scalable supervision: Extending the recent wave of work using text-based training signals for vision, but specifically targeting spatial rather than semantic information.
Efficient model adaptation: Developing methods to enhance pre-trained models without re-training from scratch, using modular architectural additions (dual-channel attention) and targeted fine-tuning data.

3. Technical Approach

3.1 Reader Orientation

SpatialBoost is a three-stage training framework that takes an existing pre-trained vision encoder (like DINOv2 or SigLIP) and enhances its internal representations with explicit 3D spatial knowledge—depth ordering, relative object positions, inter-object distances—without requiring the encoder to be re-trained from scratch on 3D data. The system solves the problem that standard vision encoders, trained only on 2D images, lack robust awareness of how objects relate to each other in 3D space; its solution takes the "shape" of a data generation pipeline that extracts 3D structure from 2D images using off-the-shelf vision models (depth estimators, segmentors, 3D reconstructors), converts that structure into a multi-turn conversation of increasingly sophisticated spatial questions and answers, and then uses a Large Language Model as a decoder to provide a language-based training signal that backpropagates spatial understanding into the vision encoder through carefully designed adapter layers.

3.2 Big-Picture Architecture (Diagram in Words)

The SpatialBoost system has five main components connected in a training pipeline with a data generation front-end:

Spatial Knowledge Extraction Pipeline — A collection of frozen, specialized vision models (Depth Pro for metric depth estimation, SAM for object segmentation, VGGT for multi-view 3D reconstruction) that take single-view or multi-view 2D images as input and output 3D point clouds, object bounding cubes, and per-pixel depth values. This component runs once to generate the raw 3D metadata.
Multi-Turn Visual Spatial Reasoning Dataset Generator — A GPT-4o-based system that converts the extracted 3D metadata into 12-turn Chain-of-Thought question-answer sequences, organized hierarchically: 5 turns of pixel-level queries (point depths, depth comparisons), 4 turns of object-level queries (bounding cube prediction, relative left/right judgments), 1 turn of scene-level queries (inter-object 3D distances), and 2 turns of general scene captioning. This component produces the training data.
Multi-Modal Architecture (Vision Encoder + Projector + LLM) — A standard LLaVA-style architecture where the vision encoder $f_V$ produces visual features, a trainable 2-layer MLP projector $g_P$ maps those features into the LLM's token embedding space, and a frozen Qwen-2.0-7B LLM $f_L$ generates text answers from the combined visual and text tokens. Information flows: image → vision encoder → projector → concatenated with text tokens → LLM → generated answer tokens. The autoregressive language modeling loss on the answer tokens provides the training signal.
Dual-Channel Attention Layers — Adapter modules inserted into each attention layer of the vision encoder during Stage 3 training. Each consists of a second attention block $\text{Attn}^+$ initialized as an exact copy of the original $\text{Attn}$ , with a learnable per-channel mixture factor $\alpha$ that interpolates between the frozen original and the trainable copy. This is the mechanism that allows spatial knowledge to be added without catastrophic forgetting of pre-trained representations.
Three-Stage Training Scheduler — The orchestration logic that determines what gets trained and when: Stage 1 trains only the projector for feature alignment, Stage 2 trains the projector and LLM for visual instruction tuning (including multi-view VQA), and Stage 3 trains the vision encoder's dual-channel copies and the projector while keeping the LLM frozen. This staged approach ensures stable vision-language alignment before the computationally expensive vision encoder fine-tuning begins.

3.3 Roadmap for the Deep Dive

First, the formal problem setup and architecture: How the vision encoder, projector, and LLM are connected, what the autoregressive training objective looks like, and how language supervision flows back to the vision encoder. This establishes the basic machinery that all three stages share.
Second, the three-stage training pipeline in detail: What gets frozen and what gets trained in each stage, why this particular staging order is necessary for stable training, and the specific datasets and hyperparameters used at each stage. This explains how the system builds up from alignment through instruction following to spatial knowledge injection.
Third, the dual-channel attention mechanism: The mathematical formulation in Equation (1), why full fine-tuning and LoRA both fail (catastrophic forgetting), and how the copy-initialization plus learnable mixture factor allows gradual acquisition of spatial awareness while preserving semantic knowledge.
Fourth, the multi-view VQA dataset construction: How image pairs are selected via LPIPS filtering, what types of questions GPT-4o generates, and why this Stage 2 dataset is necessary before the spatial reasoning data can be effectively used in Stage 3.
Fifth, the multi-turn visual spatial reasoning dataset construction: The complete pipeline from raw image to 3D point cloud to hierarchical QA pairs, including the filtering criteria for single-view images, the LPIPS constraints for multi-view pairs, the specific templates and question types at each reasoning level, and why the forward hierarchical ordering (pixel → object → scene) is essential.
Sixth, the decoder choice and why LLM beats pixel-level alternatives: The ablation study in Table 6 showing that linear, SAM decoder, and VGGT decoder all cause catastrophic forgetting while LLM-based decoding preserves classification accuracy—explaining the mechanism by which language provides richer, more structured supervision than raw pixel outputs.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that linguistically expressed 3D spatial knowledge, when used as a training signal through an LLM decoder and dual-channel attention adapters, can inject spatial awareness into pre-trained vision encoders without disrupting their existing semantic capabilities.

The training framework operates on a dataset of image–conversation pairs. For each training instance, the input consists of an image $x$ (which may be a single view or a set of multi-view images $\{x_1, \ldots, x_N\}$ ) and a multi-turn conversation represented as a sequence of question-answer pairs $(x_q^1, x_a^1, \ldots, x_q^T, x_a^T)$ , where $T$ is the total number of conversation turns and each turn contains a question $x_q^t$ (a sequence of text tokens) and an answer $x_a^t$ (a sequence of text tokens). The conversation is constructed such that later turns build on information from earlier turns, implementing Chain-of-Thought reasoning.

The architecture consists of three modules connected in sequence:

A vision encoder $f_V$ that maps an image $x$ to a set of visual features $z_v = f_V(x)$ . This is the pre-trained model being enhanced (e.g., DINOv3-ViT-7B/16, SigLIPv2-ViT-g/16, OpenCLIP-ViT-G/14). The specific architecture and patch size vary by model, but all follow the Vision Transformer design with a [CLS] token and patch tokens.
A projector $g_P$ that maps the visual features into the token embedding space of the LLM. Following the LLaVA-1.5 architecture, this is implemented as a 2-layer MLP that transforms each visual token independently from the vision encoder's output dimension to the LLM's hidden dimension (3584 for Qwen-2.0-7B). The output is a set of "visual tokens" that can be concatenated with text tokens.
A Large Language Model $f_L$ , specifically Qwen-2.0-7B, that processes the concatenated sequence of visual tokens and text tokens autoregressively to generate answer tokens. The LLM operates as a decoder: it takes the full sequence of visual prefixes, question text, and previously generated answer tokens, and predicts the next answer token.

During training, the system processes a full multi-turn conversation by feeding each turn's question and the image into the model and computing the autoregressive loss on the answer tokens. For each turn $t$ , the input to the LLM is the sequence:

$\text{[visual tokens]} \circ \text{[question 1 text]} \circ \text{[answer 1 text]} \circ \cdots \circ \text{[question t text]}$

where $\circ$ denotes token-wise concatenation. The model is trained to predict the tokens of $\text{[answer t text]}$ given all preceding tokens. The loss for the full conversation is the sum over all turns of the per-token cross-entropy between predicted and ground-truth answer tokens.

This architecture choice—using an LLM rather than a task-specific decoder—is motivated by the LLM's ability to process long, structured text sequences in a single pass. The multi-turn spatial reasoning conversations can span thousands of tokens (12 turns of questions and detailed answers), and the LLM's transformer architecture naturally handles this sequential context. A pixel-level decoder (like a depth prediction head or segmentation head) would receive a single training signal per image (e.g., a depth map), while the LLM receives a dense, multi-step signal with explicit reasoning chains that teach the vision encoder progressively more sophisticated spatial concepts.

Three-Stage Training Pipeline

The training proceeds in three distinct stages, each with different parameter freezing configurations and dataset mixtures. This staged approach is necessary because directly training the full architecture on spatial reasoning data from initialization would face two problems: (1) the vision encoder's features would be poorly aligned with the LLM's embedding space, making gradients uninformative, and (2) the LLM has never been trained to process visual tokens, so it wouldn't know how to attend to them. The three-stage design solves alignment and instruction-following before attempting spatial knowledge injection.

Stage 1: Feature Alignment

In this stage, the goal is to train the projector $g_P$ to map visual features into a form that the LLM can interpret as meaningful tokens. The vision encoder $f_V$ and the LLM $f_L$ are both frozen. Only the 2-layer MLP projector $g_P$ is trainable.

The training data for Stage 1 is the BLIP-558K dataset, a collection of 558,000 image-caption pairs from the LLaVA project. Each training instance consists of an image and a single-turn conversation where the question is a generic prompt (e.g., "Describe this image.") and the answer is the image caption. The system feeds the image through the frozen vision encoder, maps the features through the trainable projector, prepends the visual tokens to the text tokens, and computes the autoregressive loss on the caption tokens.

Training hyperparameters for Stage 1: one epoch over the 558K samples, learning rate of $2 \times 10^{-3}$ , batch size of 256, trained on 4× NVIDIA Tesla A100s. The high learning rate relative to typical fine-tuning reflects that the projector weights are randomly initialized and need to quickly align two different embedding spaces.

After Stage 1, the projector can convert visual features into tokens that fall within the LLM's expected input distribution, but the LLM itself has not yet learned to reason about visual content—it only knows that these token sequences correspond to certain captions. Stage 2 addresses this.

Stage 2: Visual Instruction Tuning

In this stage, the goal is to train the LLM to follow instructions that involve visual content and, critically, to handle multi-view visual questions. The vision encoder $f_V$ remains frozen. Both the projector $g_P$ and the LLM $f_L$ are trainable.

The training data for Stage 2 is a mixture of two datasets:

LLaVA-Instruct-158K: A standard visual instruction tuning dataset containing 158,000 multi-turn conversations about single images, covering diverse tasks like detailed description, complex reasoning, and visual question answering. This provides general vision-language instruction following capability.
Multi-View VQA Dataset (newly constructed): 200,000 multi-view visual question-answer pairs generated by SpatialBoost's data pipeline (described in detail below). This dataset teaches the model to process pairs of images from different viewpoints and answer questions that require comparing or integrating information across views.

Training hyperparameters for Stage 2: one epoch over the combined dataset, batch size of 128, trained on 4× NVIDIA Tesla A100s. The learning rate is not explicitly specified for Stage 2 in the paper, but given that the LLM has 7B parameters and is being fine-tuned, typical LLaVA setups use learning rates on the order of $2 \times 10^{-5}$ .

After Stage 2, the full architecture can process both single-view and multi-view visual inputs and generate coherent text responses following instructions. The LLM has learned to attend to visual tokens and use them for answering questions. However, the vision encoder itself has not yet been updated—its representations are still the original pre-trained features. Stage 3 is where spatial knowledge is injected directly into the vision encoder.

Stage 3: Vision Encoder Fine-Tuning with Dual-Channel Attention

In this stage, the goal is to inject 3D spatial understanding into the vision encoder's representations while preserving its existing semantic capabilities. The LLM $f_L$ is frozen (to keep the language decoding capability stable), while the vision encoder $f_V$ and projector $g_P$ are trainable. However, the vision encoder is not fine-tuned conventionally—instead, dual-channel attention layers are added (described in the next subsection), and only the new attention channels and the projector are updated.

The training data for Stage 3 is the Multi-Turn Visual Spatial Reasoning Dataset (described in detail below), consisting of approximately 300K image–conversation pairs with 12-turn hierarchical spatial reasoning chains. Each training instance provides a dense training signal: the vision encoder must produce features that enable the frozen LLM to correctly answer pixel-level depth queries, object-level relative position queries, scene-level distance queries, and general scene description queries—all in sequence, with each answer building on information from previous turns.

Training hyperparameters for Stage 3: one epoch over the reasoning dataset, learning rate of $2 \times 10^{-5}$ (selected via hyperparameter search over $\{10^{-6}, 10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}\}$ ), batch size of 128, trained on 4× NVIDIA Tesla A100s. The LLM is frozen throughout.

The choice to freeze the LLM in Stage 3 is critical: if the LLM were trainable, it could learn to answer spatial questions by adapting its own internal reasoning, without forcing the vision encoder to produce better spatial features. By keeping the LLM frozen, the only way to reduce the loss is for the vision encoder to produce features from which the LLM can more easily generate correct answers—which requires the vision encoder to encode the 3D spatial information that the questions probe.

Dual-Channel Attention Mechanism

The dual-channel attention mechanism is the architectural innovation that enables SpatialBoost to add spatial knowledge without catastrophic forgetting. The problem it solves is this: if you directly fine-tune all parameters of a pre-trained vision encoder on a new dataset, the encoder will adapt to the new data distribution but lose its previously learned features—a phenomenon known as catastrophic forgetting. In the SpatialBoost setting, fine-tuning on spatial reasoning data causes the model to forget its general semantic understanding (as demonstrated in Table 17, where full fine-tuning drops ImageNet accuracy from 86.3% to 79.5%).

The dual-channel approach draws inspiration from the CogVideo architecture for text-to-video generation (Hong et al., 2023). The key idea is to add a parallel copy of each attention layer that can learn new patterns while the original layer preserves the old ones, then blend the two outputs with a learnable interpolation.

Formally, for each attention layer $\text{Attn}(\cdot)$ in the vision encoder, an additional attention layer $\text{Attn}^+(\cdot)$ is introduced. This new layer has the same architecture and input/output dimensions as the original. At initialization, its weight parameters are set to the exact same values as those of $\text{Attn}(\cdot)$ —it is a perfect copy. This is a crucial design choice: at the start of Stage 3 training, the output of $\text{Attn}^+$ is identical to the output of $\text{Attn}$ , so the overall network behavior is unchanged despite the additional parameter count.

The two attention outputs are merged using a learnable mixture factor:

$\text{Attn}^{\text{final}}(x) = \alpha \cdot \text{Attn}(x) + (1 - \alpha) \cdot \text{Attn}^+(x)$

where:

$x \in \mathbb{R}^{\text{seq\_len} \times d}$ is the input to the attention layer (a sequence of patch tokens and the [CLS] token, each of dimension $d$ ),
$\text{Attn}(x) \in \mathbb{R}^{\text{seq\_len} \times d}$ is the output of the original (frozen) attention layer,
$\text{Attn}^+(x) \in \mathbb{R}^{\text{seq\_len} \times d}$ is the output of the new (trainable) attention layer,
$\alpha = \text{sigmoid}(a) \in (0, 1)^d$ is a per-channel mixture weight, where $a \in \mathbb{R}^d$ is a learnable parameter initialized to the zero vector, and $d$ is the hidden dimension of the vision encoder.

What it computes: For each position in the sequence and each channel in the feature dimension, the final output is a weighted average of the frozen original attention output and the trainable new attention output. The weighting is learned independently per channel via the sigmoid of a learned scalar $a$ . When $a = 0$ , the sigmoid evaluates to 0.5, so the output is the unweighted average of both branches. As $a$ increases positively during training, $\alpha$ approaches 1, meaning the channel relies more on the frozen original; as $a$ decreases negatively, $\alpha$ approaches 0, meaning the channel relies more on the new learnable attention.

Why this form: The zero-initialization of $a$ ensures that $\alpha = 0.5$ at the start of training, giving both branches equal contribution. Combined with the copy-initialization of $\text{Attn}^+$ , this means the network's behavior at the start of Stage 3 is mathematically identical to the pre-trained network (since both branches produce the same output, their average equals either branch's output). As training proceeds, $\text{Attn}^+$ can diverge from $\text{Attn}$ to learn spatial features, and $\alpha$ can shift per-channel to select which representation is more useful for each feature dimension. Critically, the original $\text{Attn}$ remains frozen, so channels where $\alpha$ stays near 1 preserve their original pre-trained behavior. This per-channel granularity means the model can re-purpose some feature dimensions for spatial understanding while keeping others specialized for semantic classification.

The alternative of simple full fine-tuning overwrites all pre-trained features, losing semantic knowledge. LoRA (low-rank adaptation), evaluated in Table 17 and Figure 6, also causes classification accuracy drops (from 86.3% to 81.7%), likely because the low-rank constraint forces adaptation to spatial data to interfere with the directions in weight space that were encoding semantic features. Dual-channel attention avoids this by adding capacity rather than repurposing existing capacity.

The parameter overhead of dual-channel attention—an additional attention block per layer—increases model size by 25–30% depending on the encoder architecture (the paper reports 30% for OpenCLIP and SigLIPv2, 25% for DINOv2 and DINOv3). During Stage 3, only the $\text{Attn}^+$ parameters and the $\alpha$ mixing weights are updated; all original parameters (including the original $\text{Attn}$ , feed-forward layers, and layer norms) remain frozen. The projector $g_P$ is also trainable in Stage 3.

The paper validates the effectiveness of dual-channel attention through an ablation experiment (Table 17 and Figure 6). Three fine-tuning strategies are compared on DINOv2-ViT-L/14 with 100K spatial reasoning samples:

Full fine-tuning: All encoder parameters trainable. ImageNet accuracy drops from 86.3% to 79.5% (a catastrophic 6.8 percentage point decline), while ADE20K segmentation improves from 47.7 to 49.4 mIoU and depth estimation RMSE improves from 0.38 to 0.31. The segmentation and depth gains come at the cost of general vision understanding.
LoRA: Low-rank adapters inserted into attention layers. ImageNet accuracy drops to 81.7%, ADE20K improves to 49.0 mIoU, depth estimation to 0.32 RMSE. Better than full fine-tuning but still showing meaningful classification degradation.
Dual-Channel Attention (used in SpatialBoost): ImageNet accuracy increases from 86.3% to 87.6%, while ADE20K reaches 49.2 mIoU and depth estimation reaches 0.32 RMSE. This is the only method that preserves and enhances both spatial and semantic capabilities simultaneously.

The qualitative attention visualizations in Figure 7 provide additional evidence: the SpatialBoost DINOv3 attention maps show more focused and spatially structured attention patterns compared to the base DINOv3, suggesting the dual-channel layers have learned to attend to spatial relationships (e.g., attending to regions at similar depth or along object boundaries) while the original channels maintain their semantic attention patterns.

Multi-View VQA Dataset Construction (Stage 2 Data)

Before spatial reasoning can be injected into the vision encoder (Stage 3), the multi-modal architecture must first be capable of processing multi-view image inputs and following instructions about them (Stage 2). The existing LLaVA-Instruct-158K dataset provides general visual instruction tuning but is limited to single-view images. The multi-view VQA dataset bridges this gap.

The data sources are a combination of 3D scene datasets and ego-centric video datasets:

3D datasets: ScanNet (indoor 3D scans with RGB-D data), Mip-NeRF360 (360-degree captured scenes), and MVImgNet (a large-scale multi-view image dataset). These provide images captured from different viewpoints of static 3D scenes.
Ego-centric video: Ego4D, a massive dataset of first-person video capturing daily activities. Pairs of frames from the same video clip provide different viewpoints of the same scene.

From these sources, pairs of images are selected using the LPIPS (Learned Perceptual Image Patch Similarity) metric, a deep-learning-based perceptual distance measure. The constraint applied is:

$0.35 \leq \text{LPIPS}(x_i, x_j) \leq 0.65$

where $x_i$ and $x_j$ are two images from the same 3D scene or video clip.

What this constraint computes: LPIPS computes a perceptual distance between two images by feeding them through a pre-trained deep network (typically AlexNet or VGG) and comparing the internal feature activations. A value of 0 means the images are perceptually identical; higher values indicate greater perceptual difference. The range $[0.35, 0.65]$ selects image pairs that are substantially different (different viewpoints) but still clearly depict the same scene.

Why this range: Pairs with LPIPS below 0.35 would be nearly identical views, providing little multi-view training signal—the model could answer questions by looking at either image alone. Pairs above 0.65 would likely show entirely different scenes or extreme viewpoint changes where correspondence is ambiguous, making question generation unreliable. The $[0.35, 0.65]$ range selects the "sweet spot" where two views are clearly different but still recognizably the same scene, forcing the model to integrate information across views.

Given each selected image pair, GPT-4o generates three types of visual questions that are designed to probe general (not specifically spatial) knowledge from multi-view inputs:

Common VQA: Open-ended questions that require referencing details visible only when considering both views. Example: "What object is visible in the second image that is occluded in the first image?" These questions force the model to compare and contrast the two views.
Adversarial VQA: Questions designed to expose failure modes where a model might answer based on one view while the other view contradicts it. Example: "Is there a red cup on the table?" when one view shows a red cup and the other shows it was actually a red book from a different angle.
Multi-choice VQA: Questions with four answer choices, where only one is correct and the distractors are "very wrong" (explicitly designed to be clearly incorrect given both views). The correct answer requires integrating information from both images. For example, asking about the total number of a certain object that can only be counted by seeing both viewpoints.

The complete prompt templates used for GPT-4o generation are provided in Table 10 of the appendix. The system prompt establishes context: "Both images are came [sic] from same scene. When referring to the image, please call it the first image or the second image." Each VQA type has a specific prompt that instructs GPT-4o to create questions that require considering both views and to format answers appropriately.

The resulting dataset of 200K multi-view VQA pairs is mixed with the 158K LLaVA-Instruct samples for Stage 2 training. This mixture ensures the model maintains general single-view instruction following while additionally learning to process multi-view inputs.

The paper notes that constructing this multi-view VQA dataset was necessary because "existing datasets are limited to enhance multi-view understanding, as most VQA datasets focus exclusively on single-view scenarios" (Appendix B). Without Stage 2's multi-view instruction tuning, the LLM would not know how to attend to multiple visual token sequences when Stage 3's spatial reasoning data (which includes multi-view questions) is introduced.

Multi-Turn Visual Spatial Reasoning Dataset Construction (Stage 3 Data)

This is the critical dataset that provides the actual spatial knowledge injection in Stage 3. The construction pipeline has four phases: (1) image selection and filtering, (2) 3D point cloud extraction using specialized vision models, (3) hierarchical QA pair generation using templated questions and extracted 3D metadata, and (4) assembly into 12-turn conversations with Chain-of-Thought structure.

Phase 1: Image Selection and Filtering

Two image sources are used:

Single-view images: 100K images randomly sampled from the SA1B (Segment Anything 1 Billion) dataset, which contains diverse, high-resolution photographs. However, generating meaningful spatial reasoning data requires images with multiple objects at different depths and positions—a close-up of a single object or a screenshot of a UI won't work. Following the filtering approach of SpatialVLM and SpatialRGPT, a CLIP-based open-vocabulary classification model (EVA-CLIP) is used to score each image against positive labels ("an iPhone photo of an indoor scene", "an iPhone photo of an outdoor scene", "a DSLR photo of an indoor scene", "a DSLR photo of an outdoor scene") and negative labels ("a close up shot of a single object", "a product displayed in front of a white background", "an artwork", "a painting", "a screenshot of a graphical user interface", "a piece of text", "a sketch"). Images with higher positive scores and lower negative scores are selected. From 314K initial SA1B samples, 100K pass this filter.
Multi-view images: 200K samples from the same sources as the Stage 2 multi-view VQA dataset (3D datasets + ego-centric video). LPIPS filtering with the same $[0.35, 0.65]$ constraint is applied to select informative image pairs. Additionally, the paper extends beyond 2-view configurations: given a pair of "anchor views," interpolated frames between them are sampled, and GPT-4o is used to verify that the QA pairs generated for the anchor views remain valid from these new viewpoints. If the QA pairs are verified as correct for more than half the interpolated views, those views are incorporated. This yields 160K 2-view samples, 30K 4-view samples, and 10K 8-view samples in the final multi-view reasoning data.

The total dataset size for Stage 3 is approximately 300K samples (100K single-view + 200K multi-view), though the paper's experiments also ablate dataset sizes of 50K and 100K (as shown in Figure 5 and Table 18).

Phase 2: 3D Point Cloud Extraction

For each selected image (or image set), a 3D point cloud is constructed using specialized vision models. The choice of model depends on the input type:

Single-view images: The pipeline applies Depth Pro (a state-of-the-art monocular metric depth estimation model) to produce a per-pixel depth map in absolute metric units (meters). SAM (Segment Anything Model) is applied to produce object segmentation masks. By combining the depth map with the segmentation masks, the pipeline can compute the 3D position of each point within each object and extract approximate 3D bounding boxes (bounding cubes) for each segmented object. The depth values represent metric distances from the camera.
Multi-view images: The pipeline applies VGGT (Visual Geometry Grounded Transformer), a state-of-the-art 3D reconstruction model that takes multiple views of a scene and outputs a unified 3D point cloud in a canonical coordinate frame. This provides a more complete and accurate 3D representation than single-view depth estimation, since it integrates information from multiple viewpoints.

For both cases, object bounding cubes are represented in the canonical space defined by SpatialVLM, which provides a standardized way to express 3D object positions relative to a reference frame. A bounding cube is a 3D axis-aligned box described by its center coordinates $(x, y, z)$ and its extents $(w, h, d)$ in the canonical frame.

Phase 3: Hierarchical QA Pair Generation

Using the extracted 3D point cloud, object bounding cubes, and pixel-wise depth values, GPT-4o generates question-answer pairs at three hierarchical levels. The paper emphasizes that the questions are constructed using templates that query specific 3D metadata, ensuring that the answers are factually grounded in the extracted 3D structure rather than being hallucinated by the LLM.

Pixel-level (turns 1–5): Two types of questions probe the finest-grained spatial understanding:

Single-point depth queries: "What is the depth value at pixel point [A]?" where [A] is specified as 2D image coordinates $(u, v)$ . The answer is obtained by looking up the depth value at those coordinates in the extracted depth map. The depth value is rounded to three decimal places and, for values less than 0.5 meters, expressed in centimeters to maintain meaningful precision. Templates for questions include variations like "How far away is point [A]?" and "Tell me the depth of point [A]." Templates for answers include "[X] away." and "Depth value of point [A] is [X]."
Comparative depth queries: "Which point is closer to a viewer? Point 1: [A], Point 2: [B]." where [A] and [B] are two sets of 2D coordinates. The answer is determined by comparing the depth values at both points—the one with the smaller depth value is closer. Other question templates include "Is point [A] closer than [B]?", "Which point has a smaller depth value?", and "Compare the depth of point [A] and point [B]." The answers follow structured templates like "Yes, point [A] is closer to the viewer than point [B]." or "No, point [A] is not closer than point [B]." for the binary case.

The pixel-level queries serve two purposes: (1) they force the vision encoder to learn fine-grained depth estimation at arbitrary image locations, and (2) they establish the building blocks that later reasoning levels will reference. For example, an object-level question about relative left/right position implicitly depends on the depth values of the objects, which were probed at the pixel level.

Object-level (turns 6–9): Two types of questions probe object-centric spatial understanding:

Bounding cube prediction: "What is the center of the 3D bounding box coordinate for [A]?" where [A] is a text description of an object in the image (e.g., "the man wearing glasses"). The answer provides the 3D coordinates of the object's bounding cube center in the canonical frame. This requires the model to identify which object corresponds to the description, locate it in 3D space, and output its coordinates. Other templates: "Identify [A] and [B]" (outputting both objects' bounding cubes).
Relative position queries: "Is the [A] to the left of the [B] from the viewer's perspective?" where [A] and [B] are object descriptions. The answer is determined by comparing the $x$ coordinates of the two objects' bounding cube centers in the canonical frame (accounting for the viewer's perspective). Templates cover left/right, above/below, and in-front-of/behind relationships. Answers follow patterns like "Yes, the [A] is to the left of the [B]." or "No, the [A] is not to the left of the [B]."

Note that the object-level questions explicitly use the pixel-level spatial information as rationales. The QA templates include the bounding cube coordinates in the question context (e.g., "The man wearing glasses is located [0.47, 0.25, 0.59, 0.55, 0.38, 0.77] in the image.") so that the LLM's answer can reference these values as the reasoning chain. This is illustrated in Figure 2 of the paper, where the object-level QA about relative position shows the bounding cube data being included in the context.

Scene-level (turn 10): A single type of question probes holistic 3D understanding:

Inter-object distance queries: "What is the distance between the [A] and the [B]?" where [A] and [B] are object descriptions. The answer is the Euclidean distance between the two objects' bounding cube centers: $\sqrt{(x_A - x_B)^2 + (y_A - y_B)^2 + (z_A - z_B)^2}$ in the canonical frame, expressed in metric units (centimeters for small distances, meters for larger ones). This requires integrating the bounding cube coordinates of both objects—information that was probed at the object level—and computing the 3D distance. Other templates: "How far is the [A] from the [B]?", "How distant is the [A] from the [B]?", "Measure the distance from the [A] to the [B]."

The scene-level question is designed as the capstone that requires coherent 3D spatial understanding: the model must have correctly encoded the 3D positions of both objects (established at the object level), which in turn depends on correctly encoding their pixel-wise depths and segmentations (established at the pixel level). This hierarchical dependency is what implements Chain-of-Thought reasoning—each level builds on the previous one.

Scene Caption (turns 11–12): After the spatial reasoning chain, two additional turns are added where GPT-4o generates a general scene caption for the image. The question is a generic prompt like "Explain this scene." and the answer is a descriptive paragraph about the scene content. These turns serve to maintain general vision-language capabilities and prevent the model from overfitting to purely spatial features, as the paper explicitly notes in Section 3.2: "to enhance general knowledge of the vision encoder, we append GPT-generated scene captions after spatial reasoning turn."

Phase 4: Assembly into 12-Turn Conversations

The complete conversation for a single image (or multi-view set) consists of 12 sequential turns in the fixed order:

$Q^{(1)} \rightarrow A^{(1)} \rightarrow \cdots \rightarrow Q^{(5)} \rightarrow A^{(5)} \quad \text{(pixel-level, turns 1–5)}$ $\rightarrow Q^{(6)} \rightarrow A^{(6)} \rightarrow \cdots \rightarrow Q^{(9)} \rightarrow A^{(9)} \quad \text{(object-level, turns 6–9)}$ $\rightarrow Q^{(10)} \rightarrow A^{(10)} \quad \text{(scene-level, turn 10)}$ $\rightarrow Q^{(11)} \rightarrow A^{(11)} \rightarrow Q^{(12)} \rightarrow A^{(12)} \quad \text{(scene captions, turns 11–12)}$

Each turn's answer builds on the information from all previous turns. During Stage 3 training, the full 12-turn conversation is processed as a single training instance: the vision encoder produces features once (for the image), the projector maps them to visual tokens, and the LLM processes the concatenation of visual tokens, all previous question-answer tokens, and the current question to generate the current answer. The autoregressive loss is computed on all answer tokens across all 12 turns.

Why this hierarchical ordering matters: The paper's ablation in Table 7 demonstrates that the forward hierarchical order (pixel → object → scene) is crucial for performance. Three orderings are compared on DINOv2-ViT-L/14:

Forward (pixel → object → scene): ADE20K mIoU 48.9, NYUd RMSE 0.34, ImageNet accuracy 87.6.
Reverse (scene → object → pixel): mIoU 48.4, RMSE 0.35, accuracy 87.4.
Random (shuffled): mIoU 48.5, RMSE 0.36, accuracy 87.4.

Forward ordering achieves the best performance across all metrics. The paper's interpretation is that starting from fine-grained pixel-level understanding and building toward scene-level integration creates a natural curriculum: the model learns the basic building blocks first (pixel depths) and then progressively combines them into higher-level spatial concepts (object positions, inter-object distances). A reverse ordering would ask scene-level questions before the model has established pixel and object understanding, making the learning problem harder. The Chain-of-Thought structure mirrors how a human would reason about spatial relationships—first figure out where individual points and objects are, then reason about their relationships—making the training signal more aligned with the desired representational structure.

Dataset scalability: Figure 5 and Table 18 show that larger reasoning datasets yield consistent improvements. Training on 50K, 100K, and 300K samples with matched training iterations (one epoch for 300K, more epochs for smaller datasets to match total steps) shows monotonic improvements: DINOv3's ADE20K mIoU goes from 55.9 (pre-trained) → 56.8 (50K) → 58.3 (100K) → 59.7 (300K), and NYUd RMSE goes from 0.31 → 0.29 → 0.28 → 0.25. This suggests the approach benefits from more data and hasn't saturated at 300K samples.

Bias propagation analysis: A potential concern is that errors from the vision models used for 3D extraction (Depth Pro, SAM, VGGT) could propagate into the reasoning dataset and then into the fine-tuned encoder. The paper addresses this in Appendix E.5 through a controlled experiment on ScanNet, where ground-truth 3D annotations are available. Two versions of the reasoning data are generated—one using vision foundation model (VFM) outputs and one using ground-truth (GT) ScanNet annotations—and the resulting fine-tuned encoders are compared. The performance difference between VFM-based and GT-based training is negligible: on DINOv2, classification differs by 0.0%, segmentation by −0.1 mIoU, depth estimation by 0.0 RMSE, and vision-language reasoning by 0.0 BLEU-1. The paper concludes that "the effect of bias propagation is marginal in our reasoning data pipeline."

Why LLM-Based Decoding Beats Pixel-Level Alternatives

A central design choice in SpatialBoost is using an LLM decoder rather than a task-specific pixel-level decoder (like a depth head, segmentation head, or 3D reconstruction head) during Stage 3 fine-tuning. The ablation study in Table 6 provides the empirical justification for this choice.

The experiment compares five decoder configurations for fine-tuning a DINOv2-ViT-L/14 encoder on 300K spatial reasoning samples:

Linear (depth): A linear layer trained to predict depth maps from vision encoder features. During Stage 3, this linear layer is used as the decoder, and the vision encoder is fine-tuned with dual-channel attention to produce features that enable accurate depth prediction.
Linear (segmentation): A linear layer trained to predict segmentation masks.
SAM decoder: The full SAM (Segment Anything) mask decoder attached to the vision encoder via an MLP adapter. This is a sophisticated pixel-level architecture designed for segmentation.
VGGT decoder: The VGGT 3D reconstruction decoder, which takes vision encoder features and produces point clouds, camera poses, and depth maps. This is a state-of-the-art 3D prediction architecture.
LLM (SpatialBoost): The Qwen-2.0-7B LLM with a 2-layer MLP projector, trained with autoregressive language modeling on the spatial reasoning conversations.

All five configurations use dual-channel attention in the vision encoder and are trained on the same 300K data samples (or equivalent subsets for the pixel-level decoders). The results on downstream tasks reveal a stark pattern:

ImageNet classification (preserving pre-trained knowledge): The pre-trained DINOv2 achieves 86.3%. After fine-tuning, Linear (depth) drops to 85.7% (−1.39%), Linear (segmentation) improves slightly to 86.6% (+0.35%), SAM decoder stays at 86.3% (+0.0%), VGGT decoder drops to 84.8% (−1.74%), while LLM rises to 88.3% (+2.32%). Only the LLM decoder both preserves and improves classification.
ADE20K segmentation: Pre-trained: 47.7. Linear (depth): 47.9 (+0.42%), Linear (seg): 48.8 (+2.31%), SAM: 50.1 (+5.03%), VGGT: 45.6 (−4.40%), LLM: 51.5 (+7.97%). The LLM achieves the highest improvement, while VGGT actually degrades segmentation.
NYUd depth estimation (RMSE, lower is better): Pre-trained: 0.38. Linear (depth): 0.35 (−7.89%), Linear (seg): 0.45 (+18.42%, worse), SAM: 0.42 (+10.53%, worse), VGGT: 0.35 (−7.89%), LLM: 0.32 (−15.79%). The LLM achieves the best depth estimation.
ScanQA vision-language reasoning (BLEU-1): Pre-trained: 39.2. Linear (depth): 36.9 (−5.87%), Linear (seg): 37.1 (−5.36%), SAM: 37.6 (−4.08%), VGGT: 37.3 (−4.85%), LLM: 40.0 (+2.04%). All pixel-level decoders cause degradation in VQA capability, while the LLM improves it.

The pattern is clear: pixel-level supervision causes catastrophic forgetting of general vision capabilities (reflected in the classification and VQA drops), and in some cases (linear segmentation, SAM) actually worsens the specific capability being supervised (e.g., depth estimation from segmentation supervision). Language supervision through the LLM is the only approach that improves both spatial and non-spatial capabilities simultaneously.

Why does language supervision work better? The paper hypothesizes two mechanisms:

Density of information: A single language-based training instance (a 12-turn conversation with thousands of tokens) provides a much richer and more structured training signal than a single pixel-level label (a depth map or segmentation mask). The LLM's autoregressive loss provides gradients for every answer token, effectively giving the vision encoder feedback on dozens of spatial queries per image rather than a single output map. The language signal is "dense" in the information-theoretic sense—it constrains the representation along many dimensions simultaneously.
Semantic abstraction: Pixel-level decoders encourage the vision encoder to produce features that are optimized for a specific low-level task (predicting depth values at pixels, classifying pixels into categories). These features may discard the semantic abstraction that makes the representations useful for general vision tasks. Language-based supervision, by contrast, requires the encoder to produce features that support high-level reasoning about objects, their identities, their spatial relationships, and their scene context—a more holistic representation that naturally retains semantic information.

This finding is critical to the SpatialBoost approach: it's not just that language is a convenient medium for expressing spatial knowledge; language-based training through an LLM provides a qualitatively better learning signal for vision encoders than direct pixel-level supervision, even for spatial tasks. The LLM acts as a "semantic bottleneck" that forces the encoder to learn representations that are both spatially informative and semantically meaningful.

4. Key Insights and Innovations

Innovation 1: Language as a Scalable and Structured Medium for Injecting 3D Spatial Knowledge into Vision Encoders

The most conceptually distinctive move in this paper is the insight that linguistic descriptions can serve as a dense, structured training signal for 3D spatial understanding, and that this signal is qualitatively superior to direct pixel-level supervision. This is not an incremental refinement of existing vision-language alignment methods—it is a fundamental reframing of what language can contribute to vision training.

Prior work on language-supervised vision learning (CLIP-style contrastive alignment, image captioning, or recent methods like AIMv2, dino.txt, TIPS, and PE-Core) treats language as a source of semantic metadata—a way to tell the model what objects are present and what they're called. The resulting representations encode object identity and category-level properties, but they remain largely 2D. Even methods explicitly aimed at spatial awareness, like TIPS and PE-Core, primarily improve 2D localization and object boundary understanding rather than full 3D spatial reasoning (as the paper's baseline results in Tables 1–5 consistently show: TIPS achieves only 37.4 BLEU-1 on ScanQA and PE-Core achieves only 44.6% registration recall on geometric understanding, both substantially below SpatialBoost's numbers).

SpatialBoost changes the role of language from semantic label to structured representation of 3D physical relationships. The key insight is that language naturally composes information hierarchically—"naturally composes information in a sequential and structured form" as Section 1 puts it—and this compositional structure can be exploited to encode dense spatial relationships that would be impossible to express in a single scalar or vector label. A depth map tells you the depth at each pixel, but it doesn't tell you that "the man wearing glasses is 50 centimeters to the left of the man wearing the orange vest, and both are approximately 2 meters from the camera, and the distance between them measured in 3D is 0.5 meters." Language can express all of this in a single coherent sentence, with the relationships explicitly articulated rather than implicit in pixel values.

The significance of this reframing goes beyond the specific SpatialBoost implementation. It suggests that language is not just a communication medium but a structure-preserving encoding format for complex relational knowledge. If language can encode 3D spatial relationships, it could presumably encode other types of structured knowledge that are difficult to supervise with pixel-level labels—temporal dynamics, causal relationships, physical affordances, multi-step procedures. The paper doesn't explore these extensions, but the conceptual framework it establishes (extract structured knowledge from specialized models → express it hierarchically in language → use an LLM decoder to inject it into a vision encoder) is general and could be applied to any domain where specialized models can extract structure faster than we can annotate it.

The ablation study in Table 6 provides the key empirical anchor for this insight. When the same vision encoder is fine-tuned with pixel-level supervision (depth prediction, segmentation, or 3D reconstruction decoders), downstream classification accuracy drops—a clear signature of catastrophic forgetting. When fine-tuned with language supervision through an LLM, classification accuracy increases. This is not a marginal difference; it's a qualitative one. The LLM-based approach achieves +2.32% on ImageNet while the best pixel-level approach (SAM decoder) achieves only +0.00%. This suggests that language supervision, because it requires the encoder to support high-level reasoning rather than low-level pixel mapping, produces representations that are both more spatially aware and more semantically rich—a double benefit that pixel-level supervision cannot deliver.

This innovation also resolves a tension in the literature. Several recent works (AIMv2, dino.txt, TIPS, PE-Core) showed that language can improve vision representations, but the improvements were concentrated on semantic and localization tasks. The question of whether language could help with 3D spatial understanding specifically—depth estimation, relative position reasoning, inter-object distances—was open. SpatialBoost demonstrates that it can, and moreover that language-based spatial supervision can achieve gains that surpass dedicated spatial training methods.

Innovation 2: Hierarchical Chain-of-Thought Reasoning as a Curriculum for Representation Learning

The second conceptual innovation is the recognition that the structure of the training data's reasoning chain—not just its content—profoundly affects what the vision encoder learns, and that a forward hierarchical ordering (pixel → object → scene) constitutes an implicit curriculum that produces better representations than any other ordering.

This is not obvious from prior work. Chain-of-Thought reasoning has been extensively studied in the context of improving LLM outputs at inference time—prompting a model to "think step by step" produces better answers. But SpatialBoost uses CoT in a fundamentally different way: as a representation-learning curriculum, where the multi-step reasoning structure shapes the internal features of the vision encoder during training, not the decoder's generation process at inference time. The LLM decoder in SpatialBoost is frozen during Stage 3; it doesn't need to learn to reason. The reasoning chain exists to structure the training signal that flows back to the vision encoder.

The hierarchical design—5 pixel-level turns establishing basic depth and point relationships, 4 object-level turns building on those to reason about object positions, 1 scene-level turn integrating everything into holistic spatial understanding—implements a deliberate progression from local to global, from concrete to abstract. This mirrors how humans learn spatial reasoning: understanding where individual points are in depth comes before understanding where objects are in 3D space, which comes before understanding how objects relate to each other in 3D.

The ablation in Table 7 provides the empirical evidence that this ordering matters causally, not just aesthetically. When the same turns are presented in reverse order (scene → object → pixel) or shuffled randomly, performance on downstream tasks degrades: forward ordering achieves 48.9 ADE20K mIoU and 0.34 NYUd RMSE, while reverse achieves 48.4 and 0.35, and random achieves 48.5 and 0.36. These differences are modest but consistent across multiple tasks, and the pattern is exactly what a curriculum-learning hypothesis would predict—starting with easier, more fundamental concepts and building toward harder, more integrative ones produces better learning outcomes.

The deeper significance is that this demonstrates conversation structure as a design space for representation learning. Most vision-language training treats conversations as flat sequences of tokens; the multi-turn structure exists only to provide more context or to enable interactive capabilities. SpatialBoost shows that the semantic structure of the conversation—which concepts are introduced when, how they build on each other, what dependencies exist between turns—is a learnable parameter that can be optimized to improve representation quality. This opens up a new axis for designing vision-language training data that goes beyond simply scaling up the number of examples or improving their factual accuracy.

Notably, this innovation is not about a new architecture or loss function—it's about data curriculum design enabled by the unique properties of language as a training medium. You cannot easily design an equivalent curriculum with pixel-level supervision because pixel-level labels (depth maps, segmentation masks) don't have compositional structure; a depth map is a depth map regardless of what order you process its pixels. Language, by contrast, naturally supports sequential, hierarchical composition, making it possible to encode curricula directly in the training data format.

Innovation 3: The Dual-Channel Attention Mechanism as a General Solution for Catastrophic Forgetting in Vision Encoder Fine-Tuning

While parameter-efficient fine-tuning methods (LoRA, adapters, prompt tuning) are well-established in NLP and increasingly used in vision, the dual-channel attention mechanism introduced in SpatialBoost represents a conceptually distinct approach to the catastrophic forgetting problem that is specifically motivated by the geometry of the fine-tuning task.

The standard view in parameter-efficient fine-tuning is that the model should adapt to new tasks by modifying as few parameters as possible—LoRA adds low-rank matrices, adapters add small bottleneck networks, prompt tuning adds learnable prefix tokens. These approaches are motivated by parameter efficiency: by keeping most weights frozen, the model is less likely to overwrite pre-trained knowledge. But they share a common assumption: that new capabilities must be acquired by repurposing existing capacity in the weight space. The low-rank constraint in LoRA, for example, forces adaptation to use directions that are already present in the weight matrices.

Dual-channel attention takes a fundamentally different approach: add capacity rather than repurpose it. By creating an exact copy of each attention layer and then interpolating between the frozen original and the trainable copy, the mechanism allows the model to develop entirely new attention patterns for spatial reasoning while leaving the original patterns untouched and always accessible. The per-channel mixture factor \alpha provides fine-grained control: some feature dimensions can remain dominated by the original attention (semantic features), others can shift toward the new attention (spatial features), and still others can blend both.

The empirical comparison in Table 17 and Figure 6 makes the case clearly. Full fine-tuning causes catastrophic forgetting (ImageNet accuracy drops from 86.3% to 79.5%). LoRA causes less severe but still meaningful forgetting (81.7%). Dual-channel attention not only prevents forgetting but improves the pre-trained capability (87.6%). This is not just a better tradeoff—it's a qualitatively different outcome. The model becomes strictly better than it was before, adding spatial understanding without sacrificing anything.

The theoretical significance is that this provides evidence against the "capacity saturation" hypothesis—the idea that pre-trained models are already using all their representational capacity and that adapting to new tasks necessarily involves trading off old capabilities for new ones. If capacity were truly saturated, adding more parameters (as dual-channel attention does, increasing model size by 25–30%) wouldn't help because the new parameters would also become saturated after fine-tuning. The fact that dual-channel attention preserves original capabilities while improving new ones suggests that pre-trained models have underutilized representational capacity in the sense that they haven't learned certain types of features (spatial) not because there's no room, but because the training objective never incentivized learning them. Adding capacity through dual-channel attention provides dedicated "real estate" for these new features without displacing existing ones.

This has implications beyond SpatialBoost. It suggests that for many fine-tuning scenarios—especially those involving knowledge that is orthogonal to pre-training objectives (like spatial understanding for models trained on 2D semantic tasks)—adding capacity may be strictly preferable to repurposing existing capacity. The dual-channel mechanism provides a clean, general-purpose implementation of this principle that could be applied to other knowledge injection tasks.

Innovation 4: The Diagnostic Finding That LLM-Based Supervision Is Qualitatively Superior to Pixel-Level Supervision for Representation Learning

This innovation is primarily a diagnostic contribution rather than a methodological one: the controlled experiment in Table 6 provides compelling evidence that language-based training signals, decoded through an LLM, produce better vision representations than pixel-level training signals, even when the target capability is fundamentally spatial/pixel-level (depth estimation, segmentation). This finding challenges a natural assumption—that to teach a vision encoder about 3D structure, you should supervise it with 3D labels (depth maps, point clouds, segmentation masks).

The experiment is clean and well-controlled: same base encoder (DINOv2-ViT-L/14), same training data (300K samples from the spatial reasoning dataset), same fine-tuning strategy (dual-channel attention), and the only variation is the decoder head through which supervision flows. The results show a consistent pattern across four evaluation tasks:

Classification (a semantic task): LLM improves (+2.32%), pixel-level methods either degrade or don't help (range: −1.74% to +0.35%).
Segmentation (a semantic-spatial task): LLM achieves the largest gain (+7.97%), pixel-level methods achieve smaller gains or degradation (range: −4.40% to +5.03%).
Depth estimation (a spatial task): LLM achieves the best improvement (−15.79% RMSE), pixel-level methods show inconsistent results—unsurprisingly, the depth-supervised linear head also improves depth (−7.89%), but the segmentation-supervised methods actually worsen depth estimation (+10.53% and +18.42%).
Vision-language reasoning (a semantic-spatial task): LLM improves (+2.04%), all pixel-level methods degrade (range: −4.08% to −5.87%).

The fact that pixel-level supervision can degrade the very capability being supervised (e.g., segmentation supervision worsening depth estimation) is particularly telling. It suggests that pixel-level losses encourage the encoder to learn narrow, task-specific features that are brittle when transferred to related but distinct tasks—the classic overfitting-to-the-proxy problem. The LLM-based language loss, by contrast, seems to encourage more general, transferable spatial features, likely because the language-based queries span multiple spatial concepts (depth, position, distance, relative orientation) and require the encoder to support reasoning about all of them.

The broader implication is a reassessment of what makes a good training signal for representation learning. The field has largely assumed that for spatial tasks, spatial labels (depth, segmentation, 3D coordinates) are the ideal supervision. This paper suggests instead that language—specifically, richly structured language that expresses spatial relationships in compositional form—provides a superior signal, not because it's more precise (the extracted depth values have measurement error) but because it's more holistic and less susceptible to proxy overfitting. The LLM decoder acts as a bottleneck that forces the encoder to learn representations that support a wide range of spatial queries rather than optimizing for a single output format.

This finding also explains why SpatialBoost works better than methods that directly predict 3D features (like Act3D or 3D Diffusion Policy). Those methods optimize for reconstructing 3D structure from 2D images—a pixel-level objective—and may suffer from the same narrow specialization that the Table 6 experiment reveals. Language-based supervision, by requiring the encoder to support flexible, multi-turn reasoning about spatial relationships, produces representations that generalize better across tasks.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses multiple benchmarks spanning dense prediction, 3D scene understanding, robotic control, image classification, and image retrieval. For monocular depth estimation: NYUd (indoor RGB-D, test split) and KITTI (outdoor driving scenes, test split). For semantic segmentation: ADE20K (scene parsing, 20K training images) and Pascal VOC (object segmentation, training + validation split). For 3D scene understanding: the Lexicon3D benchmark, which provides a unified probing framework over ScanNet scenes, with specific evaluations on ScanQA and SQA3D (vision-language reasoning), ScanRefer (visual grounding), geometric correspondence (using partial scene registration), and 3D semantic segmentation (20-class ScanNet labels). For robot learning: four domains from CortexBench (Adroit for dexterous manipulation, MetaWorld for multi-task manipulation, DMControl for locomotion, Trifinger for dexterous manipulation). For image classification: ImageNet-1K validation set. For image retrieval: Oxford-Hard, Paris-Hard, Met, and AmsterTime datasets.
Base model(s). The paper applies SpatialBoost to four families of pre-trained vision encoders: OpenCLIP ViT-G/14 (2.1B parameters), SigLIPv2 ViT-g/16 (1.1B parameters), DINOv2 ViT-g/14 (1.1B parameters), and DINOv3 ViT-7B/16 (7B parameters—the strongest baseline). Additionally, the ablation studies use DINOv2 ViT-L/14 (304M parameters) for controlled comparisons. The authors also test on two spatial-aware vision encoders from prior work: TIPS (1.1B parameters) and PE-Core (1.9B parameters). The chosen models span a wide range of scales and pre-training objectives (language-aligned vs. purely vision-based), providing evidence that SpatialBoost's benefits are not specific to any single model family.
Metrics. The paper reports a diverse set of metrics tailored to each task:
- Monocular depth estimation: RMSE (Root Mean Square Error) between predicted and ground-truth depth values in metric units. Lower is better.
- Semantic segmentation: mIoU (mean Intersection over Union) across all classes. Higher is better.
- 3D vision-language reasoning: BLEU-1 score for ScanQA and SQA3D answer generation. Higher is better.
- Visual grounding: Accuracy (%) on the overall category of the ScanRefer dataset. Higher is better.
- Geometric understanding: Registration Recall (RR) at 0.05m RMSE threshold (higher is better) and Relative Translation Error (RTE) in meters (lower is better). RR measures the fraction of point cloud pairs where the estimated transformation yields RMSE below 0.05m. RTE is the average translational error across all pairs after registration.
- 3D semantic segmentation: Classification accuracy (%) and mIoU across 20 semantic classes. Higher is better.
- Robot learning: The mean of best performance across 5 evaluation runs. For Adroit, MetaWorld, and Trifinger, this is success rate (%). For DMControl, this is normalized score. The paper reports average across domains for each encoder.
- Image classification: Top-1 accuracy (%) on ImageNet-1K validation set, using either k-nearest neighbors (kNN) with cosine similarity or linear probing on the [CLS] token.
- Image retrieval: Global Average Precision (GAP) for the Met dataset and mean Average Precision (mAP) for Oxford-Hard, Paris-Hard, and AmsterTime. Higher is better.
Baselines. The paper compares against a comprehensive set of pre-trained vision encoders, organized into two categories:
- Vision-only trained encoders: V-JEPAv2 (a video-predictive self-supervised model). Included to assess whether video-based multi-view pre-training provides spatial benefits that SpatialBoost might overlap with or exceed.
- Vision-Language trained encoders: AIMv2 (autoregressive multi-modal pre-training), dino.txt (DINOv2 fine-tuned with text alignment), TIPS (Text-Image Pretraining with Spatial awareness), PE-Core (Perception Encoder—extracts features from intermediate ViT layers rather than final layer only), OpenCLIP (contrastive vision-language pre-training), and SigLIPv2 (sigmoid-based contrastive vision-language pre-training). These are the most relevant baselines since SpatialBoost also uses language-based supervision. The paper additionally evaluates against a variant called PE-Spatial (the spatial-aware version of PE-Core).
- The paper also includes a naive post-training baseline labeled "Simple FT": fine-tuning the vision encoder with its original pre-training objective on the same 300K spatial reasoning data used for SpatialBoost, but without the dual-channel attention mechanism or the LLM decoder. This isolates whether the gains come from additional training data or specifically from SpatialBoost's architecture and training methodology.
Generation budget / compute accounting. The paper does not use "generations" or FLOPs as a compute budget metric, since spatial knowledge injection is data-dependent rather than inference-time compute-dependent. Instead, the key resource measure is training dataset size (50K, 100K, or 300K multi-turn reasoning samples). For fairness, when comparing different dataset sizes, the paper matches total training iterations: one epoch for 300K data, proportionally more epochs for smaller datasets. All experiments are conducted on 4× NVIDIA Tesla A100 GPUs, with training time determined by the fixed one-epoch schedule. The dual-channel attention mechanism increases parameter count by 25–30% during Stage 3 training (30% for OpenCLIP and SigLIPv2, 25% for DINOv2 and DINOv3), which increases memory requirements but not the number of training steps. This parameter overhead is reported transparently in the implementation details.
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. For each downstream evaluation, hyperparameters (learning rate, number of output layers, feature concatenation strategy) are selected via grid search on a validation split of the respective benchmark, and the best configuration is used to report test-set performance. For robot learning experiments, the paper evaluates each configuration 5 times (trained with a pre-defined interval) and reports the mean of the best performance. Standard deviations are reported for individual CortexBench domains (Table 4 shows e.g., DINOv3: 63.9 ± 1.5 on Adroit) but not for aggregated averages. For the dataset scalability experiments (Figure 5, Table 18), the paper varies dataset size while holding total training iterations constant through epoch adjustment, ensuring that compute spent on training is comparable across dataset sizes.

Main Quantitative Results

Dense Prediction Tasks: Monocular Depth Estimation

Table 1 reports RMSE on NYUd and KITTI benchmarks under two probing protocols: training a linear head on frozen encoder features ("lin.") or training a DPT (Dense Prediction Transformer) head, which uses features from multiple layers and is standard for depth estimation. Lower RMSE indicates better depth prediction.

The headline result: SpatialBoost consistently reduces depth estimation error across all four base encoders, under both probing protocols, on both benchmarks. On the strongest baseline—DINOv3 with DPT head on NYUd—SpatialBoost reduces RMSE from 0.31 to 0.25, a 19.4% relative improvement. On KITTI with DPT head, DINOv3 RMSE drops from 2.02 to 1.84, a 8.9% relative improvement.

The pattern is consistent across weaker encoders: OpenCLIP (the weakest spatial model) improves from 0.53 to 0.40 RMSE on NYUd linear probing, a 24.5% reduction—the largest relative gain among all encoders, suggesting SpatialBoost is particularly effective when the base encoder has minimal spatial awareness. SigLIPv2 improves from 0.51 to 0.39 (23.5% reduction) under linear probing, and DINOv2 improves from 0.37 to 0.30 (18.9% reduction).

Comparing against vision-language trained baselines: AIMv2 achieves 0.33 (linear) and 0.28 (DPT) on NYUd, while DINOv3 + SpatialBoost achieves 0.25 (linear) and 0.21 (DPT)—substantially better. PE-Core, a method specifically designed to extract rich features from intermediate ViT layers, achieves 0.25 (linear) and 0.23 (DPT) on NYUd; SpatialBoost matches its linear performance and improves DPT performance (0.21 vs. 0.23). TIPS achieves 0.35 (linear) and 0.31 (DPT), which SpatialBoost significantly exceeds.

A notable detail: the KITTI benchmark—which contains outdoor driving scenes at longer ranges and more complex geometry than NYUd's indoor scenes—shows more modest relative improvements (e.g., DINOv3 DPT: 2.02 → 1.84, only 8.9% relative). This may reflect that monocular depth estimation at long ranges is fundamentally more difficult, or that the spatial reasoning dataset (constructed primarily from indoor and close-range images) provides less transfer to driving scenarios.

Dense Prediction Tasks: Semantic Segmentation

Table 2 reports mIoU on ADE20K and Pascal VOC under two protocols: linear probing ("lin.") and multi-scale evaluation ("+ms") which uses features from the last four layers of the vision encoder. Higher mIoU indicates better segmentation.

The headline result: SpatialBoost improves semantic segmentation consistently, with DINOv3 + SpatialBoost achieving state-of-the-art performance on ADE20K. Specifically, DINOv3 with multi-scale evaluation improves from 60.3 to 63.1 mIoU on ADE20K, and from 89.8 to 90.9 mIoU on Pascal VOC. The ADE20K improvement of 2.8 mIoU points (4.6% relative) on an already-strong baseline is notable because segmentation at this level likely requires distinguishing between visually similar but semantically distinct regions—a task where spatial context (object boundaries, depth ordering) could help disambiguate.

The cross-encoder pattern mirrors depth estimation: weaker encoders benefit more in absolute terms. OpenCLIP's ADE20K multi-scale mIoU improves from 46.0 to 47.3 (+1.3 points, 2.8% relative), SigLIPv2 from 48.7 to 50.8 (+2.1 points, 4.3% relative), DINOv2 from 53.0 to 54.9 (+1.9 points, 3.6% relative). The linear probing gains are larger in absolute terms: DINOv3 from 55.9 to 59.7 mIoU (+3.8 points, 6.8% relative).

Against vision-language baselines, SpatialBoost's advantage is substantial. TIPS, a method explicitly designed for spatial awareness in text-image pretraining, achieves 49.9 (linear) and 54.1 (+ms) on ADE20K—DINOv3 + SpatialBoost exceeds this by 9.8 and 9.0 mIoU points respectively, suggesting that TIPS's spatial signal (derived from standard image-text pairs) is far weaker than SpatialBoost's explicit 3D spatial supervision. PE-Core achieves 41.5 (linear) and 48.0 (+ms) on ADE20K, substantially below SpatialBoost. AIMv2 achieves only 31.9 (linear) and 37.9 (+ms), actually underperforming the vision-only V-JEPAv2 (51.3 linear, 53.8 +ms) on this task—this is a curious result that the paper does not discuss, but it may indicate that AIMv2's autoregressive pre-training objective produces features that are less suitable for dense prediction tasks compared to DINO-style self-supervised learning.

An interesting cross-task comparison: the relative improvement from SpatialBoost is larger for depth estimation (19.4% RMSE reduction for DINOv3 DPT) than for segmentation (4.6% mIoU gain for DINOv3 +ms). This is expected—depth estimation is a purely spatial task that directly benefits from injected 3D knowledge, while segmentation also relies heavily on semantic category knowledge that the base encoder already possesses. The fact that segmentation does improve suggests that spatial understanding helps with segmentation tasks, likely by providing better boundary localization and occlusion reasoning.

Complex 3D-Centric Tasks on Lexicon3D

Table 3 reports comprehensive results on the Lexicon3D benchmark, which probes four dimensions of 3D understanding: Vision-Language Reasoning (VLR: ScanQA and SQA3D BLEU-1), Visual Grounding (VG: ScanRefer accuracy), Geometric Understanding (GU: registration recall and translation error), and 3D Semantic Understanding (3D SU: accuracy and mIoU on ScanNet point clouds). All experiments freeze the vision encoder and train task-specific heads on top of extracted features, following the Lexicon3D probing protocol.

The headline result: SpatialBoost produces broad improvements across all four categories of 3D understanding, with the most dramatic gains on encoders that start with minimal 3D capability. The clearest illustration is OpenCLIP: its 3D semantic segmentation mIoU jumps from 6.9 to 54.9—an 8× improvement, transforming it from essentially unusable for 3D understanding to competitive with much larger encoders. This is the single largest relative improvement reported in the paper and provides strong evidence that SpatialBoost genuinely injects spatial knowledge into representations that previously encoded almost none.

Vision-Language Reasoning (VLR): DINOv3's ScanQA BLEU-1 improves from 40.6 to 43.3 (+2.7 points, 6.7% relative), and SQA3D from 51.4 to 54.9 (+3.5 points, 6.8% relative). These gains are meaningful in the context of BLEU-1, where a 2–3 point improvement typically corresponds to a noticeable increase in answer quality. Across all encoders, SpatialBoost improves ScanQA by 1.3–2.7 BLEU-1 points and SQA3D by 0.6–3.5 points. Compared to baselines: PE-Core achieves 40.5 on ScanQA and 51.7 on SQA3D—SpatialBoost DINOv3 surpasses both (43.3 and 54.9). The vision-only V-JEPAv2 achieves 37.7 and 49.0, substantially below SpatialBoost variants.

Visual Grounding (VG): DINOv3's ScanRefer accuracy improves from 56.2% to 61.1% (+4.9 percentage points, 8.7% relative). This is particularly notable because visual grounding requires precisely localizing objects in 3D space from natural language descriptions—a task that directly tests spatial understanding. OpenCLIP shows the largest absolute gain: from 50.1% to 56.6% (+6.5 points, 13.0% relative). SigLIPv2 improves from 51.4% to 56.8% (+5.4 points), DINOv2 from 52.7% to 57.0% (+4.3 points). The consistent 4–6 percentage point gains across all encoders suggest that SpatialBoost injects a generalizable spatial localization capability rather than encoder-specific improvements.

Geometric Understanding (GU): DINOv3's registration recall improves from 86.9% to 97.5% (+10.6 percentage points)—approaching near-perfect registration on ScanNet scenes. Translation error drops from 0.10m to 0.06m, a 40% reduction. These are highly practical gains: a registration recall of 97.5% means the model correctly aligns nearly all point cloud pairs at 5cm precision, which is sufficient for many robotics and 3D reconstruction applications. The pattern across encoders is consistent: OpenCLIP RR improves from 22.6% to 78.8% (a 3.5× improvement), SigLIPv2 from 47.8% to 86.4% (nearly doubling), DINOv2 from 82.4% to 92.4%. The fact that even DINOv2 and DINOv3—which already have substantial geometric understanding—benefit substantially suggests that SpatialBoost's spatial knowledge is complementary to whatever geometric features these models learned from their 2D pre-training.

3D Semantic Understanding (3D SU): DINOv3's mIoU improves from 69.1 to 70.6 (+1.5 points), while accuracy improves from 91.1% to 91.9% (+0.8 points). The gains on 3D segmentation are more modest than on geometric understanding, likely because 3D semantic understanding depends heavily on semantic category knowledge that the base encoders already encode well—the additional spatial knowledge from SpatialBoost helps primarily at object boundaries and in geometrically ambiguous regions. OpenCLIP shows the largest improvement (6.9 → 54.9 mIoU), transforming a model with essentially zero 3D semantic capability into one comparable to much larger and more sophisticated encoders. SigLIPv2 improves from 9.2 to 55.5 mIoU—a 6× improvement similar in magnitude to OpenCLIP's gain. The pattern suggests that SpatialBoost provides a foundational level of 3D understanding that language-aligned models (OpenCLIP, SigLIPv2) completely lack, bringing them to a baseline level of spatial competence, while vision-only models (DINOv2, DINOv3) which already have some spatial understanding from their self-supervised pre-training see more incremental gains.

A cross-task observation: the weakest baseline encoders (OpenCLIP, SigLIPv2) benefit proportionally more from SpatialBoost on GU and 3D SU than on VLR and VG. Geometric understanding registration recall for OpenCLIP goes from 22.6% to 78.8% (3.5×), while ScanQA BLEU-1 only goes from 36.9 to 39.2 (6.2% relative). This asymmetry likely reflects that geometric understanding and 3D segmentation are "purer" spatial tasks that depend minimally on semantic language-aligned features (which OpenCLIP already has), while VLR requires both spatial and linguistic reasoning—the linguistic component, already present in OpenCLIP, cannot be improved by spatial training alone.

Vision-Based Robot Learning on CortexBench

Table 4 reports the performance of imitation learning agents trained on visual representations from different encoders. The agents learn to map from front-camera observations (224×224 resolution, no depth measurements) to expert actions using behavior cloning. Four domains are evaluated: Adroit (dexterous hand manipulation, success rate %), MetaWorld (robotic arm manipulation tasks, success rate %), DMControl (physics-based locomotion, normalized score), and Trifinger (dexterous finger manipulation, success rate %). The "Avg." column reports the mean across domains.

The headline result: SpatialBoost DINOv3 achieves 80.8% average score, an 8.0 percentage point improvement over base DINOv3 (72.8%) and the highest performance across all evaluated encoders and domains. This is a practical, impactful gain—an 11.0% relative improvement in robotic task success rate—and it holds across all four domains individually.

Domain-by-domain analysis:

Adroit (DINOv3): 63.9% → 71.8% (+7.9 pp, 12.4% relative). Adroit requires precise manipulation of a 28-DOF hand, where spatial understanding of hand-object relationships is critical. The large gain suggests SpatialBoost helps the vision encoder produce features that better capture the 3D configuration of the hand relative to objects.
MetaWorld (DINOv3): 83.8% → 92.0% (+8.2 pp). MetaWorld tasks involve reaching, pushing, and manipulating objects on a tabletop—all tasks where depth perception and object localization matter. SpatialBoost pushes success rates close to saturation.
DMControl (DINOv3): 70.8% → 80.4% (+9.6 pp). DMControl locomotion tasks require understanding body configuration and terrain geometry from visual observations. The gain here is the largest among domains, perhaps because locomotion benefits more from holistic scene-level spatial understanding than manipulation tasks that can be solved with local object-centric features.
Trifinger (DINOv3): 72.8% → 79.0% (+6.2 pp). Trifinger involves dexterous multi-finger manipulation—the most challenging domain that requires fine-grained spatial reasoning about finger-object contacts.

The cross-encoder pattern is striking: DINOv2 + SpatialBoost achieves 75.8% (vs. 68.1% base), matching SigLIPv2 + SpatialBoost at 75.8% (vs. 69.7% base). Both substantially exceed OpenCLIP + SpatialBoost at 70.5% (vs. 65.5% base). This suggests that while SpatialBoost adds spatial knowledge that improves all encoders, the underlying semantic and representational quality of the base encoder still matters—DINOv3's larger capacity and stronger pre-training give it an edge that SpatialBoost amplifies rather than equalizes.

An important negative result: the paper includes standard deviations (3 runs) for each domain. These are relatively tight (e.g., DINOv3 + SpatialBoost on Adroit: 71.8 ± 3.4, meaning the 7.9 pp improvement is approximately 2.3 standard deviations above the base model's mean of 63.9 ± 1.5), suggesting the gains are statistically meaningful despite the lack of formal hypothesis testing.

Comparison against baselines: PE-Core (a method designed for rich visual embeddings) achieves 69.0% average, substantially below DINOv3 + SpatialBoost at 80.8%. TIPS achieves 66.0% average, also substantially lower. This suggests that SpatialBoost's explicit 3D spatial supervision provides more useful features for robotic control than the implicit spatial awareness in these prior methods.

Image Classification and Retrieval Tasks (Preserving General Vision Capabilities)

Table 5 evaluates whether SpatialBoost's spatial knowledge injection causes overfitting to spatial features at the expense of general vision capabilities. The answer, from the results, is a clear no—SpatialBoost actually improves non-spatial vision tasks.

Image Classification: The headline is that DINOv3 + SpatialBoost achieves 90.2% ImageNet linear probing accuracy, the highest among all evaluated methods. This is a 1.8 percentage point gain over base DINOv3 (88.4%), which is itself a strong baseline. The improvement is not limited to DINOv3: OpenCLIP improves from 86.8% to 87.9% (+1.1 pp), SigLIPv2 from 89.1% to 90.0% (+0.9 pp), DINOv2 from 87.3% to 88.6% (+1.3 pp). The kNN accuracy (which measures representation quality without any learned classifier, purely via nearest-neighbor lookup) shows a similar pattern: DINOv3 improves from 85.8% to 87.7% (+1.9 pp).

What explains the classification improvement if SpatialBoost only injects spatial knowledge? The paper's hypothesis (Section 4.5) is that the dual-channel attention mechanism preserves pre-trained semantic knowledge while the scene captioning turns in the reasoning data (turns 11–12) provide continued general supervision. An alternative possibility—which the paper does not discuss—is that some ImageNet classes have spatial signatures (e.g., "bookcase" implies vertical planar structure, "screwdriver" has a distinctive 3D shape) and improved spatial features help disambiguate these classes. The fact that kNN accuracy (which relies purely on feature similarity) improves as much as linear probing accuracy supports the idea that the representations themselves are better, not just the learned classifier.

Comparison against baselines: PE-Core achieves 89.5% linear accuracy and 86.8% kNN—DINOv3 + SpatialBoost exceeds both (90.2%, 87.7%). AIMv2 achieves 88.3% and 85.7%—also below. The paper notes that SpatialBoost's improvement over these strong language-supervised baselines demonstrates that the gains are not simply from additional language supervision but specifically from the spatial nature of that supervision combined with the dual-channel architecture.

Image Retrieval: On instance-level retrieval (finding images of the same object or building across different views, lighting, and backgrounds), SpatialBoost consistently improves performance:

Oxford-Hard: DINOv3 from 60.7 to 64.1 mAP (+3.4 points)
Paris-Hard: DINOv3 from 87.1 to 88.6 mAP (+1.5 points)
Met (artworks): DINOv3 from 55.4 to 57.0 GAP (+1.6 points)
AmsterTime (cross-time retrieval): DINOv3 from 56.5 to 56.9 mAP (+0.4 points, negligible improvement)

The retrieval pattern is informative: spatial features help most on datasets where viewpoint variation is a challenge (Oxford and Paris contain photos of landmarks taken from different angles; spatial features may help normalize viewpoint differences). The minimal gain on AmsterTime—a dataset specifically designed for severe domain shift across time—suggests that spatial features don't help when the primary challenge is appearance change (season, lighting, architectural modification) rather than viewpoint change.

The paper also includes an important cross-method comparison within retrieval: OpenCLIP (a CLIP-style model) achieves only 23.4 mAP on Oxford-Hard and 59.7 mAP on Paris-Hard, while OpenCLIP + SpatialBoost achieves 32.8 and 69.4 respectively—gains of 9.4 and 9.7 mAP. These are the largest retrieval improvements reported, suggesting that CLIP-style contrastive training, while excellent for semantic classification, produces representations that are particularly poor for instance-level spatial matching, and SpatialBoost partially remedies this deficit.

Ablation Studies and Robustness Checks

Effect of LLM-based fine-tuning vs. pixel-level supervision (Table 6): This is the most important ablation in the paper. Fine-tuning DINOv2-ViT-L/14 with different decoder heads on 300K spatial data reveals that only the LLM decoder preserves and improves classification accuracy while also improving spatial tasks. Linear (depth) head: classification drops 1.39% but depth improves 7.89%. Linear (segmentation) head: classification improves 0.35% but depth degrades 18.42%—supervising on segmentation actively hurts depth understanding. SAM decoder: classification unchanged (0.0%) but depth degrades 10.53% and VQA degrades 4.08%. VGGT decoder: classification drops 1.74%, depth improves 7.89%, but segmentation degrades 4.40%. LLM decoder: classification improves 2.32%, segmentation improves 7.97%, depth improves 15.79%, and VQA improves 2.04%—the only method that improves all metrics simultaneously. This is the key empirical justification for the core design choice of using an LLM rather than pixel-level supervision.

Effect of multi-turn reasoning hierarchy (Table 7): Comparing forward (pixel → object → scene), reverse (scene → object → pixel), and random orderings of the reasoning turns, all with three levels (pixel + object + scene) and 100K samples: Forward achieves 87.6% classification, 48.9 mIoU segmentation, 0.34 RMSE depth. Reverse achieves 87.4%, 48.4, 0.35—modestly worse across all metrics. Random achieves 87.4%, 48.5, 0.36—slightly worse for depth. The differences are small but consistent, and the paper also reports in Appendix E.1 (Table 15) a more detailed ablation on combinations of reasoning levels, showing that the full three-level hierarchy (Pix + Obj + Scene) outperforms any two-level combination (e.g., Pix + Obj achieves 48.8 mIoU vs. 49.2 for full hierarchy on DINOv2), and that pixel-level data is particularly important for dense prediction tasks while object-level data contributes more to classification.

Effect of single-view vs. multi-view data (Table 7 and Table 16): With fixed 300K total samples, varying the ratio of single-view to multi-view data: 200K SV + 100K MV vs. 150K SV + 150K MV vs. 100K SV + 200K MV. On DINOv3 (Table 16), the 100K+200K split achieves the best performance on geometric understanding (97.5% RR vs. 96.0% for 200K+100K) and 3D semantic understanding (70.6 mIoU vs. 69.7), while classification is essentially identical (90.2 vs. 90.2). This indicates that multi-view data is particularly valuable for 3D-specific tasks (GU, 3D SU) but unnecessary for 2D tasks where single-view spatial reasoning suffices. The combination of both data types outperforms either alone (Table 7: SV-only achieves 48.9 mIoU, MV-only achieves 48.2, combined achieves 49.2), confirming complementary benefits.

Effect of dataset scalability (Figure 5, Table 18): Training on 50K, 100K, and 300K samples (with matched total iterations via epoch adjustment) shows monotonic improvements: DINOv3 ADE20K mIoU goes from 55.9 (pre-trained) → 56.8 (50K) → 58.3 (100K) → 59.7 (300K), and NYUd RMSE goes from 0.31 → 0.29 → 0.28 → 0.25. The improvement from 100K to 300K is substantial (+1.4 mIoU, −0.03 RMSE), suggesting that 300K has not saturated performance and further data scaling would likely yield additional gains. SigLIPv2 shows a similar pattern: ADE20K from 42.8 → 43.2 → 44.5 → 45.1, NYUd from 0.51 → 0.44 → 0.42 → 0.39. The paper also reports in Figure 5 the AbsRel metric (absolute relative error, another standard depth metric) which shows the same scaling trend.

Effect of dual-channel attention vs. alternative fine-tuning (Table 17, Figure 6): Full fine-tuning: ImageNet drops 6.8 pp (86.3% → 79.5%). LoRA: drops 4.6 pp (86.3% → 81.7%). Dual-channel attention: improves 1.3 pp (86.3% → 87.6%). Full fine-tuning and LoRA both produce better segmentation and depth than the pre-trained model, but at catastrophic cost to classification. Dual-channel uniquely avoids this tradeoff. The paper's Figure 7 provides qualitative evidence: attention rollout visualizations show that SpatialBoost's attention maps are more structured and spatially focused than the base encoder's, with the red cross (query patch) attending to semantically and spatially related regions rather than diffuse background.

Effect of reasoning hierarchy composition (Appendix E.1, Table 15): A detailed ablation on DINOv2 and OpenCLIP tests all combinations of reasoning levels (Pix only, Obj only, Scene only, Pix+Obj, Pix+Scene, Obj+Scene, all three). Key findings: (1) Pixel-only data provides the best depth estimation (0.34 RMSE on OpenCLIP, matching the full hierarchy's performance), confirming pixel-level spatial queries directly teach depth understanding. (2) Object-only data provides the best classification improvement (+0.9 pp on DINOv2 with Obj-only vs. +1.3 pp with full hierarchy), suggesting object-level spatial reasoning transfers to semantic category recognition. (3) Scene-only data is the weakest single level, likely because scene-level questions (inter-object distances) are too difficult without the foundation of pixel and object understanding. (4) The full three-level hierarchy consistently achieves the best or near-best performance across all tasks, with no single level or pair dominating—meaning the pyramid structure provides genuinely complementary signals.

Effect on already spatial-aware encoders (Table 9): Applying SpatialBoost to TIPS and PE-Core yields further improvements: TIPS depth RMSE drops from 0.35 to 0.25, segmentation mIoU from 49.9 to 53.5, VLR BLEU-1 from 37.4 to 41.1, robot learning from 66.0 to 76.5 (a 15.9% relative gain). PE-Core depth from 0.25 to 0.20, segmentation from 41.5 to 50.9, VLR from 40.5 to 44.1, robot learning from 69.0 to 81.5 (an 18.1% gain). PE-Spatial (PE-Core's spatial variant) sees smaller but still significant gains: depth 0.26 → 0.19, segmentation 49.3 → 56.3. This is a strong robustness check: SpatialBoost provides value even when the base encoder has been explicitly optimized for spatial awareness, suggesting the language-guided spatial reasoning data provides complementary information not captured by these methods.

Comparison with naive post-training (Table 8): Fine-tuning the vision encoder with its original pre-training objective on the same 300K spatial data ("Simple FT") yields negligible or negative improvements. DINOv3 Simple FT: depth RMSE unchanged (0.31 vs. 0.31), segmentation slightly improved (55.9 → 56.4 mIoU), VQA slightly improved (40.6 → 40.2 BLEU-1), robot learning improved (72.8 → 75.5), classification slightly improved (85.8 → 86.1). The gains are dramatically smaller than SpatialBoost's (e.g., robot learning +2.7 pp for Simple FT vs. +8.0 pp for SpatialBoost). This demonstrates that the spatial reasoning data format (language-based, multi-turn, hierarchical) and the LLM decoder are essential—simply training on more 2D images, even if those images come with 3D metadata, doesn't inject spatial understanding effectively without the right training framework.

Bias propagation analysis (Appendix E.5, Table 19): Generating reasoning data using vision foundation model outputs (Depth Pro, SAM) vs. ScanNet ground-truth 3D annotations produces negligible performance differences in the resulting fine-tuned encoder: classification differs by 0.0%, segmentation by −0.1 mIoU, depth by 0.0 RMSE, VQA by 0.0 BLEU-1 (the +2.7 BLEU-1 difference shown in the table is for the GT-based model over the pre-trained baseline, not between VFM and GT methods). This alleviates concerns that errors from upstream vision models propagate into and degrade the fine-tuned encoder's representations.

Application to MLLM visual encoders (Appendix E.6, Tables 20–21): SpatialBoost applied to InternViT-6B-v2.5 (from InternVL3) improves ADE20K mIoU from 39.4 to 48.5 (+9.1 points) and NYUd RMSE from 0.46 to 0.35. Applied to Qwen3-VL-VE (0.6B parameters), it improves segmentation from 40.8 to 44.3 mIoU and depth from 0.44 to 0.36 RMSE. When these enhanced encoders are plugged into the full MLLMs (InternVL3-38B and Qwen3-VL-32B-Instruct), VQA performance on spatial tasks improves: BLINK (visual spatial reasoning) goes from 64.0 to 69.2 for InternVL3 and from 67.3 to 70.8 for Qwen3-VL. MUIRBench (multi-image understanding) goes from 63.8 to 70.7 and 72.8 to 76.4 respectively. These are practically meaningful improvements on benchmarks specifically designed to test spatial and multi-view reasoning, and they demonstrate that SpatialBoost's vision encoder enhancements transfer to downstream MLLM performance without requiring re-training of the language model components.

Critical Assessment

Does SpatialBoost inject spatial knowledge into vision encoders? The evidence from dense prediction tasks (Tables 1–2) and 3D-centric tasks (Table 3) is strong and consistent. Every base encoder—from the spatially-impoverished OpenCLIP to the already-spatially-strong DINOv3—shows meaningful improvements on depth estimation, semantic segmentation, 3D visual grounding, geometric understanding, and 3D semantic segmentation. The improvements are not marginal: OpenCLIP's 3D semantic mIoU goes from 6.9 to 54.9 (Table 3), which represents a fundamental change in the model's capability. DINOv3's NYUd RMSE drops from 0.31 to 0.25 (Table 1), a 19.4% relative improvement that brings it to the level of or beyond dedicated depth estimation methods. The geometric understanding gains—DINOv3 registration recall from 86.9% to 97.5% (Table 3)—are particularly convincing because geometric correspondence is a pure test of 3D structural understanding with minimal confounding from semantic knowledge.

However, the paper does not demonstrate which specific spatial capabilities are being injected. The Lexicon3D benchmark tests four dimensions of 3D understanding, and SpatialBoost improves all of them, but there's no analysis of whether certain spatial concepts (e.g., absolute depth vs. relative depth ordering vs. 3D distance reasoning) benefit more than others. The per-level reasoning hierarchy ablation (Appendix E.1) provides some hints—pixel-level data helps depth most, object-level helps classification—but a more detailed analysis of what types of spatial errors are corrected would strengthen the interpretation.

Does SpatialBoost preserve general vision capabilities without overfitting? The evidence from classification and retrieval (Table 5) strongly supports this claim. ImageNet accuracy improves across all encoders (+0.9 to +1.9 percentage points for linear probing), which is a strong signal that the dual-channel attention mechanism successfully prevents catastrophic forgetting. The retrieval improvements (Oxford-Hard mAP from 60.7 to 64.1 for DINOv3) further confirm that instance-level matching capabilities are preserved and enhanced.

A subtle qualification: the paper's claim is about preserving pre-trained knowledge, but the results actually show improving it. This is stronger than claimed but raises a question: is the improvement from the spatial reasoning data, the dual-channel attention architecture, or the additional compute of Stage 3 training? The Simple FT baseline (Table 8) partially addresses this—it shows that training on the same data with the original pre-training objective yields much smaller gains—but doesn't fully isolate the effect of dual-channel attention vs. the reasoning data format. An additional ablation training with the spatial data but using full fine-tuning (which catastrophically forgets) vs. dual-channel (which improves) would cleanly demonstrate that the architecture, not just the data, drives the preservation.

Does the LLM-based supervision outperform pixel-level alternatives? The Table 6 ablation is the key evidence, and it is compelling but not exhaustive. The comparison includes linear heads, SAM decoder, and VGGT decoder—all strong choices—and the LLM consistently dominates. However, the pixel-level decoders were trained only on subsets of the reasoning data (depth maps only for the depth linear head, segmentation masks only for the SAM decoder), while the LLM was trained on the full 12-turn conversations. This means the LLM saw more diverse supervision (pixel + object + scene queries plus captions) than any single pixel-level decoder. To make the comparison perfectly fair, one would need to train a multi-task pixel-level decoder that simultaneously predicts depth, segmentation, bounding cubes, and inter-object distances—matching the diversity of the LLM's supervision. This ablation is missing, and it's possible that a sufficiently rich multi-task pixel-level decoder would close some of the gap.

Additionally, the experiment only evaluates one LLM (Qwen-2.0-7B). It's possible that larger or smaller LLMs would produce different results, or that the LLM's own pre-training quality matters. The paper doesn't ablate the LLM choice, so the claim that "LLM-based" supervision is superior should be interpreted as "Qwen-2.0-7B-based supervision is superior to the tested pixel-level decoders," not as a universal statement about any LLM vs. any pixel-level method.

Does SpatialBoost improve robot learning? The CortexBench results (Table 4) show substantial gains (+8.0 pp average for DINOv3), but the evaluation protocol has important limitations. The agents are trained on only 100 demonstrations per task, which is a relatively data-scarce regime where representation quality matters enormously. It's possible that with more demonstration data, the gap between SpatialBoost and base encoders would narrow because the behavior cloning policy could learn to extract spatial information from weaker features given enough examples. The paper doesn't test this. Additionally, the evaluation uses only front-camera observations without depth measurements—meaning all spatial information must come from the RGB representation. In practical robotics, depth sensors are often available, and it's unclear whether SpatialBoost's benefits would persist when depth is explicitly provided as input.

The standard deviations (Table 4 reports e.g., DINOv3 + SpatialBoost on Adroit: 71.8 ± 3.4) are modest, suggesting statistical reliability, but with only 5 evaluation runs per configuration, confidence intervals are wide. A more rigorous evaluation would increase the number of runs or report confidence intervals.

Do the results generalize beyond the tested encoders? The paper tests four encoder families (OpenCLIP, SigLIPv2, DINOv2, DINOv3) spanning different pre-training objectives and scales, plus two spatial-aware encoders (TIPS, PE-Core). This is good coverage but still limited to Vision Transformer architectures with patch-based processing. The paper doesn't test convolutional architectures (ResNet, ConvNeXt), which have different inductive biases and might benefit differently from spatial knowledge injection. The application to MLLM encoders (InternViT, Qwen3-VL-VE) in the appendix partially addresses this but focuses on downstream MLLM tasks rather than the full suite of dense prediction and 3D benchmarks.

Does the approach scale? The dataset scalability experiments (Figure 5, Table 18) show monotonic improvements from 50K to 300K samples with no sign of saturation, suggesting further scaling would yield additional gains. However, 300K samples is modest by modern vision pre-training standards (ImageNet alone has 1.2M images, and the SA1B dataset has over 11M). The paper doesn't test whether the gains hold at much larger scales (1M+ samples) or whether the hierarchical reasoning format remains effective when scaled. There may be diminishing returns as the spatial concepts in the dataset become redundant. Additionally, the data generation pipeline—which requires running Depth Pro, SAM, VGGT, and GPT-4o on each sample—is computationally expensive, and the paper doesn't discuss the cost of scaling this pipeline to millions of images.

Does SpatialBoost combine with other enhancement methods? The paper shows that SpatialBoost improves already spatial-aware encoders (TIPS, PE-Core, PE-Spatial in Table 9), which is a strong positive result. However, it doesn't test whether SpatialBoost's gains are complementary or overlapping with these methods. If TIPS and PE-Core are already injecting some form of spatial knowledge, the fact that SpatialBoost adds further gains suggests the spatial signals are different, but the paper doesn't analyze how they differ—do they improve different types of spatial understanding, different regions of the image, different depth ranges?

Missing experiments and analyses:

No direct measurement of 3D spatial reasoning capability: Despite being the paper's central motivation, there is no experiment that directly measures whether the fine-tuned vision encoder can answer spatial reasoning questions (like "Is A to the left of B?") without the LLM decoder. The evaluation protocol always uses the encoder features through task-specific heads (linear probes, DPT heads, Lexicon3D heads). A direct probing experiment—e.g., training a linear classifier on the [CLS] token to predict relative spatial relationships—would isolate whether the spatial knowledge lives in the encoder's representations or requires the LLM decoder at inference time.
No latency or memory analysis for dual-channel attention: The dual-channel attention adds 25–30% to the vision encoder's parameter count during Stage 3 training. However, the paper never discusses whether the extra attention channels are needed at inference time for the downstream tasks. If the enhanced encoder's features are extracted and the dual-channel layers are discarded, what is the inference cost relative to the base encoder? If the dual-channel mechanism could be "baked in" post-training (e.g., by merging the original and copy attention weights based on the learned α values), that would be valuable to know.
No analysis of what the new attention channels learn: Figure 7 shows qualitative attention visualizations, but there's no quantitative analysis of how the attention patterns differ between the original and copy channels. Do the copy channels learn to attend to depth discontinuities, object boundaries, or spatial relationships? Do they specialize by layer (early layers focusing on local depth, later layers on object-level spatial relationships)? This analysis would provide mechanistic insight into how dual-channel attention works.
No comparison to simply concatenating 3D features: A simple alternative to SpatialBoost would be to extract depth maps, point clouds, and object bounding cubes from the source images using the same vision models (Depth Pro, SAM, VGGT), encode them with a separate network, and concatenate the resulting features with the RGB vision encoder features. This would provide the same 3D information but through an explicit feature channel rather than through language-guided fine-tuning. Comparing SpatialBoost to this baseline would test whether the language-based training signal is genuinely necessary or whether the 3D information itself is what matters.
No multi-view encoder evaluation: The paper generates multi-view spatial reasoning data and uses it for training, but all downstream evaluations are single-view (a single RGB image as input). The multi-view VQA dataset and multi-view reasoning data teach the model about multi-view relationships, but the evaluations never actually require processing multiple views. It's possible that the multi-view data helps by providing richer 3D supervision even for single-view tasks, but the paper doesn't directly test whether this is true or whether single-view data alone would suffice.
The NYUd and KITTI benchmarks have different depth ranges, but this isn't analyzed: NYUd is indoor (0.5–10m depth range), KITTI is outdoor driving (up to 80m). SpatialBoost shows larger relative gains on NYUd (19.4%) than KITTI (8.9%) for DINOv3. Is this because the spatial reasoning data is mostly indoor scenes at close range, creating a domain gap for outdoor depth? An analysis of depth error by distance bin would reveal whether SpatialBoost helps primarily at close range (where the training data is concentrated) or generalizes to long-range depth.

Overall assessment: The paper provides thorough and convincing evidence that SpatialBoost injects spatial knowledge into vision encoders and that this knowledge transfers to a wide range of downstream tasks. The experimental design is careful: comparisons span multiple encoder families, multiple tasks, and multiple probing protocols. The ablation studies systematically isolate the contributions of dual-channel attention, hierarchical reasoning, data scaling, and single-view vs. multi-view data. The main weakness is the lack of direct measurement of spatial reasoning capability in the encoder's representations and the absence of experiments that would reveal what specific spatial errors are being corrected. The paper also leaves open questions about inference-time cost, scalability of the data generation pipeline, and whether the LLM-based supervision advantage over pixel-level alternatives would hold under perfectly fair comparison conditions (equal supervision diversity). However, these limitations do not undermine the central claims; rather, they represent productive directions for future work that the paper's framework enables.

6. Limitations and Trade-offs

Hard Problems Remain Fundamentally Unsolvable by Test-Time Spatial Knowledge Injection

The assumption or constraint: SpatialBoost assumes that the vision encoder possesses some baseline capacity to produce correct spatial representations, which the language-guided training signal then amplifies and refines. The paper is explicit about the boundary: problems where the base model's effective pass rate for spatial reasoning is near zero cannot be improved because there are no correct representations to amplify. The data generation pipeline—extracting 3D point clouds from 2D images using Depth Pro, SAM, and VGGT—provides "correct" spatial answers to the reasoning questions regardless of whether the vision encoder can produce features that would support those answers. But the training only succeeds if the encoder can learn to produce such features from the language supervision signal.

The consequence: On tasks that are fundamentally outside the vision encoder's current representational capacity, SpatialBoost will yield minimal or zero improvement. The paper does not characterize which spatial concepts are too difficult for which encoders. For instance, can SpatialBoost teach an encoder to understand complex occlusion relationships when the base encoder confuses object boundaries? Can it teach metric depth at long ranges when the base encoder's receptive field or resolution is insufficient? Without this characterization, a practitioner cannot predict whether SpatialBoost will help on a given problem without running the full training pipeline.

What evidence exists in the paper: The paper does not directly measure this boundary. The closest proxy is the differential improvement across benchmarks and encoders. On NYUd (indoor, close-range), DINOv3 improves from 0.31 to 0.25 RMSE—a 19.4% relative gain. On KITTI (outdoor, long-range), the same encoder improves from 2.02 to 1.84 RMSE—only an 8.9% relative gain. This asymmetry, while not discussed as a capability boundary by the authors, suggests that the spatial knowledge transfer is less effective in regimes far from the training data distribution (which is predominantly indoor and close-range). Additionally, the improvements on 3D semantic understanding (Table 3) are consistently smaller than on geometric understanding: DINOv3's 3D SU mIoU improves only from 69.1 to 70.6 (+1.5 points) while registration recall improves from 86.9% to 97.5% (+10.6 points). This may indicate that semantic 3D understanding (inferring what an object is from its 3D shape) is closer to the base encoder's capability ceiling than geometric understanding (estimating where things are in 3D space).

Mitigation status: The paper does not address this limitation. The dataset scalability experiments (Figure 5) show that more data helps, which is encouraging but does not directly test the capability boundary—adding more examples of a concept the encoder fundamentally cannot represent won't help. The paper acknowledges in Section 1 that the approach targets the gap between 2D-trained encoders and 3D spatial understanding, but does not specify what aspects of spatial understanding are learnable vs. unlearnable via language-guided fine-tuning. Future work to characterize this boundary—e.g., by measuring per-concept improvement as a function of base encoder capability—would significantly improve the framework's practical utility.

The Data Generation Pipeline Imposes a Heavy, Unaccounted Computational Cost

The assumption or constraint: SpatialBoost's headline results report the performance improvements after Stage 3 fine-tuning on 300K multi-turn spatial reasoning samples. However, generating these 300K samples requires running a pipeline of computationally expensive models before any training begins: Depth Pro for monocular metric depth estimation, SAM for object segmentation, VGGT for multi-view 3D reconstruction, and GPT-4o for question generation. The paper does not include this cost in any reported efficiency metric. The VQA dataset construction for Stage 2 (200K multi-view pairs) similarly requires LPIPS computation across large image collections and GPT-4o question generation.

The consequence: The true cost of deploying SpatialBoost is cost(data generation) + cost(Stage 1 training) + cost(Stage 2 training) + cost(Stage 3 training), but the paper only reports the Stage 3 training cost (4× A100s, one epoch, 300K samples). For a practitioner attempting to replicate SpatialBoost on a new domain or scale it to larger datasets, the data generation pipeline may dominate the total compute budget. Specifically: generating 3D point clouds for 300K single-view and multi-view samples using Depth Pro, SAM, and VGGT; running GPT-4o to generate 12-turn conversations for each sample; and applying the CLIP-based filtering to select appropriate images—this could easily cost more than the Stage 3 fine-tuning itself.

The paper reports incremental improvements from 50K to 100K to 300K samples (Figure 5, Table 18), showing that performance continues to improve with more data. But a practitioner needs to know where the cost-performance trade-off plateaus when including generation costs. If generating 300K samples costs 10× the compute of generating 100K samples (because the multi-view data requires expensive VGGT reconstruction, for instance), the marginal benefit of scaling from 100K to 300K might not justify the cost. The paper provides no information about this trade-off.

What evidence exists in the paper: The paper provides no cost analysis for the data generation pipeline. The implementation details (Appendix A) describe training costs: Stage 3 trains on 4× A100s with batch size 128 for one epoch, which is a known, modest cost for a 7B-parameter architecture. But the data generation uses unspecified computational resources to run Depth Pro (a state-of-the-art metric depth model requiring a forward pass per image), SAM (a large vision model), VGGT (a 3D reconstruction model processing multiple views), and GPT-4o (a commercially-accessible LLM charged per token). The paper does not list the number of GPU-hours, the cost of GPT-4o API calls, or the wall-clock time for generating the full 300K + 200K + 558K samples.

Mitigation status: The paper does not address this limitation. The only acknowledgment that data generation is non-trivial comes indirectly through the bias propagation analysis (Appendix E.5), which validates that the generated data quality is sufficient, not that the generation process is efficient. Future work could explore cheaper alternatives for 3D extraction (e.g., using smaller, faster depth and segmentation models), template-based question generation rather than GPT-4o, or active learning strategies that focus data generation on the most informative samples.

The Dual-Channel Attention Mechanism Increases Inference-Time Parameter Count by 25–30%

The assumption or constraint: The dual-channel attention mechanism adds a parallel copy of each attention layer, initialized from pre-trained weights and trained during Stage 3. The paper reports that this increases parameter count by 25–30% depending on the encoder (Appendix A.2: "the number of model parameters increased by 30% in OpenCLIP and SigLIPv2 and by 25% in DINOv2 and DINOv3"). All downstream evaluations use the enhanced encoder with these additional parameters, meaning the inference cost (FLOPs, memory, latency) is 25–30% higher than the base model. However, the paper does not report whether this overhead is necessary at inference time or could be eliminated post-training.

The consequence: A practitioner comparing DINOv3 vs. DINOv3 + SpatialBoost on a downstream task is not comparing equal-compute models. The SpatialBoost version has ~8.75B parameters (7B base + 25% overhead) vs. 7B for the base model. Some fraction of the reported improvements may come from the increased capacity rather than the spatial knowledge itself. An important ablation is missing: what happens if you train the same 25–30% parameter increase without the spatial reasoning data (e.g., using standard pre-training data or the Simple FT baseline) and compare to SpatialBoost? This would separate the effect of additional capacity from the effect of spatial knowledge injection. The paper's Simple FT baseline (Table 8) uses the original architecture without dual-channel attention, making it an unequal comparison for assessing the contribution of added capacity.

Furthermore, for deployment scenarios where inference efficiency matters (on-device, real-time), a 25–30% parameter increase translates directly to higher latency, higher memory usage, and higher energy consumption. The paper does not discuss whether the enhanced features could be extracted from a merged model post-training. Given that dual-channel attention is defined as α · Attn(x) + (1 − α) · Attn^+(x), and both Attn and Attn^+ are linear operations in their input (modulo the softmax in self-attention), it might be possible to merge the weights post-training into a single attention layer that approximates the blended output. If such merging is feasible, the inference cost would return to baseline. If not, the cost is permanent.

What evidence exists in the paper: The paper reports the parameter increase transparently in Appendix A.2 but does not include any analysis of inference cost, latency measurements, or memory usage for the enhanced encoders. No experiment compares equal-parameter baselines. No merging or distillation experiment is presented to test whether the spatial knowledge can be preserved without the dual-channel overhead. The qualitative attention visualizations in Figure 7 show that SpatialBoost's attention patterns differ from the base encoder's, but this does not address whether both attention channels are needed or whether their contributions could be consolidated.

Mitigation status: The paper does not address this limitation. The dual-channel mechanism is presented as a solution to catastrophic forgetting, but the cost of that solution—permanently increased model size—is not discussed. Future work could explore: (1) knowledge distillation from the enhanced encoder back into the original architecture, (2) post-hoc weight merging strategies that eliminate the dual channels while preserving spatial features, (3) comparisons to capacity-matched baselines that control for parameter count, or (4) pruning the dual-channel layers after training to identify which channels learned spatial features and which remained near their initialization (suggesting they could be removed without performance loss).

All Experiments Use a Single Mathematical Reasoning Benchmark Family and Do Not Establish Cross-Domain Generalization

The assumption or constraint: SpatialBoost is evaluated on a specific set of vision benchmarks: NYUd and KITTI (depth), ADE20K and Pascal VOC (segmentation), Lexicon3D (3D understanding from ScanNet), CortexBench (robotic control), ImageNet-1K (classification), and Oxford/Paris/Met/AmsterTime (retrieval). While diverse, these benchmarks share important characteristics: they all involve natural images of physical scenes, they all have unambiguous ground truth (metric depth values, segmentation masks, object identities, 3D coordinates, answer strings), and none require spatial reasoning about abstract or non-physical visual content (diagrams, charts, medical images, satellite imagery, artistic renderings). The paper does not test whether SpatialBoost's spatial knowledge transfers to domains with different visual statistics or different types of spatial reasoning.

The consequence: A practitioner working in medical imaging (understanding 3D organ relationships from CT or MRI slices), satellite imagery (reasoning about building heights and terrain from overhead views), autonomous driving (where depth ranges, occlusion patterns, and object categories differ dramatically from indoor scenes), or document understanding (spatial layout of text and figures) cannot assume that SpatialBoost will provide similar gains. The spatial reasoning dataset is constructed primarily from SA1B images (consumer photographs), ScanNet (indoor scenes), and ego-centric video (human activities from body-mounted cameras). The depth and spatial relationships in these domains—indoor object distances of 0.5–5 meters, human-scale object sizes, front-facing camera perspectives—may not transfer to domains with different depth statistics, object scales, or viewing geometries.

The KITTI results (Table 1) provide indirect evidence of this limitation: the relative improvement on outdoor driving depth estimation (8.9% for DINOv3 DPT) is less than half the improvement on indoor depth estimation (19.4% for DINOv3 DPT). This gap could reflect the domain shift between the reasoning data (indoor-heavy) and the KITTI evaluation (outdoor driving). If so, applying SpatialBoost to domains even further from the training distribution—e.g., medical imaging, microscopy, remote sensing—may yield even smaller or zero gains.

What evidence exists in the paper: The paper provides no cross-domain evaluation. All benchmarks are standard computer vision datasets within the natural image domain. The paper does not test on abstract spatial reasoning (e.g., Raven's Progressive Matrices applied to visual patterns), on non-RGB modalities (depth-only, infrared, medical scans), or on tasks where spatial relationships must be inferred from non-visual cues. The KITTI vs. NYUd discrepancy is visible in Table 1 but is not discussed as a generalization limitation.

Mitigation status: The paper does not address this limitation. Given that the data generation pipeline could in principle be applied to any domain where depth estimation, segmentation, and 3D reconstruction models exist, the framework is domain-adaptable. But this would require re-running the entire pipeline—filtering domain-appropriate images, extracting 3D structure using domain-appropriate models, generating QA pairs—which is expensive and not validated. The paper's claim to be a "scalable framework" (Abstract) should be understood as scalable within the natural image domain, not across domains, until cross-domain evidence is provided.

The Paper Does Not Demonstrate That the Vision Encoder Itself Learns Spatial Reasoning—Only That Its Features Support Downstream Spatial Tasks

The constraint: All evaluations of SpatialBoost-enhanced encoders use the encoder as a frozen feature extractor with task-specific heads trained on top. The dense prediction experiments (Tables 1–2) train linear or DPT heads to predict depth or segmentation from encoder features. The Lexicon3D experiments (Table 3) use Q-Former, attention-based fusion, and linear heads. The CortexBench experiments (Table 4) train behavior cloning policies on [CLS] token features. The classification and retrieval experiments (Table 5) use linear layers or cosine similarity. In no experiment does the vision encoder directly answer spatial reasoning questions (e.g., "Is object A to the left of object B?") without a trained task-specific decoder.

The consequence: The paper's central claim—that SpatialBoost "enhances the spatial awareness of existing pre-trained vision encoders" (Abstract) by injecting "3D spatial knowledge" (Section 3)—is supported indirectly through improved downstream task performance, not through direct measurement of spatial reasoning capability in the encoder's representations. It is possible, though unlikely, that the improvements are driven by secondary effects: better feature discrimination for depth boundaries, improved texture representation that incidentally helps depth estimation, or representations that are more linearly separable for segmentation classes for reasons unrelated to 3D understanding. The paper cannot rule out these alternatives because it never probes whether the encoder has learned spatial concepts (depth ordering, relative position, 3D distance) vs. spatial features (patterns that correlate with 3D structure but do not constitute an explicit model of it).

This limitation matters for scientific understanding: an encoder that predicts correct depth values because its features are highly discriminative for object boundaries (which correlate with depth discontinuities) is different from an encoder that has an internal representation of 3D scene geometry. The paper's language—"spatial awareness," "spatial knowledge," "spatial understanding"—implies the latter, but the experiments can only confirm the former.

What evidence exists in the paper: The ablation in Table 6 provides the closest thing to a direct measurement: when trained with LLM supervision, the encoder's features support better ScanQA vision-language reasoning (BLEU-1 40.0 vs. 39.2 for the pre-trained model). Since ScanQA questions include spatial queries like "What is the object on the table?" which require understanding spatial relationships, this suggests the encoder's features encode some spatial information. But the ScanQA evaluation uses a Q-Former projection head trained specifically for this task, so the spatial reasoning capability could reside partially in the Q-Former rather than the encoder. The paper does not, for example, train a linear probe on the [CLS] token to classify relative spatial relationships (left/right, in-front/behind, closer/farther) and show that this probe achieves high accuracy. Such an experiment would directly measure whether spatial concepts are linearly decodable from the encoder's representations.

Mitigation status: The paper does not address this limitation. The MLLM evaluation in Appendix E.6 (Tables 20–21) partially speaks to it: when the SpatialBoost-enhanced vision encoder is plugged into a full MLLM (InternVL3-38B, Qwen3-VL-32B-Instruct), performance on spatial reasoning benchmarks (BLINK, MUIRBench) improves. Since the MLLM's language model is frozen (only the vision encoder changed), the improvement must come from better visual features. However, the MLLM evaluation still uses task-specific training of the full model, not zero-shot spatial reasoning. A clean probe experiment—training a linear classifier on the [CLS] token to answer spatial questions directly—would resolve this ambiguity and should be straightforward to conduct with the existing reasoning dataset (which conveniently provides ground-truth answers to spatial questions). Its absence is a notable gap in the experimental validation.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

[This limitation does not apply to SpatialBoost. The paper does not use revision models or sequential answer refinement. This limitation is from the reference example (about the paper analyzed there) and should not appear in the SpatialBoost analysis. I have replaced it with a genuine limitation from the SpatialBoost paper.]

The Framework Cannot Be Applied to Domains Where Reliable 3D Extraction Models Do Not Exist

The constraint: SpatialBoost's data generation pipeline depends entirely on the availability and accuracy of specialized vision models for 3D extraction: Depth Pro for monocular depth, SAM for segmentation, VGGT for multi-view reconstruction, plus GPT-4o for question generation. Section 3.2 and Appendix C describe this dependency explicitly. The paper acknowledges the reliance in Appendix F: "our proposed pipeline for constructing the visual spatial reasoning dataset relies on vision models, which is a limitation also shared by many recent works on spatial VQA." The bias propagation experiment (Appendix E.5) shows that errors from these models do not meaningfully degrade the fine-tuned encoder on the tested tasks (ScanNet ground truth vs. VFM-based data produces ~0 performance difference). But this result is limited to the specific models and datasets tested and does not guarantee robustness to model errors in all domains.

The consequence: SpatialBoost cannot be applied—at least not without substantial modification—to domains where depth estimation, segmentation, or 3D reconstruction models are unavailable, inaccurate, or inapplicable. Examples include:

Non-RGB modalities (medical CT/MRI, infrared, radar, sonar): Depth Pro, SAM, and VGGT are designed for RGB natural images and will fail on these modalities.
Microscopy and scientific imaging: The scale of spatial relationships (micrometers vs. meters) and the nature of 3D structure (translucent objects, overlapping cells) differ fundamentally from the natural scenes these models are trained on.
Abstract or diagrammatic spatial reasoning: The pipeline extracts 3D point clouds, which assumes the visual content depicts a physical 3D scene. For diagrams, charts, architectural plans, or abstract spatial puzzles, there is no 3D structure to extract, so the pipeline cannot generate spatial reasoning data.
Low-resource domains: Even within natural images, applying Depth Pro, SAM, and VGGT to domains with unusual visual statistics (e.g., underwater imagery, nighttime scenes, extreme weather conditions) may produce unreliable 3D extractions, and the bias propagation analysis only tested on ScanNet (indoor, well-lit, standard cameras).

The paper's framing as a "scalable framework" (Abstract, Section 1) should therefore be qualified: it is scalable within the domain of natural RGB images for which high-quality 3D extraction models exist. For other domains, the framework requires either new extraction models (which may not exist) or a fundamentally different approach to obtaining spatial supervision.

What evidence exists in the paper: The bias propagation experiment in Appendix E.5 tests whether errors from Depth Pro and SAM propagate into the fine-tuned encoder. The result—negligible performance difference between VFM-based and ground-truth-based training—is encouraging but only tested on ScanNet, a standard indoor dataset. The paper does not evaluate on domains where extraction models might fail more catastrophically (e.g., medical images where SAM produces nonsense segmentations). Furthermore, the experiment only tests whether errors propagate; it does not test whether missing spatial information (e.g., the extraction model fails to detect thin structures, transparent objects, or objects at extreme distances) leads to systematic blind spots in the fine-tuned encoder's spatial understanding.

Mitigation status: The paper partially addresses the concern of model error propagation through the ScanNet experiment (Appendix E.5) and acknowledges the limitation explicitly in Appendix F: "fundamentally addressing this concern would require large-scale ground-truth spatial annotations. However, collecting and annotating such data remains highly costly and largely unexplored." The authors leave "the curation of large-scale spatial ground-truth data and the development of more accurate vision foundation models as promising directions for extending our pipeline, and encourage the community to pursue these efforts." This is a fair acknowledgment but does not mitigate the limitation for practitioners who need spatial reasoning in domains without reliable extraction models. The dependency on extraction model availability and accuracy is a fundamental architectural constraint of the SpatialBoost framework, not a temporary limitation that can be overcome by better engineering of the existing pipeline.

7. Implications and Future Directions

How This Work Changes the Landscape

SpatialBoost does not introduce a new pre-training paradigm, architecture, or loss function. Its contribution is a reframing of what language can contribute to vision training: language can serve not just as semantic metadata (the CLIP-style alignment view) but as a structured, hierarchical encoding of 3D physical relationships that can be injected into vision encoders through decoder-based fine-tuning. This reframing is significant because it opens a new axis for enhancing pre-trained models—one that was not obvious from prior work and that produces qualitatively different results from pixel-level supervision.

The most important methodological shift is the demonstration that large language models can act as training signal translators, converting structured knowledge from specialized models (depth estimators, segmentors, 3D reconstructors) into a format that backpropagates useful gradients to a vision encoder. The ablation study in Table 6 is the empirical linchpin: the same spatial data, when decoded through a pixel-level supervision head (linear layer, SAM decoder, VGGT decoder), causes varying degrees of catastrophic forgetting, while LLM-based decoding simultaneously improves spatial and semantic capabilities. This finding—that the decoder architecture matters as much as the training data for knowledge injection—is not predicted by existing theories of transfer learning and represents a genuine discovery rather than an incremental refinement.

The work also resolves a tension that was latent but unarticulated in the literature. On one side, several recent papers (AIMv2, dino.txt, TIPS, PE-Core) showed that language can improve vision representations, but the improvements were concentrated on semantic and coarse localization tasks—the question of whether language could convey dense 3D spatial knowledge (metric depth, relative position, inter-object distances) was open. On the other side, the literature on 3D visual understanding (3D Diffuser Actor, Act3D, 3D Diffusion Policy) showed that explicit 3D supervision helps, but required multi-view data or simulation environments that limit scalability. SpatialBoost synthesizes these threads by showing that language can carry 3D knowledge, and that this knowledge can be extracted from single-view 2D images using off-the-shelf models, making the approach scalable in a way that direct 3D supervision is not.

The landscape shift is not that SpatialBoost replaces SSL pre-training or vision-language models—the paper is careful to position it as a post-training enhancement—but that it adds a new stage to the standard vision encoder lifecycle. The three-stage training pipeline (feature alignment → visual instruction tuning → spatial knowledge injection) suggests a template for how future vision encoders might be enhanced after large-scale pre-training: use specialized extraction models to convert implicit knowledge (3D structure, but potentially also physical dynamics, causal relationships, affordances) into linguistic form, then use an LLM decoder to inject that knowledge through fine-tuning. This is more modular and more data-efficient than joint pre-training from scratch.

The paper also sharpens the diagnosis of why existing vision-language models fail at spatial reasoning (a problem documented by Fu et al., Cheng et al., Wang et al., and others). The answer from SpatialBoost is not that spatial reasoning is inherently beyond the reach of vision-language training, but that standard image captions—the dominant training signal for vision-language models—do not encode spatial relationships in a structured, hierarchical form. When spatial relationships are encoded in this form, as in SpatialBoost's multi-turn reasoning dataset, vision-language training can produce substantial spatial understanding. This suggests that the spatial reasoning failures of current MLLMs are a data problem (the captions don't describe 3D structure) rather than an architectural or capability problem, and that fixing the data—by enriching captions with structured spatial descriptions—could yield significant improvements without architectural changes.

Follow-Up Research This Work Enables

Direct spatial reasoning probing of enhanced encoders. The paper demonstrates improved depth estimation, segmentation, and 3D task performance, but never measures whether spatial concepts (depth ordering, relative position, inter-object distance) are linearly decodable from the encoder's representations. A natural follow-up would train linear probes on the [CLS] token of SpatialBoost-enhanced encoders to answer the exact spatial questions from the reasoning dataset without the LLM decoder (e.g., "Is object A to the left of object B?" → binary classification, "What is the depth at pixel (x, y)?" → regression). This would isolate whether spatial knowledge lives in the encoder's representations or requires the LLM decoder at inference time, and would allow measuring which spatial concepts are learned and which remain dependent on the decoder. The finding that the LLM decoder matters for training (Table 6) but may or may not be necessary for inference would clarify the mechanism of knowledge injection.

Applying the SpatialBoost framework to other structured knowledge domains. The data generation pipeline—extract structured knowledge from specialized models, convert to hierarchical linguistic QAs, inject via LLM decoding—is architecturally general. A concrete follow-up would test this template on temporal dynamics (extracting object trajectories and event sequences from video using tracking and action recognition models, then generating multi-turn temporal reasoning QAs about "what happens next" or "how long between events"), physical affordances (extracting object materials, masses, and stability from physics simulators applied to 2D images, generating QAs about "can object A support object B?" or "which object would roll down the slope?"), or causal relationships (extracting causal graphs from video understanding models, generating QAs about "what caused the cup to fall?"). The hypothesis—that language-structured knowledge injection works for any domain where specialized models can extract the relevant structure—would be supported if these variants produce gains analogous to Table 1-3, and refuted if spatial reasoning is uniquely suited to this approach (e.g., because depth has a natural mapping to language that causal graphs lack).

Cross-domain generalization of SpatialBoost spatial features. The paper's experiments are confined to natural images of physical scenes—mostly indoor, human-scale, close-range. A critical stress-test would evaluate SpatialBoost-enhanced encoders on domains with systematically different depth statistics and visual appearance: medical imaging (CT/MRI slices requiring 3D organ relationship understanding), satellite imagery (overhead views with building heights and terrain), underwater imagery (different optical properties, depth ranges), microscopy (micron-scale 3D structure), and abstract spatial reasoning (Raven's Progressive Matrices, block design tasks). The existing results hint at a domain gap—DINOv3 improves 19.4% on NYUd (indoor) vs. 8.9% on KITTI (outdoor driving)—but this needs systematic characterization. A finding that SpatialBoost features transfer well to medical imaging (where spatial relationships are critical but the base models are trained on natural images) would dramatically expand the framework's scope; a finding that transfer fails would identify a fundamental limitation and motivate domain-specific extraction models.

Weight merging or distillation to eliminate the dual-channel attention overhead. The dual-channel attention mechanism adds 25–30% parameter overhead, but the paper does not test whether this overhead is necessary at inference time or can be eliminated post-training. A concrete experiment: after Stage 3 training, attempt to merge the original and copy attention weights into a single layer. Since the dual-channel output is α · Attn(x) + (1 − α) · Attn+(x), and both operations are linear in their inputs (for the value and output projections, though the attention softmax introduces nonlinearity), it may be possible to approximate the blended output with a single attention layer whose weights are a linear combination of the original and copy weights, weighted by the learned α. An alternative approach: distill the enhanced encoder's features back into the original (non-dual-channel) architecture using the spatial reasoning dataset as supervision. If either approach preserves the performance gains of Table 1-5 while returning the model to its original parameter count, SpatialBoost becomes a training-time-only cost with no deployment penalty—dramatically increasing its practical appeal. If merging fails (because the original and copy channels learn complementary attention patterns that cannot be linearly combined), it would suggest that the dual-channel mechanism is providing a genuine capacity expansion, not just preventing forgetting, and that the cost is fundamental.

Scaling the spatial reasoning dataset by an order of magnitude. The scalability experiments (Figure 5, Table 18) show monotonic improvement from 50K to 300K samples with no sign of saturation. A natural follow-up would test whether scaling to 1M or 3M reasoning samples continues to yield gains, and whether the benefits follow a power-law characteristic of neural scaling laws. This experiment would require optimizing the data generation pipeline (GPT-4o generation at this scale is expensive; template-based QA generation with structured 3D metadata may suffice) and would reveal whether spatial knowledge injection is fundamentally data-hungry (gains accrue slowly with more data) or data-efficient (most of the benefit comes from the first 100K samples, as the depth estimation scaling curves in Figure 5a-b suggest). If the gap between 100K and 300K samples is modest for depth (AbsRel barely changes for SigLIPv2 and DINOv3 in Figure 5) but larger for segmentation (Figure 5c), this would suggest different data scaling laws for different spatial capabilities—a practically important finding for resource allocation in data generation.

Multi-task pixel-level decoder comparison to isolate the LLM's advantage. The Table 6 ablation compares the LLM decoder (which receives diverse, multi-turn spatial supervision) against several pixel-level decoders (each trained on a single spatial modality: depth or segmentation). A more diagnostic experiment would train a multi-task pixel-level decoder that simultaneously predicts depth, segmentation masks, object bounding cubes, and inter-object distances—matching the diversity of the LLM's supervision as closely as possible without using language as the medium. If this multi-task decoder approaches the LLM's performance, it would suggest that the LLM's advantage is primarily about supervision diversity rather than the linguistic format of the supervision. If the multi-task decoder still underperforms the LLM significantly, it would strengthen the claim that language provides a uniquely effective training signal, perhaps because the sequential, hierarchical structure of the linguistic QAs imposes a beneficial curriculum or because the LLM's pretraining provides a useful inductive bias for structuring the knowledge.

Practical Applications and Downstream Use Cases

Robotics and embodied AI with limited 3D sensing. The CortexBench results (Table 4) demonstrate that SpatialBoost-enhanced DINOv3 improves average success rate from 72.8% to 80.8% using only monocular RGB input without depth sensors. This is directly applicable to low-cost robot platforms that lack dedicated depth cameras—drones, consumer robot vacuums, educational robot arms—where adding SpatialBoost to the vision encoder could provide substantial manipulation and navigation improvements without hardware changes. The 92.0% success rate on MetaWorld tasks with DINOv3 + SpatialBoost (vs. 83.8% base) is near saturation and suggests that standard manipulation benchmarks become solvable with appropriately enhanced visual representations. Production deployment would require running the SpatialBoost fine-tuning once (an offline cost) and then deploying the enhanced encoder with the 25–30% parameter overhead, which is modest for server-side or on-robot compute.

Single-view 3D scene understanding for AR/VR and content creation. The Lexicon3D results (Table 3) show that SpatialBoost substantially improves geometric understanding (registration recall from 86.9% to 97.5% at 5cm precision, translation error halved from 0.10m to 0.06m for DINOv3) and 3D visual grounding (ScanRefer accuracy from 56.2% to 61.1%). These capabilities are directly relevant to augmented reality applications that need to localize virtual objects in 3D space from a single camera feed, and to content creation tools that generate 3D scenes from reference images. A SpatialBoost-enhanced encoder could serve as the visual backbone for monocular SLAM pipelines, 3D asset placement systems, or view synthesis models, providing more accurate geometry from single images without requiring multi-view capture. The geometric correspondence improvements at the 5cm threshold are practically useful for applications like virtual furniture placement, where errors larger than 5cm would cause visible misalignment.

API-based vision-language model enhancement without retraining. The MLLM results in Appendix E.6 (Tables 20–21) show that swapping a SpatialBoost-enhanced vision encoder into an existing MLLM improves spatial reasoning benchmarks: InternVL3-38B's BLINK score rises from 64.0 to 69.2, and MUIRBench from 63.8 to 70.7. This is practically significant because it means an organization deploying a closed-source or frozen MLLM through an API could enhance its spatial reasoning simply by replacing the vision encoder—if the API exposes the encoder separately or if the MLLM's architecture supports encoder swapping. The 6.9-point MUIRBench improvement for Qwen3-VL (72.8 → 76.4) on a multi-image understanding benchmark directly addresses a known weakness of current MLLMs (Fu et al.'s finding that they "can see but not perceive" 3D). For applications involving spatial question-answering from images—real estate virtual tours, insurance claim photo analysis, warehouse inventory tracking from shelf photos—the encoder swap could provide meaningful accuracy improvements without the cost of re-training the entire MLLM.

Cost-efficient spatial data annotation and quality assurance. The reasoning data generation pipeline—which uses Depth Pro, SAM, VGGT, and GPT-4o to automatically produce spatial QAs from unlabeled images—could serve as a spatial annotation engine for datasets that currently lack 3D labels. For instance, an e-commerce company with millions of product photos could run the pipeline to generate depth maps, object segmentations, relative position labels, and inter-object distances, then use this data to train downstream models for product dimension estimation, automated photo composition, or packaging optimization. The bias propagation analysis (Appendix E.5) showing negligible difference between VFM-based and ground-truth-based training data suggests that the automatically generated labels are sufficiently high-quality for training purposes, even if individual labels contain errors. The main bottleneck is the computational cost of running Depth Pro, SAM, and VGGT at scale, which may be prohibitive for billion-image datasets but manageable for curated subsets.

When to Prefer This Method

The paper does not explicitly articulate a decision framework comparing SpatialBoost against named alternatives for enhancing vision encoders with spatial knowledge. The closest it comes is the comparison against prior methods in Tables 1–5, which shows that SpatialBoost outperforms TIPS, PE-Core, AIMv2, and dino.txt across dense prediction, 3D understanding, and robotics tasks, but this is an empirical rank ordering rather than a principled tradeoff analysis. The paper also does not discuss the conditions under which one would prefer SpatialBoost to the alternative paradigm of training vision encoders directly on 3D data (multi-view training, explicit 3D feature prediction, or simulation-based pre-training), beyond noting in Section 1 that those approaches face scalability constraints.

Since the paper does not articulate a clear "prefer SpatialBoost when X, prefer alternative Y when Z" framework with specific decision criteria, I do not include a forced decision matrix. The experimental results imply that SpatialBoost is preferable when: (1) you have a pre-trained vision encoder that you do not want to re-train from scratch, (2) you have access to or can run the 3D extraction models required for data generation, and (3) your target domain consists of natural RGB images of physical scenes. These conditions are implicit in the paper's design but are never presented as a decision rule.