From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors

ArXiv: 2510.17439

🎯 Pitch

Robot policies built on 2D vision-language models consistently fail at spatial reasoning, but FALCON shows that injecting off-the-shelf 3D foundation model tokens into the action head—rather than the backbone—preserves language alignment while enabling state-of-the-art real-world manipulation, including a 41.7% success rate on a difficult drawer-and-apple task where competitors score near zero. Strikingly, adding depth or pose at inference time only helps if the model already has sufficient spatial capacity, revealing that pretraining scale, not sensor richness, is the real bottleneck on the hardest problems.

---

1. Executive Summary

This paper introduces FALCON (From Spatial to Action), a vision-language-action model that injects rich 3D spatial tokens from spatial foundation models into the action head of a VLA rather than into the vision-language backbone, addressing the spatial reasoning gap that limits generalization in existing 2D-VLM-based robot policies. Built on a Kosmos-2 VLM backbone (1.6B parameters) and evaluated across three simulation benchmarks (CALVIN, SimplerEnv on WidowX and Google Robot setups) and 11 real-world manipulation tasks, FALCON achieves state-of-the-art performance through two complementary mechanisms—an Embodied Spatial Model that optionally fuses depth and camera pose when available without architectural changes (e.g., RGB-only depth prediction achieves 90.91% δ<1.25, improving to 99.79% when depth is added), and a Spatial-Enhanced Action Head that fuses spatial tokens with semantic action representations via element-wise addition rather than disrupting the VLM’s pretrained alignment. The method consistently outperforms competitive baselines, improving the average success rate in real-world base tasks by 25.6% over SpatialVLA (70.0% vs. 44.4%) and achieving an impressive 41.7% on the challenging "Open Top Drawer and Place Apple" task where most baselines score near zero, while establishing that test-time compute from optional 3D modalities provides complementary gains only when the base model already possesses the spatial reasoning capacity to exploit them—on the hardest problems, pretraining-scale capability remains the binding constraint.

2. Context and Motivation

The Core Problem: VLAs Operate in 3D but Reason in 2D

The fundamental problem this paper addresses is a spatial reasoning gap in modern vision-language-action (VLA) models for robotics. VLAs are the dominant paradigm for building generalist robot policies that can interpret natural language instructions (e.g., "pick up the yellow pepper and place it on the cutting board") and produce appropriate motor commands. These models are typically built by fine-tuning Vision-Language Models (VLMs)—which are trained on 2D internet images and text—on robot manipulation data. The VLMs provide strong semantic understanding: they know what a "yellow pepper" looks like and what "pick up" means linguistically. But they were never designed to reason about the three-dimensional physical world.

This creates a critical mismatch: VLAs must act in 3D space, but their core perceptual backbone operates entirely in the 2D image plane. As the paper states in Section 1:

"while VLMs operate purely in the 2D domain, VLAs must interact with the 3D physical world. This discrepancy results in a critical gap: current VLAs lack reliable 3D spatial understanding, leading to persistent challenges in generalization and adaptability."

The consequences are concrete and well-documented in the literature they cite. Without explicit 3D awareness, VLAs exhibit two failure modes that the paper identifies as major bottlenecks for deploying reliable generalist robot policies:

1. Limited generalization to novel scenes and objects. VLAs fail to transfer robustly when visual conditions change—different backgrounds, lighting, object appearances, or distractor objects—because they rely on 2D appearance correlations rather than understanding the underlying 3D geometry of the scene. If a model learned to grasp a red block based purely on 2D pixel patterns, changing the tablecloth color or adding a distractor object can catastrophically break performance. This is not a hypothetical concern; it is one of the central challenges preventing VLAs from being deployed outside carefully controlled lab settings (Section 1, citing Ze et al., 2025).

2. Failure to adapt to geometric variations. VLAs struggle with changes in object scale, height, or spatial relationships that are trivially handled by any 3D-aware system. A policy trained to stack 4 cm blocks may fail completely when the blocks are 5 cm or 3 cm—not because the task is conceptually different, but because the model lacks an internal representation of "how big the block is" and "how far the gripper needs to move." Similarly, a policy trained to place a cup on a table at one height will fail if the cup is elevated by a few centimeters (via blocks underneath it), because it cannot reason about the absolute height of the target surface. The paper demonstrates these exact failures in their baselines (Section 4.2, Figure 6): RoboVLM either collides with larger blocks during placement or prematurely releases smaller blocks, while FALCON's 3D-aware policy adapts seamlessly.

These limitations are not merely incremental issues—they form, in the authors' words, "a major bottleneck to developing reliable generalist robotic policies" (Section 1). A generalist home robot that works in one kitchen but fails in another, or that can only handle objects of a specific size, is not truly general. The ability to reason about the 3D structure of the world is the binding constraint on VLA reliability.

Why This Problem Is Important

The significance of closing the spatial reasoning gap manifests at three levels: practical deployment, theoretical understanding of embodied AI, and the economics of robot learning.

Practical deployment. The commercial viability of generalist home and industrial robots depends on robustness under real-world variation. A robot that needs to be re-trained for every new kitchen layout, every new object geometry, and every lighting condition is economically infeasible at scale. The paper's real-world experiments (Section 4.2) are designed to stress-test exactly these conditions—cluttered scenes with random distractors, unseen object poses, novel task descriptions, varying object sizes and target heights—representing the types of variation a deployed robot would encounter daily. The fact that leading baselines like SpatialVLA drop from 50% to 10–27.5% success in unseen conditions (Tables 9–13) while FALCON maintains 60–80% underscores the practical importance of robust spatial understanding.

Theoretical significance. The paper engages with a deeper question about how VLAs should be architected. The dominant approach—fine-tuning 2D VLMs on robot data—treats the VLM as a monolithic reasoning engine and adds an action head as a downstream output layer. But this implicitly assumes that the VLM's internal representations, learned from 2D internet data, are sufficient for 3D reasoning tasks. The paper challenges this assumption, arguing instead for a division of labor: the VLM handles semantic reasoning (what to do and in what order) while a separate spatial processing stream handles geometric reasoning (where things are and how to reach them). This brain-inspired architecture—explicitly analogized to the cerebrum and cerebellum in Section 1—represents a principled design philosophy that may generalize beyond the specific implementation tested here.

Economics of robot learning. Training VLAs on large-scale robot datasets (like the Open X-Embodiment dataset with its ~1 million trajectories) is expensive enough that redoing it for every new sensor configuration is prohibitive. Many prior 3D-enhanced VLA approaches require specific sensor inputs (point clouds from structured light cameras, depth maps from specific sensors) and fail entirely when those inputs are unavailable—a property the paper calls low modality transferability. As the paper notes (Section 1):

"many large-scale manipulation datasets (e.g., Open X-Embodiment dataset) lack aligned 3D annotations, limiting scalability."

This creates an economic tension: 3D sensors improve robot performance but drastically reduce the amount of training data available and tie the policy to specific hardware. A method that can ingest 3D data when available but degrade gracefully to RGB-only operation (which is what FALCON's Embodied Spatial Model provides) is therefore not just a technical improvement but an economically necessary one for scaling robot learning.

Prior Approaches and Where They Fall Short

The paper organizes prior work on 3D-enhanced VLAs into two categories, each with distinct failure modes (Section 2.1).

Category 1: Explicit 3D Input Methods

These methods directly consume 3D representations—typically point clouds from depth cameras—as input to the policy. Representative works include PointVLA (Li et al., 2025), which injects point cloud encodings into the VLA, and GeoVLA (Sun et al., 2025), which similarly relies on explicit 3D geometry. 3D diffusion policy methods (Ze et al., 2024; Ke et al., 2024) also fall into this category.

What they get right. By providing explicit 3D information (point clouds encode the exact 3D coordinates of every observed surface point), these methods achieve strong geometric awareness. The 3D Diffuser Actor (Ke et al., 2024), for instance, achieves 93.8% on the first task and 53.3% at task 4 in the CALVIN ABC→D benchmark (Table 1), demonstrating effective use of 3D geometry.

Where they fall short. The paper identifies low modality transferability as the fundamental weakness. These methods are trained end-to-end assuming specific 3D sensor inputs are always available. Three problems cascade from this:

1. Sensor dependency. At deployment, they require specialized sensors (stereo cameras, LIDAR, structured light) that may be unavailable due to cost, power, or form-factor constraints. An RGB-D camera costs more and consumes more power than a simple RGB camera; a LIDAR unit is even more expensive and bulky.

2. Dataset incompatibility. The largest available robot manipulation datasets—most notably the Open X-Embodiment dataset with 1M+ trajectories from 33 different robot platforms—overwhelmingly lack aligned 3D annotations. Training methods that require point clouds or registered depth maps cannot leverage this data at scale, which limits their ability to learn general semantic skills from broad data.

3. Brittleness to missing inputs. If the depth sensor fails (common in outdoor or high-glare environments), policies trained to expect point cloud inputs produce undefined behavior. They do not degrade gracefully; they break completely. The paper explicitly notes (Section 1): "such methods are tied to specific input modalities and break down when those inputs are unavailable."

This brittleness is visible in the CALVIN results (Table 1): 3DDP (3D Diffusion Policy, Ze et al., 2024) achieves only 28.3% on the first task and 0% on tasks 3–5 in the ABC→D split despite using ground-truth point clouds. The existence of FALCON, which outperforms these methods using only RGB input at inference time (and optionally using depth when available), demonstrates that explicit 3D input is neither necessary nor sufficient for strong spatial reasoning—the architecture matters more than the sensor.

Category 2: Weak 3D Cue Methods

Rather than requiring explicit 3D sensors, these methods inject approximate or learned 3D information into the VLA. Examples include pseudo-depth estimates from monocular depth models like ZoeDepth (Bhat et al., 2023), learnable spatial embeddings as in SpatialVLA (Qu et al., 2025), and other approaches that encode 3D features into the VLM's token space (3D-VLA, Zhen et al., 2024; Evo-0, Lin et al., 2025).

What they get right. These methods are sensor-agnostic—they work with RGB input alone, removing the hardware dependency of Category 1. SpatialVLA, for instance, achieves strong results on SimplerEnv tasks (100% on Put Eggplant in Basket, 42.7% overall on WidowX tasks), demonstrating that some 3D awareness can be extracted from RGB images.

Where they fall short. The paper identifies three fundamental limitations (Section 1):

(1) Limited spatial representation. The 3D signals provided by these methods are weak—they provide coarse, approximate depth or learned embeddings that lack the geometric precision needed for fine manipulation. A pseudo-depth map from a monocular estimator like ZoeDepth gives relative depth ordering ("this object is in front of that one") but not accurate metric distances ("the block is exactly 15.3 cm from the gripper"). When a robot needs to grasp an object of a specific size or place something at a precise height, relative depth is insufficient. The paper's Spatial Understanding experiments (Figure 6, Section 4.2) demonstrate this concretely: baseline methods like RoboVLM, which use no explicit 3D priors beyond what the VLM implicitly learns, either collide with large blocks or prematurely release small blocks, while FALCON's stronger spatial priors from a foundation spatial model enable robust size-adaptive behavior.

(2) Lack of modality transferability—in the opposite direction. While Category 1 methods fail without 3D sensors, Category 2 methods fail to exploit higher-quality 3D inputs when they are available. SpatialVLA, for instance, is designed around learned spatial embeddings from RGB images and has no mechanism to absorb explicit depth or pose measurements when the robot has them. This means the policy leaves performance on the table—it cannot benefit from the RGB-D cameras that are already prevalent on research robots (like the Intel RealSense D435i used in the paper's own real-world setup). The paper frames this as a modality transferability failure in the reverse direction: the method transfers poorly from RGB-only to RGB-D, not just from RGB-D to RGB.

(3) Challenges in vision-language alignment. When 3D features are injected directly into the VLM's token space (concatenated with text and image tokens), they risk disrupting the carefully learned alignments between visual and linguistic representations that make VLMs powerful in the first place. The VLM was pre-trained on billions of image-text pairs to develop representations where, for instance, the concept "banana" is close to visual features of bananas and linguistic features of the word "banana." Injecting a new modality—3D spatial tokens—into this space can push representations in unpredictable directions, a phenomenon the paper calls embedding drift. The problem is especially acute because 3D robotics data is scarce compared to internet-scale vision-language data; there is simply not enough 3D-annotated robot data to re-align all modalities from scratch. The paper explicitly notes (Section 1):

"The scarcity of 3D data makes it difficult to properly align modalities, causing embedding drift that degrades zero-shot generalization, especially in tasks requiring high-level reasoning like spatial prompts."

This is not just speculation—the paper's own ablation (Table 7 in Appendix 10) tests a variant called FALCON VLM-tokens where spatial tokens from the Embodied Spatial Model are injected directly into the VLM (following the paradigm of most 3D-based VLAs). The results show significant degradation: in the zero-shot CALVIN ABC→D setting, average task length drops from 3.91 to 3.79, and success on task 5 drops from 62.2% to 57.6%. This is the same model, the same spatial tokens, the same training data—the only difference is where the spatial information is injected. The performance decline provides direct causal evidence for the alignment disruption hypothesis.

How This Paper Positions Itself

FALCON is positioned as a unifying architecture that resolves the limitations of both prior paradigms through three design principles, each directly targeted at one of the identified failure modes (Section 1, "Contribution").

Against weak spatial representation (Limitation 1): FALCON replaces pseudo-depth or learned embeddings with rich spatial tokens from spatial foundation models—specifically, VGGT (Wang et al., 2025a), which is pre-trained on large-scale 3D reconstruction tasks. Unlike ZoeDepth or learned embeddings, VGGT encodes scenes into dense token sequences that capture holistic 3D structure: per-pixel metric depth, point maps (3D coordinates for every pixel), and camera pose. These tokens provide strong geometric priors—not approximate relative depth, but actual 3D understanding—extracted from RGB alone. The paper demonstrates this quantitatively: FALCON's Embodied Spatial Model achieves 90.91% δ<1.25 on monocular depth estimation from RGB only (Table 5), which is real metric depth, not just relative ordering.

Against modality transferability failures (Limitation 2): FALCON's Embodied Spatial Model introduces a stochastic conditioning strategy during training. Specifically (Section 3.3, Equation 4), depth and pose tokens are randomly injected or withheld using Bernoulli sampling with probability $p$:

$(\mathbf{T}_{\text{spl}},\hat{\mathbf{t}}_{\text{cam}}) = \mathcal{E}_{\text{spl}}\big(\mathbf{T}_{\text{vis}} + b_{\text{d}}\mathbf{T}_{\text{dpt}},\; b_{\text{p}}\mathbf{t}_{\text{gt-cam}} + (1-b_{\text{p}})\mathbf{t}_{\text{cam}}\big)$

where $b_{\text{d}}, b_{\text{p}} \sim \text{Bernoulli}(p)$. This means the same model learns to process pure RGB input (when $b_{\text{d}} = 0, b_{\text{p}} = 0$), RGB-D input (when $b_{\text{d}} = 1$), RGB-pose input (when $b_{\text{p}} = 1$), or RGB-D-pose input (when both are 1). At inference time, the model works with whatever inputs are available, and its spatial representations improve when more modalities are provided—as shown in Table 5, where depth prediction δ<1.25 jumps from 90.91% (RGB-only) to 99.79% (RGB-D). This is genuine modality transferability: the model degrades gracefully when sensors are limited and improves immediately when more sensors are added, all without architectural changes.

Against alignment disruption (Limitation 3): Rather than injecting spatial tokens into the VLM's input stream—which the ablation in Table 7 shows degrades generalization—FALCON routes spatial tokens to a separate Spatial-Enhanced Action Head. The action head fuses a single compressed spatial vector (obtained by max-pooling the spatial token sequence) with the VLM's semantic action token via element-wise addition (Section 3.4, Equation 5):

$\mathbf{f}_{\text{fused}} = \hat{\mathbf{t}}_{\text{act}} + \widetilde{\mathbf{t}}_{\text{spl}}$

This design preserves the VLM's pre-trained vision-language alignment completely—the VLM never "sees" the spatial tokens and thus cannot have its representations disrupted by them. The spatial information enters only at the action prediction stage, where geometric reasoning naturally belongs. The paper's ablation (Table 7) validates this choice empirically: element-wise addition in the action head outperforms cross-attention and FiLM-gated modulation, and the VLM-tokens variant (injecting into the VLM) consistently underperforms.

The paper also positions itself through a two-stage training paradigm (Appendix 6.1) that further protects the VLM's pre-trained representations. Stage 1 freezes all pre-trained components (VLM, action head, Embodied Spatial Model) and trains only a lightweight adapter with a zero-initialized final linear layer—ensuring spatial information is introduced gradually. Stage 2 then unfreezes the VLM for joint refinement, but only after the adapter has learned to align spatial and semantic features. This phased approach prevents the spatial features from "overwhelming the semantic representations during initial learning phases."

Summary of the Gap and FALCON's Positioning

The field of VLA research was caught between two unsatisfactory options: methods that use strong 3D sensors but break without them, and methods that work from RGB alone but provide only weak geometric understanding and disrupt the VLM's semantic representations. FALCON argues that this tradeoff is false—it is possible to have (1) strong geometric priors from RGB alone, (2) optional use of 3D sensors when available for further improvement, and (3) preserved vision-language alignment by routing spatial information to the action head rather than the VLM backbone. The paper's three contributions directly map to resolving the three limitations of prior work, and the extensive empirical evaluation is designed to prove that all three resolutions are necessary and that their combination achieves state-of-the-art performance.

3. Technical Approach

3.1 Reader Orientation

FALCON is an end-to-end neural network policy for robot manipulation — given a camera image and a natural language command (e.g., "put the banana on the red plate"), it outputs the sequence of motor commands (gripper pose, open/close state) needed to execute the task. The system addresses the problem that existing vision-language-action models reason about the world in flat 2D terms but must act in a physical 3D space, and it solves this by attaching a dedicated 3D perception pipeline (the Embodied Spatial Model) to an existing 2D vision-language model, then fusing their outputs only at the final motor-control stage rather than forcing 3D information into the language model's internal representations.

3.2 Big-Picture Architecture (Diagram in Words)

FALCON has three major components connected in a feedforward pipeline at each timestep $t$:

1. 2D Vision-Language Model (VLM) — a pre-trained Kosmos-2 model (~1.6B parameters) that takes as input the camera image(s) $O_t$ and the language instruction $L$, and produces a semantic action token $\hat{\mathbf{t}}_{\text{act}}$, a single vector in $\mathbb{R}^{D_{\text{act}}}$ that encodes what task needs doing and how to do it in semantic terms.

2. Embodied Spatial Model (ESM) — a spatial foundation model (~1.0B parameters) based on VGGT that takes the same third-view image $I_t^{\text{3rd}}$ (plus optional depth $D_t$ and camera pose $P$ when available) and produces a set of spatial tokens $\mathbf{T}_{\text{spl}} \in \mathbb{R}^{M \times D_s}$, where $M$ is the number of tokens and $D_s$ is the token dimension. These tokens encode the 3D geometry of the scene — per-pixel depth, surface point coordinates, and camera viewpoint.

3. Spatial-Enhanced Action Head — a lightweight fusion module that compresses the spatial tokens into a single vector $\mathbf{t}_{\text{spl}}$ via max-pooling, projects it to the VLM's feature dimension via an MLP adapter, adds it element-wise to the semantic action token $\hat{\mathbf{t}}_{\text{act}}$, and feeds the fused vector into an action predictor (either MLP or LSTM) that outputs the 7-DoF action chunk $A_t = [a_t, \ldots, a_{t+C-1}]$.

Information flow: At timestep $t$, the third-view image enters both the VLM (alongside language instructions and optionally wrist-view images) and the ESM. The VLM outputs a semantic action token; in parallel, the ESM outputs spatial tokens. The two streams converge only in the action head, where element-wise addition fuses them into a single control vector. This means the VLM never "sees" the spatial tokens — its internal representations remain untouched — and all geometric reasoning enters the pipeline at the motor-command stage, where it belongs.

3.3 Roadmap for the Deep Dive

• First, the formal problem definition and notation (Equation 1), which establishes the input/output contract and the optional modality framework.
• Second, the Embodied Spatial Model in full detail — how spatial tokens are generated from RGB alone, how depth and pose are optionally injected during training, the stochastic conditioning strategy that enables modality transferability, and the reconstruction-based training objective.
• Third, the Spatial-Enhanced Action Head — the max-pooling compression, the MLP adapter, the three fusion strategies (cross-attention, FiLM-gated, element-wise addition) and why element-wise addition wins, and the two action predictor architectures (MLP vs. LSTM).
• Fourth, the two-stage training paradigm — Stage 1 alignment with frozen backbones and zero-initialised adapter, Stage 2 joint refinement with unfrozen VLM, and the loss function (MSE + BCE) used throughout.
• This order matters because the ESM is the novel perceptual contribution, the action head is the architectural novelty, and the training paradigm is the procedure that makes both work together without destroying pre-trained representations.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems architecture paper whose core idea is that 3D spatial understanding in VLAs should be provided by a dedicated spatial foundation model whose outputs are fused at the action-decision stage, not injected into the language model.

---

Problem Definition and Notation

The paper formalises the robot control problem as a mapping function $\mathcal{F}(\cdot)$ that transforms sensory observations and language into motor commands (Section 3.1). At each timestep $t$, the robot receives:

• Visual observations $O_t = \{I_t^1, \ldots, I_t^n\}$ — one or more camera images, typically a static side camera $I_t^{\text{3rd}}$ providing global scene context and optionally a wrist-mounted camera $I_t^{\text{hand}}$ for fine-grained object detail.
• A language instruction $L$ — a natural language sentence describing the task, e.g., "lift the yellow pepper" or "put the red coke can on the bottom shelf."
• Optional depth maps $D_t \in \mathbb{R}^{H \times W}$ — per-pixel metric distances from the camera.
• Optional camera poses $P \in \mathbb{R}^7$ — encoding the intrinsic and extrinsic parameters (a 7-dimensional representation of the camera's position and orientation).

The goal is to produce an action chunk $A_t = [a_t, \ldots, a_{t+C-1}]$, a sequence of $C$ future actions, where $C$ is the action horizon (how many timesteps into the future the model predicts). Each individual action $a_i$ is a 7-dimensional vector: six dimensions encode the 6-DoF gripper pose (position and orientation, typically as Euler angles), and one dimension is a binary gripper open/close state. This is absolute Cartesian control: the model directly outputs the target end-effector pose.

Formally, the mapping is:

$A_t = \mathcal{F}(O_t, L, D_t, P)$

where $O_t$ are the image observations, $L$ is the language instruction, $D_t$ is the optional depth, and $P$ is the optional camera pose.

What it computes: a function that, given all available sensory and linguistic inputs at time $t$, predicts the next $C$ actions the robot should execute. This is a standard behaviour-cloning formulation: the model learns to imitate expert demonstrations by mapping observations to actions.

Why this form: the optionality of $D_t$ and $P$ is the crucial design choice. Rather than hard-coding a specific sensor suite (as in prior 3D VLA methods that require point clouds), the function accepts whatever geometric information is available. When depth and pose are present, the model uses them; when they are absent, the model still functions from RGB alone. This optionality is what enables modality transferability — the ability to deploy the same trained policy across different hardware configurations without architectural changes or retraining.

The focus on table-top manipulation with a static side camera and robot arm establishes the domain: the robot sees the workspace from a fixed external viewpoint (providing global scene context) and controls a single 6-DoF arm with a parallel gripper. This is the standard setup for most research manipulation platforms (the paper uses an xArm 6 with a Robotiq gripper and Intel RealSense D435i camera).

---

Embodied Spatial Model: Architecture and Spatial Token Generation

The Embodied Spatial Model (ESM) is the component that extracts 3D geometric information from images. It is based on VGGT (Visual Geometry Grounded Transformer, Wang et al., 2025a), a recent spatial foundation model that predicts dense 3D structure from single or multiple images. The paper adopts VGGT's architecture but extends it with the ability to ingest optional depth and pose inputs during training — this is the key innovation that enables modality transferability.

Base Architecture: From Image to Spatial Tokens

The ESM processes a single third-view image $I_t$ through the following stages (Section 3.3):

Step 1: Visual tokenisation. The image is first encoded into visual tokens $\mathbf{T}_{\text{vis}}$ using a frozen DINOv2 model (Oquab et al., 2024). DINOv2 is a self-supervised vision transformer pre-trained on diverse internet images that produces dense, semantically meaningful features — each token corresponds to a patch of the input image and captures both appearance and coarse spatial layout. The authors do not fine-tune DINOv2; it serves as a fixed feature extractor providing rich initial representations to the spatial encoder.

Step 2: Camera token concatenation. A learnable camera token $\mathbf{t}_{\text{cam}} \in \mathbb{R}^{D_s}$ is created, where $D_s$ is the dimension of the spatial token space. This token is a learned embedding that represents the camera viewpoint — it starts as a random vector and, through training, learns to encode the extrinsic and intrinsic parameters implicitly when no explicit pose is provided. The visual tokens $\mathbf{T}_{\text{vis}}$ and the camera token $\mathbf{t}_{\text{cam}}$ are concatenated (the camera token is appended to the sequence of visual tokens).

Step 3: Spatial encoding via cross-attention and self-attention. The concatenated sequence is fed into the Spatial Encoder $\mathcal{E}_{\text{spl}}(\cdot)$, which consists of $N$ alternating cross-attention and self-attention blocks. Cross-attention allows the camera token to attend to all visual tokens, aggregating viewpoint-related information; self-attention allows all tokens to interact, building global geometric consistency. The encoder outputs:

• Spatial tokens $\mathbf{T}_{\text{spl}} \in \mathbb{R}^{M \times D_s}$ — one token per image patch, each encoding the local 3D structure at that patch (depth, surface normal, and point coordinate information).
• Refined camera token $\hat{\mathbf{t}}_{\text{cam}} \in \mathbb{R}^{D_s}$ — the camera token after attending to the image, now encoding the estimated camera parameters.

Formally:

$(\mathbf{T}_{\text{spl}}, \hat{\mathbf{t}}_{\text{cam}}) = \mathcal{E}_{\text{spl}}(\mathbf{T}_{\text{vis}}, \mathbf{t}_{\text{cam}})$

where $\mathbf{T}_{\text{spl}}$ are the spatial tokens, $\hat{\mathbf{t}}_{\text{cam}}$ is the refined camera token, $\mathbf{T}_{\text{vis}}$ are the DINOv2 visual tokens, and $\mathbf{t}_{\text{cam}}$ is the learnable camera token.

What it computes: a function that takes patch-level visual features and a learned camera embedding, applies $N$ rounds of cross-attention and self-attention to build globally consistent 3D representations, and outputs per-patch spatial tokens plus a refined camera estimate.

Why this form: the attention architecture is chosen because 3D reconstruction requires reasoning about global consistency — the depth at one pixel constrains and is constrained by the depth at neighbouring pixels. Self-attention across all patches enables this global reasoning. Cross-attention with a camera token provides a mechanism for the model to learn about viewpoint implicitly (the camera token learns to represent the camera parameters through gradient-based optimisation) without requiring explicit pose supervision during training. The VGGT architecture was chosen over alternatives like DUSt3R or MASt3R because VGGT processes images holistically rather than in pairs, producing better reconstruction consistency, and because its token-based output format is naturally compatible with the token-based VLM architecture.

Step 4: Reconstruction heads. The spatial tokens $\mathbf{T}_{\text{spl}}$ are passed to a depth predictor (a lightweight decoder that outputs a per-patch depth map), and the refined camera token $\hat{\mathbf{t}}_{\text{cam}}$ is passed to a camera predictor (which outputs the estimated camera intrinsic and extrinsic parameters). These heads are used only during ESM training to provide reconstruction supervision; at FALCON inference time, only the spatial tokens $\mathbf{T}_{\text{spl}}$ are used, with the reconstruction heads discarded.

3D Condition Encoding: Ingesting Depth and Pose

The key innovation over VGGT is the ability to optionally inject explicit 3D measurements — depth maps and camera poses — when available. This is done through two specialized encoders that transform raw sensor data into token representations compatible with the spatial encoder (Section 3.3).

Depth encoding. Given a depth map $D_t \in \mathbb{R}^{H \times W}$ (per-pixel metric distances from the camera), the depth is first normalised:

$D'_t = D_t / \text{Norm}(D_t)$

where $\text{Norm}(D_t)$ is a normalising constant (likely the mean or maximum depth). Normalisation is critical because depth values can vary wildly across scenes — a depth of 0.5 m in a desktop setting versus 5.0 m in a room-scale scene — and without normalisation, the model would need to learn to handle arbitrary scale factors. Normalisation ensures the depth values lie in a consistent numerical range at train and test time, regardless of the absolute scene scale.

The normalised depth map $D'_t$ is concatenated channel-wise with its corresponding validity mask $M_{\text{dpt}} \in \mathbb{R}^{H \times W}$. The validity mask is a binary map indicating which pixels have valid depth measurements (1 for valid, 0 for missing). Depth sensors frequently produce holes — pixels where the depth is unknown due to occlusions, reflective surfaces, or sensor limitations — and the validity mask tells the model where to trust the depth signal versus where to rely on image-based inference. The concatenation $[D'_t \| M_{\text{dpt}}]$ is a 2-channel tensor (one channel for normalised depth, one for validity).

This 2-channel tensor is then processed by a depth encoder $\mathcal{E}_{\text{dpt}}(\cdot)$, which consists of a stack of convolutional layers with a kernel size of $14 \times 14$. The large kernel size is deliberate: it partitions the image into patches aligned with the DINOv2 visual token grid, ensuring each depth token corresponds to exactly one visual token. The output is a set of depth tokens:

$\mathbf{T}_{\text{dpt}} = \mathcal{E}_{\text{dpt}}([D'_t \| M_{\text{dpt}}]), \quad \mathbf{T}_{\text{dpt}} \in \mathbb{R}^{M \times D_s}$

where $[ \cdot \| \cdot ]$ denotes channel-wise concatenation, $\mathcal{E}_{\text{dpt}}$ is the depth encoder (a CNN), and $M$ is the number of tokens (matching the number of visual tokens $\mathbf{T}_{\text{vis}}$).

Pose encoding. Given a camera pose $P \in \mathbb{R}^7$ (encoding intrinsic and extrinsic parameters), a lightweight MLP-based camera encoder $\mathcal{E}_{\text{cam}}(\cdot)$ projects the 7-dimensional pose vector into the spatial token dimension $D_s$:

$\mathbf{t}_{\text{gt-cam}} = \mathcal{E}_{\text{cam}}(P)$

where $\mathbf{t}_{\text{gt-cam}} \in \mathbb{R}^{D_s}$ is a single token representing the ground-truth camera parameters in the spatial encoder's embedding space.

What these encodings compute: depth and pose encodings transform explicit 3D measurements — raw sensor data that the spatial encoder cannot directly consume — into the same dimensional token representation as the visual and spatial tokens, enabling them to be seamlessly injected into the attention-based spatial encoder.

Why normalise and mask depth: without normalisation, the model would see vastly different numerical ranges for near scenes (depths ~0.3) versus far scenes (depths ~5.0), making training unstable. Without the validity mask, the model would treat missing depth pixels as containing valid information (likely zero values), corrupting the learned geometry. The channel-wise concatenation of depth and mask is the minimal representation that preserves both the geometric signal and the uncertainty about where that signal is reliable.

Why MLP for pose: the camera pose is a low-dimensional structured vector (7 numbers representing translation, rotation, and intrinsics), not a spatial grid. An MLP is the natural encoder for fixed-dimensional vectors, providing a learned non-linear mapping from pose parameters to a token that the spatial encoder's attention mechanism can then use to condition all spatial token computations.

Stochastic Conditioning Strategy: Enabling Modality Transferability

The paper's central challenge is to build a model that works well with any subset of the available 3D modalities — RGB-only, RGB-D, RGB-pose, or RGB-D-pose — without retraining. This is achieved through a stochastic conditioning strategy during training (Section 3.3).

During each training forward pass, two binary flags are sampled independently:

$b_{\text{d}} \sim \text{Bernoulli}(p), \quad b_{\text{p}} \sim \text{Bernoulli}(p)$

where $p$ is the probability of injecting each modality (the paper does not specify the exact value of $p$, but the symmetric sampling suggests it is likely 0.5). The flags control whether depth and/or pose are used for that particular training sample:

• $b_{\text{d}} = 1$ means depth tokens are added to the visual tokens.
• $b_{\text{d}} = 0$ means no depth injection (the model must rely on RGB alone).
• $b_{\text{p}} = 1$ means the ground-truth camera token replaces the learned camera token.
• $b_{\text{p}} = 0$ means the learned camera token is used (no explicit pose).

The spatial encoder's input is then modified accordingly:

$(\mathbf{T}_{\text{spl}}, \hat{\mathbf{t}}_{\text{cam}}) = \mathcal{E}_{\text{spl}}\big(\mathbf{T}_{\text{vis}} + b_{\text{d}}\mathbf{T}_{\text{dpt}},\; b_{\text{p}}\mathbf{t}_{\text{gt-cam}} + (1-b_{\text{p}})\mathbf{t}_{\text{cam}}\big)$

where $\mathbf{T}_{\text{vis}} + b_{\text{d}}\mathbf{T}_{\text{dpt}}$ means depth tokens are element-wise added to visual tokens when $b_{\text{d}} = 1$ (otherwise just the visual tokens pass through), and $b_{\text{p}}\mathbf{t}_{\text{gt-cam}} + (1-b_{\text{p}})\mathbf{t}_{\text{cam}}$ interpolates between the ground-truth camera token and the learned camera token.

What it computes: a training procedure where, for each sample, the model randomly sees one of four input configurations — (RGB only), (RGB + depth), (RGB + pose), or (RGB + depth + pose). Over many training iterations, the spatial encoder learns to extract 3D structure from RGB alone (because it frequently sees samples with $b_{\text{d}} = 0, b_{\text{p}} = 0$) while also learning to exploit depth and pose cues when they are provided (because it also sees samples where those modalities are present).

Why element-wise addition for depth vs. replacement for pose: depth tokens are added to visual tokens because they provide complementary information — depth tells the model about metric distances, while visual features tell it about appearance and semantics. Adding them preserves both signals. The camera token, by contrast, is replaced (not added) because the learned camera token and the ground-truth camera token serve the same role — encoding the camera viewpoint — but the ground-truth version is objectively more accurate. Replacing the learned token with the ground-truth token provides a strict information gain without redundancy.

Why Bernoulli sampling: random dropping during training is the standard technique for building models robust to missing inputs at inference time (analogous to dropout in neural networks). By seeing all combinations of available modalities during training, the model learns to produce good spatial tokens regardless of which inputs are present. This is what enables modality transferability: the same trained model works with any sensor configuration because it has been explicitly trained to handle all of them.

ESM Training Objective

The ESM is trained with a multi-task reconstruction loss following VGGT's approach (Section 3.3). The exact loss weights and formulations are not detailed in the main paper, but the authors state they adopt "depth, point map, and pose losses to formulate multi-task supervision." These are standard reconstruction losses:

• Depth loss: the discrepancy between predicted per-pixel depth and ground-truth depth from the depth sensor (when available).
• Point map loss: the discrepancy between predicted 3D point coordinates for each pixel and ground-truth 3D points (derived from depth and camera parameters).
• Pose loss: the discrepancy between the predicted camera parameters and ground-truth camera extrinsics/intrinsics.

The ESM is trained on the same dataset and follows the same preprocessing as VGGT (Section 6.2): each training batch consists of 1–12 randomly sampled frames from a randomly selected scene, totalling 24 images per batch. Training uses the AdamW optimiser with differentiated learning rates: $1 \times 10^{-6}$ for the large unified transformer backbone (the spatial encoder) and $1 \times 10^{-5}$ for the depth, camera, and point heads (the reconstruction decoders). The differentiated rates reflect the fact that the backbone is already pre-trained (from VGGT) and needs only fine-tuning, while the reconstruction heads need to be trained for the specific depth/pose distribution of the robotic manipulation data. Training requires 16 A100 GPUs and runs for approximately 2 days.

Why differentiated learning rates: the spatial encoder backbone has already learned general 3D reconstruction capabilities from VGGT's pre-training on diverse internet data. A high learning rate would cause catastrophic forgetting of these pre-trained representations. The reconstruction heads, by contrast, are randomly initialised and need to adapt to the specific depth ranges and camera parameters of the robotics setting, requiring a higher learning rate. This is standard practice when fine-tuning large pre-trained models on domain-specific data.

Why multi-task supervision with depth, point maps, and pose: jointly supervising all three geometric outputs ensures that the spatial tokens $\mathbf{T}_{\text{spl}}$ encode a complete and consistent 3D representation. Depth alone provides per-pixel distances but not 3D coordinates (which require camera intrinsics). Point maps alone provide 3D coordinates but are harder to supervise directly. Pose supervision ensures the camera token learns to encode viewpoint information accurately, which is critical for tasks where the robot needs to reason about object positions relative to the camera. The combination of all three losses forces the spatial tokens to capture the full geometric structure of the scene, not just a single projection of it.

ESM Performance Under Different Modality Configurations

The paper provides a quantitative ablation of ESM depth prediction accuracy under different modality inputs on the CALVIN benchmark (Table 5, Section 4.3):

What these metrics mean: $\delta < 1.25$ is the standard thresholded accuracy metric for depth estimation — the percentage of pixels where the predicted depth is within 25% of the ground-truth depth (i.e., $\max(\text{pred}/\text{gt}, \text{gt}/\text{pred}) < 1.25$). Abs. Rel is the mean absolute relative error: $|\text{pred} - \text{gt}| / \text{gt}$, averaged over all pixels.

Interpretation: With RGB only (no depth, no pose), FALCON's ESM achieves performance comparable to VGGT — 90.91% vs. 91.33% $\delta < 1.25$, and 8.61 vs. 8.53 Abs. Rel. This is expected because, without additional modalities, the ESM is essentially VGGT. When ground-truth depth is provided during inference (Depth ✓, Camera ✗), accuracy jumps dramatically to 99.79% $\delta < 1.25$ and Abs. Rel drops to 0.91 — the model is nearly perfect at depth estimation because it receives the answer as input and only needs to propagate it through the attention mechanism. Adding camera pose as well (Depth ✓, Camera ✓) yields a marginal difference (99.47% vs. 99.79%), suggesting that depth alone provides most of the geometric signal for depth estimation.

Why this matters for the full FALCON system: the ESM's spatial tokens $\mathbf{T}_{\text{spl}}$ are the input to the action head. Better depth prediction accuracy (higher $\delta < 1.25$, lower Abs. Rel) means the spatial tokens encode more accurate 3D geometry. When depth sensors are available at inference time, the ESM produces substantially more precise spatial representations, which the action head can then use for more precise motor commands. This is the mechanism by which optional 3D modalities improve policy performance — not through a separate pathway, but by making the spatial tokens fed to the action head more accurate.

---

Spatial-Enhanced Action Head: Fusion and Action Prediction

The Spatial-Enhanced Action Head is where semantic understanding (from the VLM) meets 3D geometry (from the ESM). It performs three operations: compress the spatial tokens, fuse them with the semantic action token, and predict motor commands (Section 3.4).

Step 1: Compressing Spatial Tokens

The ESM outputs a set of spatial tokens $\mathbf{T}_{\text{spl}} \in \mathbb{R}^{M \times D_s}$ — a sequence of $M$ vectors, each representing the 3D structure at one image patch. Feeding an entire sequence into the action predictor would be high-dimensional and computationally expensive. The paper compresses the sequence into a single vector through max-pooling over the token dimension:

$\mathbf{t}_{\text{spl}} = \text{MaxPool}(\mathbf{T}_{\text{spl}}), \quad \mathbf{t}_{\text{spl}} \in \mathbb{R}^{D_s}$

where max-pooling takes the element-wise maximum across all $M$ tokens, producing a single vector of dimension $D_s$.

What it computes: for each feature dimension in the spatial token space, max-pooling selects the largest activation value across all spatial locations. The resulting vector captures the most salient geometric information across the entire image — if a particular geometric feature is present anywhere in the scene, it will be represented in $\mathbf{t}_{\text{spl}}$.

Why max-pooling over average-pooling or attention pooling: max-pooling is a hard selection mechanism — it chooses the strongest signal per dimension, which is appropriate for geometric features where the presence of a surface at a particular depth or orientation somewhere in the scene is more important than the average geometry. Average-pooling would dilute strong geometric signals (e.g., the exact depth of a small object of interest would be averaged with the background depth, losing precision). Attention pooling could theoretically learn to focus on task-relevant regions, but max-pooling is non-parametric and computationally free, and the paper's ablation results (Table 7) show that even this simple compression, combined with element-wise addition, outperforms more complex alternatives.

Step 2: Projecting to VLM Feature Space

The pooled spatial vector $\mathbf{t}_{\text{spl}}$ lives in the spatial token dimension $D_s$, which may differ from the VLM's action token dimension $D_{\text{act}}$. A lightweight MLP adapter $\mathcal{D}$ projects the spatial vector into the VLM's feature space:

$\widetilde{\mathbf{t}}_{\text{spl}} = \mathcal{D}(\mathbf{t}_{\text{spl}}), \quad \widetilde{\mathbf{t}}_{\text{spl}} \in \mathbb{R}^{D_{\text{act}}}$

where $\mathcal{D}$ is a small multi-layer perceptron (the exact architecture — number of layers, hidden dimension — is not specified in the paper, but its role as a "lightweight adapter" suggests 1–2 hidden layers).

What it computes: a learned linear/non-linear transformation that maps from the spatial foundation model's representation space into the VLM's representation space, aligning the two modalities so they can be meaningfully combined.

Why a separate adapter rather than training the ESM to output $D_{\text{act}}$-dimensional tokens directly: keeping the ESM's output dimension fixed to $D_s$ preserves its pre-trained spatial representations, which were optimised for 3D reconstruction. The adapter learns the alignment to the VLM's space as a separate, lightweight mapping that can be trained without modifying the ESM's internal representations (in Stage 1, the ESM is frozen). This is another instance of the paper's design philosophy: preserve pre-trained representations and introduce cross-modal alignment through small, dedicated modules.

Step 3: Fusing Spatial and Semantic Representations

The core architectural decision is how to combine the projected spatial vector $\widetilde{\mathbf{t}}_{\text{spl}}$ with the semantic action token $\hat{\mathbf{t}}_{\text{act}}$. The paper evaluates three fusion strategies (Section 3.4, Figure 3):

Option A: Cross-Attention Fusion. The action token $\hat{\mathbf{t}}_{\text{act}}$ serves as the query, while the projected spatial feature $\widetilde{\mathbf{t}}_{\text{spl}}$ provides the key and value inputs to a multi-head attention mechanism. This allows the action token to selectively attend to different aspects of the spatial representation based on task context — for example, attending more to depth features when the task is "place the cup on the table" (requiring precise height information) and more to positional features when the task is "move the block to the left" (requiring horizontal displacement). Multi-head attention enables the model to attend to multiple geometric aspects simultaneously (one head for depth, another for surface orientation, etc.).

Option B: FiLM-Gated Modulation. The spatial feature $\widetilde{\mathbf{t}}_{\text{spl}}$ is used to generate affine parameters $(\gamma, \beta)$ for feature-wise linear modulation of the action token:

$\text{FiLM}(\hat{\mathbf{t}}_{\text{act}}) = \gamma(\widetilde{\mathbf{t}}_{\text{spl}}) \odot \hat{\mathbf{t}}_{\text{act}} + \beta(\widetilde{\mathbf{t}}_{\text{spl}})$

where $\gamma(\cdot)$ and $\beta(\cdot)$ are learned functions (likely linear projections) that map the spatial vector to scale and shift parameters, and $\odot$ denotes element-wise multiplication. This is followed by a gating mechanism that learns to blend the modulated semantic features with the original spatial features, providing a learned trade-off between the two modalities.

Option C: Element-wise Addition. The simplest possible fusion — directly adding the two vectors:

$\mathbf{f}_{\text{fused}} = \hat{\mathbf{t}}_{\text{act}} + \widetilde{\mathbf{t}}_{\text{spl}}$

where $\mathbf{f}_{\text{fused}} \in \mathbb{R}^{D_{\text{act}}}$ is the fused representation that proceeds to the action predictor.

Ablation results (Table 7, Section 4.3 + Table 7, Appendix 10): On the CALVIN ABCD→D benchmark, element-wise addition achieves 4.08 average task length, outperforming both FiLM-Gated (4.04) and Cross-Attention (3.98). The gap is even larger in the zero-shot ABC→D setting: addition achieves 3.91, versus 3.76 for FiLM-Gated and 3.68 for Cross-Attention. Element-wise addition also achieves the highest task 5 completion rates in both settings (70.9% vs. 69.6% vs. 67.5% for ABCD→D; 62.2% vs. 58.4% vs. 57.2% for ABC→D).

Why element-wise addition outperforms more sophisticated mechanisms: the authors attribute this to training stability and computational efficiency. Cross-attention and FiLM-gated modulation introduce additional learnable parameters (the attention projections, the $\gamma$/$\beta$ networks) that must be optimised jointly with the VLM and adapter. In the low-data regime of robot manipulation (relative to internet-scale pre-training), these additional parameters may overfit or introduce optimisation challenges. Element-wise addition is parameter-free — it introduces no new learnable weights — which means it cannot overfit to the spatial-semantic alignment in the limited robot training data. The addition operation also preserves both signals equally, whereas attention might learn to ignore either the semantic or spatial stream in certain contexts.

More profoundly, element-wise addition aligns with the paper's design philosophy of preserving pre-trained representations. The VLM's action token $\hat{\mathbf{t}}_{\text{act}}$ was trained to predict actions from semantic understanding alone during pre-training (the Kosmos-VLA-2D pre-training phase, Appendix 7). Element-wise addition means the spatial vector acts as an additive correction to this pre-existing action signal — the pre-trained behaviour is preserved as a baseline, and spatial information adjusts it. Cross-attention, by contrast, could allow the spatial information to completely override the semantic signal, potentially causing the model to ignore language instructions in favour of geometric cues.

The final fused representation:

$\widetilde{\mathbf{t}}_{\text{spl}} = \mathcal{D}(\text{MaxPool}(\mathbf{T}_{\text{spl}})), \quad \mathbf{f}_{\text{fused}} = \hat{\mathbf{t}}_{\text{act}} + \widetilde{\mathbf{t}}_{\text{spl}}$

This single vector $\mathbf{f}_{\text{fused}}$ encodes both what the task requires (semantics from the VLM) and where things are in 3D space (geometry from the ESM), and is the input to the action predictor.

Step 4: Action Prediction

The fused feature vector $\mathbf{f}_{\text{fused}}$ is passed to an action predictor $\pi$ that outputs the action chunk $A_t$. The paper explores two architectures for the action predictor (Section 3.4):

MLP-based predictor (for single-step tasks). A multi-layer perceptron that directly maps the current fused feature to actions:

$A_t = \pi(\mathbf{f}_{\text{fused}}^t)$

This is used in the SimplerEnv and real-world experiments (Table 6), where the window size (history length) is 1 and the chunk size (action horizon $C$) is 5. The MLP predictor is stateless — each timestep's action prediction depends only on the current observation, with no memory of previous states.

LSTM-based predictor (for long-horizon tasks). A long short-term memory network that maintains a hidden state across timesteps and processes a sequence of fused features:

$A_t = \pi(\mathbf{f}_{\text{fused}}^{t-H+1}, \ldots, \mathbf{f}_{\text{fused}}^t)$

where $H$ is the history length (window size) — the number of previous timesteps provided as context. The LSTM processes this sequence and its final hidden state is fed through an MLP to produce the action chunk. This is used in the CALVIN experiments (Table 6), where the window size is 16 frames and the chunk size is 10 actions. The LSTM predictor is stateful — it can learn temporal dependencies, such as "the drawer was already opened three steps ago, so the next action should be to place the object inside."

Input configuration differences across benchmarks (Table 6):

Why LSTM for CALVIN and MLP for others: CALVIN involves long-horizon task sequences where the robot must complete up to 5 consecutive sub-tasks (e.g., "open drawer, pick pink block, place in drawer"). These require temporal reasoning — the robot must remember what it has already done and infer what remains. An LSTM with 16-frame history can learn these temporal dependencies. SimplerEnv and the real-world tasks involve single instructions executed over shorter horizons, where the current observation provides sufficient information and temporal context is less critical.

Chunk size reasoning: CALVIN uses a larger chunk size (10) because the tasks involve longer action sequences — each sub-task may require multiple seconds of continuous motion. Predicting 10 actions at once provides smoother trajectories and reduces the frequency of re-planning. SimplerEnv and real-world tasks use a smaller chunk (5) because the actions are shorter and more reactive.

Inference-time execution strategies (Section 8): For CALVIN and SimplerEnv, the model uses ensemble action execution: it generates the full action chunk, executes some fraction of it (the paper does not specify exactly how many actions are executed before re-planning), and then generates the next chunk. This is standard temporal ensembling to smooth out trajectory discontinuities at chunk boundaries. For real-world tasks, the entire action chunk is executed at once before the next observation is captured.

---

The VLM Backbone: Semantic Action Token Extraction

While the paper's contributions focus on spatial integration, the VLM backbone is a necessary prerequisite for understanding the full pipeline. The VLM is Kosmos-2 (Peng et al., 2023), a ~1.6B parameter vision-language model pre-trained on large-scale image-text data with grounding capabilities (it can link textual references to image regions). The VLM processes two types of input (Section 3.2):

• Visual observations $O_t$ — one or more images tokenized into a sequence of visual tokens.
• Language instruction $L$ — the task description tokenized into text tokens.

These are concatenated into a unified multi-modal sequence: [image tokens; text tokens]. A learnable action token $\mathbf{t}_{\text{act}}$ is appended to this sequence — this is a special token whose output hidden state will encode the action-relevant information after the VLM processes the entire multi-modal context. The VLM's transformer processes this sequence, and the output hidden state corresponding to the action token position is extracted:

$\hat{\mathbf{t}}_{\text{act}} \in \mathbb{R}^{D_{\text{act}}}$

where $D_{\text{act}}$ is the VLM's hidden dimension. This single vector $\hat{\mathbf{t}}_{\text{act}}$ serves as the semantic action representation — it encodes the task-oriented behaviour grounded in the multi-modal context. The information contained in this vector is purely semantic/2D: it captures what the task is, what objects are involved, and what the robot should do based on the image appearance and language meaning, but it has no explicit 3D geometric understanding.

Why a learnable action token rather than pooling all output tokens: the action token approach (common in VLA architectures like RT-2 and OpenVLA) provides a learnable query that the VLM's self-attention mechanism can use to aggregate task-relevant information from across the entire multi-modal sequence. The action token attends to relevant image regions, relevant text tokens, and their cross-modal relationships, distilling all this information into a single vector. Pooling over all output tokens would mix in irrelevant information (e.g., visual features of background regions, partial text tokens) that could degrade action prediction.

Pre-training of the VLM backbone: Before FALCON's post-training, the VLM is pre-trained as a 2D VLA (referred to as Kosmos-VLA-2D in Appendix 7) on the target robotics datasets (CALVIN, Open X-Embodiment). This pre-training uses learning rate $2 \times 10^{-5}$, global batch size 128, and a warmup ratio of 0.25 epochs (for CALVIN) or 2,500 steps (for OXE). The pre-training establishes a strong baseline capability for language-conditioned manipulation from 2D inputs alone, which FALCON then augments with spatial information during post-training.

---

Two-Stage Training Paradigm

FALCON's training follows a carefully designed two-stage post-training procedure after the initial VLA pre-training (Appendix 6.1). The design principle is to gradually introduce spatial information without disrupting pre-trained representations.

Notation for Training Stages

Let $\Theta_V$ denote the VLM parameters, $\Theta_A$ the action head parameters, $\Theta_G$ the ESM parameters (the spatial encoder), and $\Theta_D$ the lightweight adapter parameters (the MLP that projects spatial tokens to the VLM dimension). The overall objective is to minimise the action prediction loss $\mathcal{L}$ (defined below) over the robot manipulation dataset $\mathcal{S}$.

Loss Function

The action prediction loss consists of two components, reflecting the dual nature of the action space (continuous pose + discrete gripper):

$\mathcal{L} = \sum_{i=t}^{t+C-1} \left[ \text{MSE}(\hat{a}_{i,\text{pose}}, a_{i,\text{pose}}) + \lambda \cdot \text{BCE}(\hat{a}_{i,\text{gripper}}, a_{i,\text{gripper}}) \right]$

where the sum runs over the action horizon $C$ (actions from time $t$ to $t+C-1$), $\hat{a}_{i,\text{pose}} \in \mathbb{R}^6$ is the predicted 6-DoF gripper pose, $a_{i,\text{pose}}$ is the ground-truth pose, $\hat{a}_{i,\text{gripper}} \in [0, 1]$ is the predicted gripper open/close probability, $a_{i,\text{gripper}} \in \{0, 1\}$ is the ground-truth binary gripper state, $\text{MSE}$ is the mean squared error, $\text{BCE}$ is binary cross-entropy, and $\lambda$ is a weighting factor balancing the two loss terms (the paper does not specify the exact value of $\lambda$, but it is described as "ensuring stable and representative learning across heterogeneous action components").

What it computes: for each predicted action in the chunk, the loss penalises errors in the continuous pose prediction (via MSE) and the discrete gripper state (via BCE). The losses are summed over the entire action horizon, meaning the model is supervised to predict the full action sequence correctly, not just the next action.

Why MSE for pose and BCE for gripper: the pose is a continuous variable — Euclidean distances and angles — for which MSE is the standard regression loss (it corresponds to maximum likelihood estimation under a Gaussian error model). The gripper state is binary, making BCE the natural choice (maximum likelihood under a Bernoulli model). The weighting factor $\lambda$ is necessary because the MSE and BCE operate on different scales and the pose loss (6 dimensions) naturally dominates the gripper loss (1 dimension) without balancing.

Stage 1: Feature Space Alignment

In Stage 1, all pre-trained components are frozen: the VLM ($\Theta_V$), the ESM ($\Theta_G$), and the action head ($\Theta_A$). Only the lightweight adapter $\Theta_D$ is trained:

$\min_{\Theta_D} \mathbb{E}_{(O_t, L, \hat{A}_t) \sim \mathcal{S}} \left[ \mathcal{L}\left(\hat{A}_t, \pi\left(\mathcal{V}(O_t, L) + \mathcal{D}(\text{MaxPool}(\mathcal{G}(I_t^{\text{3rd}})))\right)\right) \right]$

What it computes: a gradient-based optimisation where only the adapter's weights are updated. The VLM and ESM produce fixed features (since their weights are frozen), and the adapter learns to project the ESM's spatial tokens into a form that, when added to the VLM's action token, improves action prediction. The expectation $\mathbb{E}_{(O_t, L, \hat{A}_t) \sim \mathcal{S}}$ means the loss is averaged over the robot dataset $\mathcal{S}$ of image-language-action tuples.

Critical design choice — zero-initialised final layer: the adapter's final linear layer is initialised to zero, meaning at the start of Stage 1, $\mathcal{D}(\text{MaxPool}(\mathcal{G}(I_t^{\text{3rd}}))) = \mathbf{0}$ and therefore $\mathbf{f}_{\text{fused}} = \hat{\mathbf{t}}_{\text{act}} + \mathbf{0} = \hat{\mathbf{t}}_{\text{act}}$. The model initially behaves identically to the pre-trained 2D VLA, producing exactly the same actions as without spatial information. Over the course of Stage 1 training, the adapter gradually learns to produce non-zero spatial features that improve action prediction when added to the semantic token.

Why Stage 1 and zero-initialisation: directly training all components jointly from the start would cause the spatial features — which are novel to the VLM — to produce large gradients that could perturb the VLM's carefully learned representations. The paper explicitly states that Stage 1 "prevents the spatial features from overwhelming the semantic representations during initial learning phases." Zero-initialisation ensures the spatial contribution starts at zero and grows gradually, providing a smooth learning trajectory rather than an initial shock to the VLM's feature space.

Hyperparameters for Stage 1 (Appendix 7): learning rate $1 \times 10^{-4}$ (higher than pre-training because only a small adapter is being trained, requiring faster convergence), global batch size 128 (CALVIN) or 512 (real-world), no warmup (unlike pre-training — warmup is unnecessary when training a small adapter from a zero initialisation), and the AdamW optimiser with constant learning rate schedule.

Stage 2: Joint Feature Refinement

In Stage 2, the VLM parameters $\Theta_V$ are unfrozen alongside the adapter $\Theta_D$, while the ESM ($\Theta_G$) and action head ($\Theta_A$) remain frozen:

$\min_{\Theta_V, \Theta_D} \mathbb{E}_{(O_t, L, \hat{A}_t) \sim \mathcal{S}} \left[ \mathcal{L}\left(\hat{A}_t, \pi\left(\mathcal{V}(O_t, L) + \mathcal{D}(\text{MaxPool}(\mathcal{G}(I_t^{\text{3rd}})))\right)\right) \right]$

What it computes: the same loss as Stage 1, but now both the VLM and the adapter are optimised. The VLM can adapt its internal representations to better integrate the spatial information that the adapter learned to provide in Stage 1, while the adapter can further refine its projection to work better with the VLM's updated representations.

Why unfreeze the VLM but not the ESM or action head: the ESM is kept frozen because its spatial representations are already high-quality from VGGT pre-training and Stage 1's adapter has already learned to align them with the VLM's space — further ESM training could overwrite these pre-trained 3D reconstruction capabilities. The action head is kept frozen to prevent the spatial-semantic fusion from changing the action decoder's behaviour too drastically, which could cause the model to produce physically implausible actions (e.g., gripper poses outside the workspace).

Why unfreeze the VLM at all, rather than keeping it frozen and only training the adapter: the adapter learns a fixed projection from spatial to semantic space, but the VLM's semantic action token $\hat{\mathbf{t}}_{\text{act}}$ was trained in a 2D-only context. Unfreezing the VLM allows it to adjust its feature extraction so that the semantic action token represents information that complements the spatial information rather than redundantly encoding the same scene. For example, in Stage 1, the VLM's action token might encode approximate depth from monocular cues (e.g., object size, shading), which overlaps with the ESM's spatial tokens. In Stage 2, the VLM can learn to suppress these redundant depth encodings and instead focus on purely semantic information (object identity, task phase), leaving depth encoding to the ESM stream.

Hyperparameters for Stage 2 (Table 6, Appendix 7):

The lower learning rate for Stage 2 (compared to Stage 1's $1 \times 10^{-4}$) reflects that the VLM is a large pre-trained model that should be fine-tuned carefully to avoid catastrophic forgetting. The larger batch size for real-world data (512 vs. 128) suggests the real-world dataset contains fewer but more diverse trajectories, requiring larger batches for stable gradient estimates.

Checkpoint selection protocol (Appendix 7): The paper notes that "policy performance does not fully correlate with offline evaluation metrics (e.g., validation loss) due to compounding errors in long-horizon rollouts, making checkpoint selection challenging." To ensure fair comparison across all baselines, all models are trained for a fixed number of epochs/iterations and the final checkpoint is evaluated (for CALVIN and Real-World) or the best-performing checkpoint from periodic evaluations is reported (for SimplerEnv, where evaluations are performed every 10K iterations). This protocol prevents cherry-picking of checkpoints and ensures that the comparison reflects the training procedure rather than checkpoint selection heuristics.

---

Ablation: Spatial Token Injection Position

The paper validates its core architectural claim — that spatial tokens should be injected at the action head, not the VLM — through a direct ablation (Appendix 10, Table 7). A variant called FALCON VLM-tokens injects spatial tokens from the ESM directly into the VLM's input sequence (concatenated with image and text tokens), following the approach of most prior 3D-based VLAs. All other components (ESM, action head, training data) remain identical.

Results:

The gap is most pronounced on the hardest tasks — in the zero-shot ABC→D setting, task 5 completion drops from 62.2% to 57.6%, a 4.6 percentage point decline. This confirms the paper's hypothesis that injecting 3D features into the VLM disrupts its pre-trained vision-language alignment, particularly harming the generalisation capability needed for zero-shot transfer to unseen environments.

Why VLM-tokens underperforms: the VLM was pre-trained on billions of image-text pairs where the visual stream consisted of 2D image tokens. Fine-tuning with an additional modality (spatial tokens) forces the VLM to adjust its internal representations to accommodate this new input. However, the robot manipulation datasets used for fine-tuning (CALVIN, OXE) are orders of magnitude smaller than the VLM's pre-training data. The VLM's pre-trained representations therefore get partially overwritten by the fine-tuning data, losing some of the generalisation capability that came from internet-scale pre-training. This is the embedding drift problem the paper identifies. In contrast, FALCON's action-head injection keeps the VLM's internal representations untouched — the VLM sees exactly the same inputs (image + text) as during pre-training, and only the action head learns to incorporate spatial information on top of the VLM's stable semantic representations.

---

Summary of Design Choices and Their Justifications

• VGGT as the spatial backbone over alternatives like DUSt3R or MASt3R because VGGT processes images holistically (multi-view, not pairwise), produces dense per-patch 3D tokens, and is architected as a transformer — making its token outputs naturally compatible with the VLM's token-based processing.
• Stochastic conditioning with Bernoulli sampling for modality transferability because it is the standard, theoretically grounded approach for building models robust to missing inputs — the model sees all modality combinations during training and learns to produce good spatial tokens regardless of which sensors are available at test time.
• Max-pooling spatial tokens rather than attention pooling because it is non-parametric (no additional parameters to overfit on limited robot data) and captures the strongest geometric signal per feature dimension, which is appropriate for action prediction where precise geometry matters more than average scene layout.
• Element-wise addition for fusion over cross-attention or FiLM-gated modulation because it is parameter-free (cannot overfit), preserves both the semantic and spatial signals equally, and aligns with the additive-correction design philosophy where the pre-trained 2D VLA's action prediction serves as a baseline that spatial information adjusts.
• Action-head injection over VLM injection because it preserves the VLM's pre-trained vision-language alignment, which the ablation in Table 7 shows is critical for zero-shot generalisation — the VLM-tokens variant loses 4.6 percentage points on the hardest task in the ABC→D split.
• Two-stage training with zero-initialisation because it prevents spatial features from causing disruptive gradient updates to the VLM's pre-trained representations during early training, while Stage 2's selective unfreezing allows the VLM to adapt its representations to complement (rather than compete with) the spatial stream.
• LSTM for long-horizon tasks and MLP for single-step tasks because CALVIN requires temporal reasoning over 16-frame histories to track task progress across 5 consecutive sub-tasks, while SimplerEnv and real-world tasks involve single instructions over shorter horizons where the current observation provides sufficient context.
• Differentiated learning rates in ESM training ($1 \times 10^{-6}$ for backbone, $1 \times 10^{-5}$ for reconstruction heads) because the backbone is pre-trained and needs careful fine-tuning to avoid catastrophic forgetting, while the task-specific heads are randomly initialised and need faster convergence.

4. Key Insights and Innovations

Innovation 1: Spatial Information Should Route to the Action Head, Not the Vision-Language Backbone

The dominant assumption in 3D-enhanced VLA research—from PointVLA to SpatialVLA to 3D-VLA—has been that 3D information belongs in the VLM's input stream, concatenated with image and text tokens so the model can reason jointly over all modalities. This seems intuitive: the VLM is where reasoning happens, so it should have access to all available information. FALCON argues this intuition is wrong, and the consequences of being wrong are significant.

The paper's key conceptual move is to decouple spatial reasoning from semantic reasoning architecturally, inspired by the brain's cerebrum-cerebellum division. The VLM (cerebrum) handles high-level semantic reasoning—what objects are present, what the instruction means, what sequence of sub-goals is needed. The action head (cerebellum) handles fine-grained sensorimotor integration—exactly how far to move, at what angle to grasp, how high to lift. In between sits the Embodied Spatial Model, which transforms raw pixels into geometric tokens, and a lightweight adapter, which aligns those tokens with the VLM's feature space. Crucially, the spatial stream and the semantic stream converge only at the final action-prediction step, never inside the VLM.

This is not an incremental architectural tweak. It is a fundamental rejection of the monolithic VLM paradigm that has dominated VLA design. The paper provides direct causal evidence that the monolithic approach is harmful: the FALCON VLM-tokens ablation (Table 7), where spatial tokens are injected into the VLM following prior practice, drops task-5 completion in the zero-shot CALVIN ABC→D split by 4.6 percentage points (from 62.2% to 57.6%) compared to action-head injection. The same spatial tokens, the same training data, the same model—only the injection point differs. This is a clean experiment demonstrating that the performance cost is not about spatial token quality but about disruption of the VLM's pre-trained vision-language alignment, a phenomenon the paper terms embedding drift.

The significance goes beyond this paper's results. If the finding generalises—that VLMs are fragile vessels whose cross-modal alignments degrade when third modalities are forced into them—then all prior 3D-VLA work that injects geometry into the VLM stream faces a fundamental tension: the more spatial information you add, the more you damage the semantic capabilities that made the VLM valuable in the first place. FALCON's solution—route geometry to the motor-control stage where it belongs—resolves this tension entirely and establishes a new design principle: the VLM should never see spatial tokens. This is a reframing of the VLA architecture problem, not just a performance improvement.

Innovation 2: Modality Transferability as a Training Objective, Not an Architectural Afterthought

Before FALCON, 3D VLA methods were divided into two camps with opposite failure modes. Methods requiring explicit 3D sensors (PointVLA, GeoVLA, 3D Diffusion Policy) achieved strong geometric awareness but collapsed entirely when those sensors were absent or produced noisy data—a property the paper calls low modality transferability. Methods using learned or pseudo-3D cues from RGB alone (SpatialVLA, Evo-0) worked without specialised hardware but could not exploit higher-quality 3D inputs when they were available, leaving performance on the table. Neither camp treated modality transferability as a first-class design objective; it was an accidental property of their architectural choices.

FALCON reframes modality transferability as something you train for explicitly through stochastic conditioning. The Embodied Spatial Model's training procedure (Equation 4) randomly decides, per sample, whether to inject depth tokens ($b_{\text{d}} \sim \text{Bernoulli}(p)$), ground-truth camera pose ($b_{\text{p}} \sim \text{Bernoulli}(p)$), both, or neither. Over many training iterations, the same model learns to extract 3D structure from RGB alone (the modal case, since most robot datasets lack 3D annotations), while also learning to exploit additional geometric inputs when they are present. At inference time, the model works with whatever sensors the robot has, and its spatial representations improve monotonically with more modalities—Table 5 shows depth prediction δ < 1.25 jumping from 90.91% (RGB-only) to 99.79% (RGB-D), and the real-world experiments (Figure 7) show task success rates rising from 60% to 80% when depth and pose are added for height-varying scenarios.

This is more than a training trick. It represents a shift from hardware-defined to data-defined robot policies. Prior 3D VLA methods tie the policy to specific sensor configurations—change the camera or lose the depth channel, and the policy breaks. FALCON decouples the policy from the sensor suite: the same trained model can be deployed on a cheap RGB-only robot for routine tasks and an expensive RGB-D robot for precision tasks, without retraining, without architectural modification, and with performance scaling naturally with sensor quality. For the economics of robot learning—where training a VLA on large-scale datasets costs thousands of GPU-hours and must amortise across diverse deployment scenarios—this is a critical practical advance. It means a single trained policy can serve an entire fleet of heterogeneous robots with different sensor configurations, rather than needing per-hardware variants.

The key intellectual move is recognising that modality transferability is a robustness property that should be designed into the training procedure, not an architectural claim about representation learning. Prior work that injected 3D features into the VLM implicitly assumed the model would learn cross-modal representations that support any input modality; FALCON shows this assumption fails in practice (the VLM-tokens ablation demonstrates that adding modalities to the VLM stream degrades generalisation) and replaces it with explicit stochastic training that forces the spatial encoder to handle all modality subsets.

Innovation 3: Verifier-Free 3D Grounding via Spatial Foundation Models

A less explicit but equally important conceptual contribution is FALCON's demonstration that spatial foundation models can provide strong geometric priors for robot manipulation without task-specific training or verifier signals. This distinguishes FALCON from two common alternatives in the VLA literature.

First, many approaches to improving spatial reasoning rely on task-specific reward signals or verifiers—for instance, training a separate depth estimation network that is supervised on robot-collected depth data, or using reinforcement learning with shaped rewards that encode geometric desiderata (e.g., "grasp the object from above"). These approaches require substantial in-domain data collection: the depth estimator must be trained on images from the specific robot and environment, and reward functions must be hand-designed for each task family. The paper's Embodied Spatial Model, by contrast, is pre-trained on general-purpose 3D reconstruction data (VGGT's training set of internet images with Structure-from-Motion geometry) and transfers zero-shot to robot manipulation scenes. The CALVIN depth estimation results in Table 5 confirm this: with RGB only, the ESM achieves 90.91% δ < 1.25 on robot manipulation images from a domain it was never explicitly trained on, comparable to VGGT's 91.33%.

Second, prior work on "weak 3D cues" (pseudo-depth from ZoeDepth, learnable spatial embeddings in SpatialVLA) provided approximate geometric signals that lacked precision—relative depth ordering rather than metric distances, making them insufficient for tasks requiring exact height or size reasoning. The paper's spatial understanding experiments (Section 4.2, Figure 6) demonstrate the consequences concretely: RoboVLM, which relies on the VLM's implicit 2D spatial understanding, either collides with larger-than-trained blocks or releases smaller blocks prematurely. FALCON's use of dense per-patch spatial tokens from a foundation model, encoding metric depth and 3D point coordinates, provides the precision needed for size- and height-adaptive manipulation.

The intellectual contribution here is establishing that spatial foundation models are a drop-in source of geometric priors for VLAs, analogous to how language foundation models provide semantic priors. Just as VLAs inherit language understanding from pre-trained LLMs without needing to re-learn syntax and semantics from robot data, FALCON shows they can inherit 3D understanding from pre-trained spatial models without needing to re-learn geometry from manipulation data. This is a transfer learning insight with broad implications: as spatial foundation models improve (and the field is moving fast—DUSt3R, MASt3R, VGGT, CUT3R represent rapid progress), VLA spatial reasoning improves automatically, without retraining the policy. FALCON's modular architecture makes this straightforward: swap in a better spatial backbone, keep the VLM and action head frozen, and the policy immediately benefits from better geometry.

Innovation 4: The Training-Inference Modality Asymmetry as a Design Principle

FALCON introduces a design pattern that has no direct precedent in the VLA literature: train with all available modalities using stochastic conditioning, but deploy with whatever subset is available, and the model works across all of them. This is not simply "train with dropout" or "data augmentation." It is an architectural commitment to asymmetry between training and inference modalities as a core robustness property.

The significance becomes clear when contrasted with the standard approach in robot learning. Most VLA methods are trained with a fixed sensor configuration—RGB images from two cameras, point clouds from a depth sensor, proprioceptive state from joint encoders—and deployed with exactly that configuration. If a sensor fails or is absent at deployment, the policy produces undefined behaviour because it has never seen input with missing modalities. Standard solutions to this problem (sensor fusion with Kalman filtering, explicit redundancy) add complexity at the hardware and software level.

FALCON's stochastic conditioning approach solves this at the representation learning level: the spatial encoder learns to produce geometrically meaningful tokens from any subset of available inputs because it has been trained on all subsets. The critical design detail is how modalities are injected: depth is added element-wise to visual tokens (providing complementary geometric signal) while pose replaces the learned camera token (providing a strict information gain). This asymmetric treatment—addition for complementary signals, replacement for redundant ones—reflects a nuanced understanding of what each modality contributes. Depth adds information the image does not contain (metric scale); pose provides the same information the learned camera token would estimate, but with ground-truth accuracy.

The practical implication is substantial for real-world deployment. A robot fleet with heterogeneous sensors—some with RGB-D cameras, some with RGB-only, some with calibrated poses, some without—can all run the same FALCON policy. Organisations can deploy cheaper hardware for routine tasks and premium sensors for precision tasks, with the policy degrading gracefully rather than failing catastrophically when running on lower-spec hardware. This is the kind of robustness property that separates research demonstrations from production-ready systems, and it emerges from a training procedure that treats modality variability as a feature to be learned rather than a problem to be engineered around.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three simulation benchmarks and 11 real-world manipulation tasks. CALVIN (Mees et al., 2022) is a long-horizon language-conditioned manipulation benchmark with four scene splits (A, B, C, D), each containing 24k human-teleoperated demonstrations across 34 basic skills. Evaluation uses the standard ABC→D and ABCD→D splits, where letters denote training environments and D is the held-out test environment; each rollout requires completing 5 consecutive sub-tasks, with 1000 rollouts evaluated on the D split. SimplerEnv (Li et al., 2024c) provides simulated replicas of real-robot evaluation setups for policies trained on the Bridge V2 and Google Robot datasets, testing transfer to variations in lighting, textures, colors, and viewpoints. The real-world benchmark comprises 1,030 expert teleoperated trajectories across five scenarios and 11 tasks, ranging from simple object interactions (e.g., "lift the yellow pepper") to long-horizon sequential activities (e.g., "place the red coke can on the bottom shelf"), designed to test language grounding (cluttered scenes with distractors), semantic understanding (unseen object poses), spatial reasoning (varying object sizes and target heights), and few-shot adaptation to unseen objects, backgrounds, and task descriptions. Detailed task descriptions and trial counts for each benchmark are provided in Appendix 9.

• Base model(s). FALCON is built on a Kosmos-2 (Peng et al., 2023) VLM backbone with approximately 1.6B parameters, chosen for its grounding capabilities—the ability to link textual references to image regions—which is valuable for language-conditioned manipulation. The Embodied Spatial Model is based on VGGT (Wang et al., 2025a), a 1.0B-parameter spatial foundation model pre-trained on internet-scale 3D reconstruction data, chosen because it processes images holistically (multi-view rather than pairwise) and produces dense per-patch spatial tokens compatible with the VLM's token-based architecture. The full FALCON model totalling 2.9B parameters. The authors argue Kosmos-2 is representative of open-source VLMs suitable for VLA fine-tuning and that its scale (1.6B) sits in a regime where spatial augmentation can demonstrably improve performance without being bottlenecked by base model capability.

• Metrics. For CALVIN, the primary metrics are success rates for completing 1 through 5 consecutive sub-tasks (reported as percentages) and average task length (Avg. Len.) —the mean number of sub-tasks successfully completed per rollout sequence, computed over 1000 rollouts. For SimplerEnv, the metric is per-task success rate (percentage of trials where the robot successfully executes the instructed task) and average success rate across tasks. For real-world experiments, the metric is task success rate—the fraction of trials where the robot completes the specified task. Sub-task success rates (e.g., separately measuring whether the robot successfully grasps an object vs. completes the full task) are reported for the few-shot adaptation experiments (Tables 9–13) to provide granular diagnostics. All success rates are binary: a rollout either achieves the task specification or does not.

• Baselines. The paper compares against a comprehensive set of generalist robot policies spanning multiple architectures and training paradigms:

  • RT-1 (Brohan et al., 2022) and RT-1-X (O'Neill et al., 2024): Transformer-based policies trained on the Open X-Embodiment dataset, representing early large-scale multi-robot imitation learning.
  • RT-2 and RT-2-X (Brohan et al., 2023): VLAs built on large VLMs (PaLI-X, PaLM-E) with actions discretised as language tokens, representing the state of the art in large-scale closed-source VLA models. RT-2-X has 55B parameters, substantially larger than FALCON.
  • OpenVLA (Kim et al., 2024): An open-source 7B-parameter VLA built on a Prismatic VLM and fine-tuned on OXE, representing the leading open-source VLA baseline. Also evaluated with OpenVLA-OFT (Kim et al., 2025), a parameter-efficient fine-tuning variant using LoRA with rank 32.
  • Octo-Base (Octo Model Team et al., 2024): A transformer-based diffusion policy pre-trained on OXE using a generalist architecture designed for multi-robot transfer.
  • RoboVLM (Li et al., 2024a): A VLA that systematically studies design choices in building vision-language-action models, representing a strong 2D-only VLA baseline with careful training methodology.
  • SpatialVLA (Qu et al., 2025): A VLA that injects learnable spatial embeddings into the VLM's input space, representing the most direct architectural comparison for FALCON's spatial injection approach—SpatialVLA is the "weak 3D cues" paradigm that FALCON argues against.
  • Robo-Flamingo (Li et al., 2024b) and GR-1 (Wu et al., 2024): VLAs based on Flamingo-style architectures with video pre-training.
  • 3DDP (Ze et al., 2024) and 3D Diffuser Actor (Ke et al., 2024): Diffusion policies that consume explicit 3D representations (point clouds), representing the "explicit 3D input" paradigm.
  • MCIL (Lynch and Sermanet, 2020): An earlier language-conditioned imitation learning baseline.
  • Seer-Large (Tian et al., 2024): A scalable inverse dynamics model for manipulation.
  • TraceVLA (Zheng et al., 2024): A VLA that incorporates visual trace prompting for spatial-temporal awareness.

• Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of LLM sampling—this is a deterministic policy, not a search or sampling method. Computation is measured by model size (parameter count) and inference throughput. FALCON requires 12.8 GB of GPU memory and runs at approximately 57 Hz on a single NVIDIA RTX 4090 GPU during real-world evaluation. Training compute is reported as 32 A100 GPUs for the full FALCON post-training (across both stages) and 16 A100 GPUs for approximately 2 days for the ESM pre-training. Baseline comparisons use the same training budget (same number of epochs or iterations, same batch sizes) to ensure fairness—all VLAs are trained for a fixed number of epochs or iterations rather than using validation-based early stopping, because the paper notes that "policy performance does not fully correlate with offline evaluation metrics due to compounding errors in long-horizon rollouts" (Appendix 7).

• Cross-validation / statistical protocol. For CALVIN, 1000 rollouts with 5 consecutive sub-tasks each are evaluated on the D split, providing a large sample size that yields stable estimates. For SimplerEnv, each task is evaluated with 24–75 trials depending on the task (Appendix 9 details: 24 trials for WidowX tasks with varying object positions and orientations; 75 trials for Google Robot "pick coke can" with three can orientations at 25 grid positions each; 60 trials for "move near" with five object triplets and two triangular patterns at six configurations each; 54 trials for drawer open/close with three drawers, two actions, and nine robot positions; 27 trials for the multi-step drawer-and-apple task with three robot positions and nine apple positions). For real-world experiments, Base Tasks are evaluated with 10 different scene layouts at 10 trials each (90 total rollouts across 9 task suites). Few-shot Adaptation tasks are evaluated with 5 different layouts at 2 trials each. Spatial Understanding evaluations use 5 layouts at 2 trials each. The checkpoint selection protocol is standardised across all baselines to prevent cherry-picking: for CALVIN, the final checkpoint at 5 epochs is evaluated; for SimplerEnv, checkpoints at 10K-iteration intervals are evaluated and the best is reported; for real-world, only the final checkpoint at 30 epochs (or equivalent iterations for OpenVLA-OFT) is evaluated. This protocol ensures that FALCON's advantages are not artifacts of more favourable checkpoint selection.

Main Quantitative Results

CALVIN Long-Horizon Manipulation

Table 1 presents the CALVIN benchmark results across two evaluation splits. FALCON achieves state-of-the-art performance in both settings.

In the ABCD→D setting (trained on all four environments, tested on D), FALCON achieves a 4.53 average task length, surpassing the previous best method RoboVLM (4.49) and substantially outperforming prior work. The per-task success rates show FALCON completing 97.2% of task-1, 93.3% of task-2, 90.3% of task-3, 88.0% of task-4, and 84.0% of task-5 —meaning the model successfully executes all 5 consecutive sub-tasks in the majority of 1000-rollout sequences. Compared to the baseline that FALCON extends (the Kosmos-2-based VLA without spatial tokens, which is not tabulated separately but is implicitly the pre-training baseline referenced in Appendix 7), the improvements are attributable to the spatial token injection, but the paper does not isolate this baseline number in Table 1. The gap between FALCON and RoboVLM is 0.04 Avg. Len., which is modest but represents the upper end of a highly saturated benchmark where multiple methods cluster between 4.08 and 4.49.

In the zero-shot ABC→D setting (trained on environments A, B, C only, tested on D), FALCON achieves a 4.40 average task length, significantly outperforming the previous best of 4.28 from Seer-Large. This is the more challenging evaluation because it tests generalisation to an entirely unseen environment configuration. The per-task success rates are 98.4% (task 1), 94.5% (task 2), 88.6% (task 3), 82.5% (task 4), and 75.5% (task 5). FALCON surpasses methods that rely on ground-truth point clouds—3DDP achieves only 0.27 Avg. Len. and 3D Diffuser Actor achieves 3.35—demonstrating that strong spatial priors from RGB alone, properly integrated, can outperform explicit 3D sensors. FALCON improves over 3D Diffuser Actor by 1.05 Avg. Len. and over the point-cloud-based 3DDP by 4.13 Avg. Len. The comparison with SpatialVLA is not available in the CALVIN table (SpatialVLA was not evaluated on CALVIN), but RoboVLM—a strong 2D VLA baseline—achieves 4.25 Avg. Len., making the improvement 0.15 Avg. Len.

A subtle but important observation: FALCON's advantage over prior methods grows as the task horizon extends. The gap in task-1 success is small (98.4% vs. 98.0% for RoboVLM), but grows to 5.1 percentage points by task 5 (75.5% vs. 70.4%). This pattern—larger gains on longer sequences—is consistent with the hypothesis that spatial understanding helps prevent compounding errors in long-horizon execution, where small geometric misalignments in early sub-tasks cascade into failures in later sub-tasks.

SimplerEnv Evaluations

Table 2 reports results on the WidowX Robot setup (Bridge V2 environment). FALCON achieves an average success rate of 56.3%, substantially outperforming all baselines:

• Put Spoon on Towel: 62.5% vs. RoboVLM (45.8%) and SpatialVLA (16.7%). This task requires precise positioning—the spoon must be carried to a specific corner of a 15 cm square—and FALCON improves by 16.7 percentage points over the next best method.
• Put Carrot on Plate: 41.7% vs. SpatialVLA (25.0%) and RoboVLM (20.8%).
• Stack Green Block on Yellow Block: 20.8% vs. SpatialVLA (29.2%)—this is the one task where FALCON underperforms SpatialVLA, suggesting that explicit learnable spatial embeddings may have an advantage for the precise vertical alignment needed in block stacking, or that FALCON's ESM depth estimates from RGB alone are less reliable for small (3 cm) objects.
• Put Eggplant in Yellow Basket: 100.0%—matching SpatialVLA's 100.0%, both substantially outperforming RoboVLM (79.2%) and Octo-Base (43.1%).

The overall improvement over SpatialVLA is 13.6 percentage points on average (56.3% vs. 42.7%), though this is largely driven by the Put Spoon task gap—on the other three tasks, FALCON and SpatialVLA are either comparable or SpatialVLA leads slightly. This suggests FALCON's spatial tokens provide advantages for tasks requiring metric distance reasoning (placing an object at a specific location offset from the start) more than for tasks requiring fine relative alignment (stacking small objects).

Table 3 reports results on the Google Robot setup. FALCON achieves an average success rate of 62.9%, outperforming all baselines:

• Pick Coke Can: 90.7% vs. SpatialVLA (86.0%) and RT-2-X (78.7%).
• Move Near: 79.2% vs. SpatialVLA (77.9%) and RT-2-X (77.9%).
• Open/Close Drawer: 39.8% vs. RT-1-X (59.7%) and SpatialVLA (57.4%). This is a notable underperformance—FALCON is approximately 17–20 percentage points below the best baselines on articulated object manipulation. The paper does not discuss this discrepancy, but it may reflect FALCON's spatial tokens being optimised for free-space object manipulation (where depth to surfaces matters) rather than for mechanisms like drawer handles (where the relevant geometry is thin structures and contact points). Alternatively, the side-camera-only input for FALCON vs. potentially multi-view inputs for some baselines may limit handle perception.
• Open Top Drawer and Place Apple: 41.7%—the standout result where most baselines score near zero. RT-2-X (55B parameters) achieves only 3.7%, OpenVLA 0.0%, SpatialVLA 0.0%, and RoboVLM 24.1%. FALCON's 41.7% represents a 17.6 percentage point improvement over the next best method (RoboVLM). This task requires multi-step reasoning (open drawer, then place apple inside) with precise spatial understanding of the drawer's depth and the apple's target location relative to the drawer interior, and FALCON's advantage strongly supports the claim that geometric priors matter for compound manipulation tasks.

The Google Robot results reveal an interesting pattern: FALCON excels on tasks requiring precise object placement relative to 3D structures (drawer-and-apple: 41.7% vs. 0–24.1% for baselines) and fine-grained object grasping with geometric variations (pick coke can: 90.7%), but is weaker on articulated mechanism operation (open/close drawer: 39.8% vs. 57.4–59.7%). This suggests the ESM's spatial tokens are particularly informative about surface geometry and object location, but may not capture the kinematic structure of articulated objects as effectively.

Real-World Base Tasks

Figure 4 presents the real-world Base Tasks results across nine task suites spanning three scenarios (Dining Table, Bedroom, Kitchen), testing language grounding (cluttered scenes with random distractors) and semantic understanding (unseen object poses). FALCON achieves an average success rate of 70.0% across all 90 rollouts (9 tasks × 10 layouts × 10 trials), compared to:

• SpatialVLA: 44.4%
• RoboVLM: 61.1% (estimated from the bar chart in Figure 4)
• OpenVLA-OFT: 14.4% (estimated from the bar chart)

The 25.6 percentage point improvement over SpatialVLA—the most directly comparable method since both augment 2D VLAs with spatial information—represents the largest single margin across all experimental settings. The paper highlights a specific diagnostic: in the task "pick banana and place on red plate," baseline methods (RoboVLM, OpenVLA-OFT) frequently place the banana on a yellow distractor plate rather than the instructed red plate, while FALCON consistently selects the correct target. This demonstrates that the spatial information is not merely improving grasp precision but also enhancing instruction following—the model better distinguishes between visually similar objects at different spatial locations, a capability that requires integrating language semantics with geometric scene layout.

The Base Tasks results also show that FALCON's advantage is not uniform across tasks—some task suites show larger improvements than others—but the paper does not provide per-task breakdowns in the main text (only aggregated bar charts in Figure 4). The qualitative rollout images in Appendix 14.1 (Figure 12) provide examples but do not include numerical per-task success rates.

Real-World Few-Shot Adaptation

Figure 5 presents the few-shot adaptation results with only 20 demonstrations per task (80 total trajectories across 4 tasks). FALCON achieves the highest performance across all settings:

• Simple setting (standard objects and backgrounds): FALCON achieves 87.5% average success rate, compared to RoboVLM (60.0%), SpatialVLA (50.0%), and OpenVLA-OFT (27.5%)—a 27.5 percentage point improvement over the next best method. The task-level breakdown in Table 9 reveals that FALCON achieves 100% on "open drawer and place bread" and 90% on "place the green Sprite can on the top shelf," with strong sub-task performance (100% drawer opening success, 90% Sprite can grasping success).
• Unseen Average (aggregating Unseen Object, Unseen Background 1, Unseen Background 2, and Unseen Task Description settings): FALCON achieves an average that is 27% higher than the second-best model (Figure 5, right bars). The detailed tables (Tables 10–13) show per-setting results:
  • Unseen Object (Table 10): FALCON 60.0% vs. SpatialVLA 27.5% and RoboVLM 15.0%. On "open drawer and place tennis ball" (unseen object), FALCON achieves 80% while all baselines score 0–20%.
  • Unseen Background 1 (Table 11): FALCON 40.0% vs. SpatialVLA 25.0%.
  • Unseen Background 2 (Table 12): FALCON 50.0% vs. SpatialVLA 32.5%.
  • Unseen Task Description (Table 13): FALCON 77.5% vs. RoboVLM 55.0% and SpatialVLA 45.0%.

The most striking detail is the Unseen Object setting for "open drawer and place bread" (Table 10, Place Tennis Ball column): FALCON achieves 80% success (100% drawer opening, 80% overall), while RoboVLM and OpenVLA-OFT achieve 0%, and SpatialVLA achieves 20%. The paper's Figure 1 highlights this task specifically, noting that "FALCON achieves an impressive success rate of 80%, while other models demonstrate near-zero success." This is a clean demonstration that spatial understanding—the ability to perceive the tennis ball's 3D location relative to the drawer despite never having seen that object during training—is the differentiating factor. A 2D VLM might recognise the tennis ball semantically but fail to localise it precisely in 3D space for grasping; FALCON's ESM provides the geometric grounding that enables reliable manipulation of novel objects.

Real-World Spatial Understanding Capability Evaluations

Figure 6 evaluates four tasks designed to probe spatial perception:

• Stack blue block on red block (larger: 5 cm) and Stack blue block on red block (smaller: 3 cm) test adaptation to object scale variation (training blocks are 4 cm). The paper reports that "baseline methods such as RoboVLM often struggle with objects of varying sizes. For larger blocks, collisions frequently occur during the placement of the blue block, while smaller blocks are prematurely released before placement, leading to task failure." FALCON exhibits "strong robustness to scale variations, avoiding these issues and achieving the highest success rates in both scenarios." Exact success rates are not provided in the main text for Figure 6 tasks—only bar chart visualisations—but the qualitative description indicates FALCON substantially outperforms baselines on both size variants.

• Put white cup on pink cloth (cup height change) tests adaptation to target height variation (the cup is elevated by two 3 cm blocks placed underneath, changing the placement height from training). Figure 7 reports this task separately as a modality transferability evaluation: FALCON with RGB-only achieves approximately 60% success, while adding depth and camera pose raises success to approximately 80% (detailed in the modality transferability subsection below). This confirms that metric depth information is particularly valuable for height-sensitive tasks where visual appearance alone is ambiguous about absolute elevation.

• Two spatial-prompt tasks (adapted via efficient fine-tuning with 50 demonstrations each) test the ability to follow language instructions that require spatial reasoning (e.g., "place the object on the left/front shelf"). The exact tasks are not enumerated in the main text, but Figure 6 shows FALCON outperforming baselines on both.

The spatial understanding evaluations demonstrate that FALCON's geometric priors address the two specific failure modes identified in the paper's motivation: adaptation to object scale variation (the ESM provides metric size information that the VLM lacks) and adaptation to height variation (the ESM provides absolute depth/distance that monocular 2D estimation cannot reliably produce).

Modality Transferability

CALVIN ablation (Table 4). The paper compares FALCON with different input modalities against Kosmos-VLA—a variant where the ESM is replaced by a lightweight point cloud encoder while retaining other components:

• ABCD→D setting: FALCON with RGB-only achieves 4.08 Avg. Len., comparable to Kosmos-VLA with RGB-D at 4.05. FALCON with RGB-D achieves 4.09. The improvement from adding depth is marginal in this setting (4.08 to 4.09), suggesting that CALVIN's ABCD→D tasks are already well-served by RGB-only spatial priors.
• ABC→D setting (zero-shot): Here the pattern differs. FALCON with RGB-only achieves 3.91 Avg. Len., while Kosmos-VLA with RGB-D achieves 3.98. FALCON with RGB-D achieves 3.97—essentially matching the RGB-D baseline. This suggests that in zero-shot generalisation, explicit depth provides a benefit that FALCON's RGB-only spatial tokens partially but not fully recover. However, the gap is small (3.91 vs. 3.98, or 0.07 Avg. Len.) and may not be statistically significant given the 1000-rollout evaluation.

Real-world modality experiments (Figure 7). Two tasks are evaluated:

• Lift yellow pepper (left panel): RGB-only achieves approximately 65–70% success; adding depth and camera pose achieves approximately 75–80%. The improvement is modest (~10 percentage points), consistent with the CALVIN finding that easy single-step tasks benefit marginally from additional modalities.
• Put white cup on pink cloth with cup height change (right panel): RGB-only achieves approximately 60%; adding depth and camera pose achieves approximately 80%. This 20 percentage point gap is substantial and demonstrates that the additional modalities matter most when the task requires precise absolute height reasoning—exactly the type of geometric reasoning that monocular depth estimation struggles with (metric scale is inherently ambiguous from a single image without known camera parameters or reference objects).

ESM depth estimation accuracy (Table 5). The depth prediction metrics on CALVIN provide the mechanistic explanation for these policy-level findings:

• RGB-only: 90.91% δ < 1.25, 8.61 Abs. Rel (comparable to VGGT baseline of 91.33% / 8.53)
• RGB + depth: 99.79% δ < 1.25, 0.91 Abs. Rel (near-perfect depth estimation when depth sensor input is provided)
• RGB + depth + pose: 99.47% δ < 1.25, 0.87 Abs. Rel (marginal additional improvement from pose)

The 8.61→0.91 reduction in absolute relative error when depth is provided is a 9.5× improvement in depth accuracy. The fact that policy-level success rates do not improve by a similar factor (e.g., Figure 7 shows ~60%→80%, not ~60%→near-100%) indicates that depth estimation accuracy is not the sole bottleneck for task success—other factors (grasp stability, action prediction accuracy, language grounding) impose ceilings that cannot be overcome by better geometry alone.

Ablation Studies and Robustness Checks

• Spatial token injection position (Table 7): Injecting spatial tokens into the VLM (FALCON VLM-tokens) versus the action head (standard FALCON). In the ABCD→D setting, VLM-tokens achieves 4.00 Avg. Len. vs. 4.08 for action-head injection—a 0.08 drop. In the zero-shot ABC→D setting, the gap widens to 3.79 vs. 3.91, with task-5 success dropping from 62.2% to 57.6%. This ablation is the strongest direct evidence for the paper's central architectural claim that spatial information disrupts the VLM's pre-trained representations, and the larger degradation in the zero-shot setting specifically supports the hypothesis that the disruption primarily harms generalisation rather than in-distribution performance.

• Modality fusion strategy (Table 7): Three fusion methods are compared: cross-attention, FiLM-gated modulation, and element-wise addition (the FALCON default). Element-wise addition achieves the best performance in all settings: 4.08 Avg. Len. (ABCD→D) vs. 4.04 for FiLM-Gated and 3.98 for Cross-Attention; 3.91 Avg. Len. (ABC→D) vs. 3.76 and 3.68 respectively. The performance ordering (addition > FiLM-gated > cross-attention) is consistent with the hypothesis that parameter-free fusion prevents overfitting on limited robot manipulation data—cross-attention introduces the most learnable parameters and performs worst, while element-wise addition introduces none and performs best. The gap is larger in the zero-shot setting (0.23 Avg. Len. between addition and cross-attention) than in ABCD→D (0.10), again suggesting that parameter-heavy fusion disproportionately harms generalisation.

• ESM modality inputs (Table 5): Ablating the ESM's depth and camera pose inputs shows that RGB-only achieves 90.91% δ < 1.25 and 8.61 Abs. Rel, essentially matching the VGGT baseline (91.33% / 8.53). Adding depth (without pose) dramatically improves accuracy to 99.79% δ < 1.25 and 0.91 Abs. Rel. Adding both depth and pose yields 99.47% δ < 1.25 and 0.87 Abs. Rel—a marginal further improvement. The key finding is that depth provides the vast majority of the geometric signal, while camera pose contributes modestly. This aligns with the policy-level results (Figure 7) where the large gap is between RGB-only and RGB-D, not between RGB-D and RGB-D-pose.

• Wrist camera integration into ESM (Table 8): Adding wrist camera images to the ESM input further improves CALVIN performance: Avg. Len. increases from 4.08 to 4.10 in ABCD→D. The improvement is modest (0.02 Avg. Len.) but consistent, suggesting that multi-view geometry provides complementary cues beyond what a single third-view image can capture. The paper flags this as a direction for future work (Appendix 11).

• Checkpoint selection robustness (Appendix 7): The paper addresses potential concerns about checkpoint selection bias by standardising the protocol: all models are trained for a fixed number of epochs or iterations, and either the final checkpoint (CALVIN, real-world) or the best periodic checkpoint (SimplerEnv) is evaluated. This protocol applies identically to FALCON and all baselines, eliminating the possibility that FALCON's advantages arise from more favourable checkpoint selection. The explicit acknowledgment that validation loss correlates poorly with policy performance ("due to compounding errors in long-horizon rollouts") supports the decision to standardise training duration rather than using validation-based early stopping, which could introduce bias.

• Training data scale sensitivity (Few-shot adaptation, Tables 9–13): While not presented as a formal ablation, the few-shot experiments (20 demonstrations per task, 80 total) demonstrate that FALCON's spatial priors provide substantial benefits even at very low data scales. FALCON with 80 trajectories outperforms baselines trained on the same data by 27.5 percentage points (Simple setting, 87.5% vs. 60.0%), suggesting that the spatial priors from VGGT pre-training are sample-efficient—they provide useful geometric information even when task-specific manipulation data is extremely limited.

• Negative result: FALCON underperforms on articulated objects (Table 3, Open/Close Drawer): FALCON achieves 39.8% on the Google Robot drawer opening/closing task, substantially below RT-1-X (59.7%) and SpatialVLA (57.4%). The paper does not discuss this result in the main text, but it represents a clear domain where FALCON's spatial tokens do not provide an advantage and may even be detrimental. This negative result is informative: it suggests that the ESM's 3D reconstruction capabilities are optimised for surface geometry (depth to tables, objects, shelves) rather than for the thin structures, gaps, and kinematic constraints of articulated mechanisms. The spatial tokens may encode the drawer front as a planar surface without capturing the affordance of the handle or the sliding degree of freedom, potentially confusing the action head.

Critical Assessment

The experimental evaluation is extensive in breadth—three simulation benchmarks, 11 real-world tasks, multiple generalisation axes—and the results consistently show FALCON outperforming competitive baselines. However, several aspects of the evaluation warrant scrutiny.

Claim: FALCON's spatial token injection at the action head is superior to VLM injection. This claim is directly tested by the FALCON VLM-tokens ablation (Table 7), which shows a 4.6 percentage point drop in task-5 success and 0.12 Avg. Len. degradation in the zero-shot ABC→D setting. The evidence is clean and the experiment controls for spatial token quality (same ESM, same training data). However, the experiment does not test whether the degradation is due to the injection point per se or due to inadequate fine-tuning of the VLM with spatial tokens. If FALCON VLM-tokens were trained with a more sophisticated strategy (e.g., gradual unfreezing, larger dataset, different learning rates), the gap might narrow. The paper does not report any hyperparameter search for the VLM-tokens variant, so it is possible that FALCON's advantage reflects optimisation difficulty rather than a fundamental architectural limitation. Additionally, the VLM-tokens variant uses concatenation into the VLM input stream, but does not specify how many spatial tokens are injected (all M tokens? a compressed subset?), how positional encodings are handled, or whether the spatial tokens replace or augment existing visual tokens. These details matter—prior work like SpatialVLA uses learned spatial embeddings rather than foundation model tokens, and the failure mode may be specific to FALCON's particular injection implementation rather than to all VLM-injection approaches.

Claim: FALCON achieves state-of-the-art performance. The CALVIN results (Table 1) are SOTA by a narrow margin (4.53 vs. 4.49 Avg. Len. in ABCD→D, 4.40 vs. 4.28 in ABC→D). The SimplerEnv results (Tables 2–3) show larger margins, particularly on the challenging drawer-and-apple task (41.7% vs. ≤24.1%), and the real-world results show substantial improvements in few-shot and spatial understanding settings. However, several caveats apply:

• SpatialVLA was not evaluated on CALVIN, preventing a direct comparison between the two spatial augmentation approaches on the most widely used long-horizon benchmark. The SimplerEnv comparison (Tables 2–3) shows FALCON and SpatialVLA are comparable on some tasks (Stack Block: SpatialVLA leads 29.2% vs. 20.8%; Put Eggplant: both 100%) and FALCON leads substantially on others (Put Spoon: 62.5% vs. 16.7%; Drawer Apple: 41.7% vs. 0%). This mixed picture makes it difficult to claim unqualified superiority.
• The 55B-parameter RT-2-X is a stronger baseline in terms of raw model scale but uses no test-time optimisation (no ensembling, no search). A fairer comparison might give RT-2-X the same inference-time strategies or evaluate FALCON at comparable parameter counts.
• The real-world evaluations use only 5 layouts with 2–10 trials per layout (90 total rollouts for Base Tasks, 40 for Few-shot, 40 for Spatial Understanding). With such small sample sizes, the statistical reliability of the per-task success rates is uncertain—a difference of 20 percentage points on 10 trials corresponds to only 2 rollouts, and confidence intervals would be wide. The paper does not report confidence intervals or statistical significance tests.

Claim: FALCON demonstrates modality transferability. The evidence in Table 5 (ESM depth accuracy) and Figure 7 (policy success rates) shows that adding depth and pose improves both spatial token quality and task performance. The stochastic conditioning strategy clearly enables the model to function with any subset of modalities. However, the paper does not evaluate a critical aspect of transferability: what happens when the model is trained with depth but deployed without it? The stochastic conditioning means the model sees both RGB-only and RGB-D samples during training, so it should handle both at inference. But the paper does not compare this against a model trained exclusively on RGB-D and then evaluated on RGB-only (which would be expected to fail), so there is no direct evidence that the stochastic training is necessary rather than merely sufficient for RGB-only performance. Additionally, the paper does not test transferability across different depth sensor types—the real-world setup uses an Intel RealSense D435i, and it is unclear whether FALCON would maintain its depth-conditioned improvements with a different depth sensor (e.g., stereo camera, time-of-flight sensor) that produces depth maps with different noise characteristics, resolution, or missing-data patterns.

Claim: Spatial foundation models provide strong geometric priors that improve manipulation. The ESM's depth estimation accuracy (90.91% δ < 1.25 from RGB alone, Table 5) is strong, and policy-level improvements in spatial understanding tasks (Figures 6–7) provide indirect evidence that these priors are useful. However, the paper does not provide a direct causal link: there is no experiment showing how policy success rates vary with ESM depth accuracy. A useful ablation would be to compare FALCON against a version where the ESM's spatial tokens are replaced with pseudo-depth from a monocular estimator (e.g., ZoeDepth) or with random spatial tokens, to isolate the contribution of foundation model priors versus any spatial signal. Without this, it is possible that the policy-level improvements come from the action-head fusion architecture and training procedure, not from the VGGT-based spatial tokens specifically. The comparison with SpatialVLA (which uses learned spatial embeddings, not foundation model tokens) partially addresses this, but SpatialVLA differs in multiple ways beyond the spatial representation (different VLM backbone, different injection point, different training data), so the comparison is not controlled.

Missing experiments that would strengthen the paper:

• Per-task CALVIN success rates. Table 1 reports aggregate success rates but not per-skill breakdowns (e.g., success on "open drawer" vs. "rotate blue block"). Such a breakdown would reveal whether FALCON's advantages are concentrated in specific skill types (e.g., placement tasks requiring spatial precision vs. articulation tasks) or are uniform.
• Ablation of ESM pre-training. How does FALCON perform with a randomly initialised ESM (trained from scratch on robot data) versus the VGGT-pretrained ESM? This would isolate the contribution of internet-scale 3D pre-training to the spatial token quality.
• Monocular depth estimator baseline. Replace the ESM's spatial tokens with pseudo-depth from a state-of-the-art monocular depth estimator (ZoeDepth, Depth Anything) encoded into tokens, to test whether any 3D signal helps or whether foundation-model-strength priors specifically are necessary.
• Larger-scale real-world evaluation. With only 5 layouts per task and 2–10 trials per layout, the real-world results are suggestive but not statistically robust. Expanding to 20+ layouts with cross-validation would strengthen confidence in the reported improvements.
• Latency comparison. The paper reports FALCON's inference speed (57 Hz on an RTX 4090) but does not compare it to baselines. If FALCON is substantially slower due to the additional ESM forward pass, the accuracy-speed tradeoff should be quantified for practitioners considering deployment.
• Failure mode analysis. Beyond the qualitative observations in Figure 6 (RoboVLM collisions and premature releases), a systematic categorisation of failure modes (grasp failures, placement errors, instruction misinterpretation, spatial mislocalisation) across methods would strengthen the claim that spatial understanding specifically is the differentiating factor.

Where claims hold and where they are conditional:

• FALCON improves generalisation — strongly supported in the zero-shot CALVIN ABC→D setting (Table 1) and few-shot unseen settings (Tables 10–13), where FALCON's advantages over baselines are larger than in the fully-supervised ABCD→D setting. The pattern of growing gains with longer task horizons (Table 1, task 1 vs. task 5 success rates) is consistent with spatial understanding reducing compounding errors.
• Modality transferability works — supported for the forward direction (adding depth/pose improves performance, Figure 7), but not tested for the reverse direction (training with depth, deploying without) against a depth-only-trained baseline. The evidence supports "graceful improvement with additional sensors" but not necessarily "graceful degradation without them" relative to an RGB-only-trained model.
• Action-head injection preserves VLM alignment — supported by the VLM-tokens ablation (Table 7) but only for one injection implementation. The claim generalises only to the extent that the VLM-tokens variant is representative of prior VLM-injection methods, which is uncertain given implementation differences across SpatialVLA, 3D-VLA, and others.
• Spatial understanding benefits scale with task difficulty — the largest gains are on the hardest tasks (CALVIN ABC→D task 5, SimplerEnv drawer-and-apple, real-world unseen object adaptation), consistent with the hypothesis that spatial priors matter most when the task pushes the limits of the policy. However, the counterexample of open/close drawer (Table 3) where FALCON underperforms suggests that task difficulty is not a sufficient condition—spatial priors must be of the right type for the specific task geometry.

6. Limitations and Trade-offs

GridLocked to a Single Spatial Foundation Model with Unknown Sensitivity

The assumption or constraint. FALCON’s Embodied Spatial Model is built exclusively on VGGT (Wang et al., 2025a), a specific spatial foundation model architecture pre-trained on a specific internet-scale 3D reconstruction dataset. The paper does not test with alternative spatial backbones — DUSt3R (Wang et al., 2024), MASt3R (Leroy et al., 2024), CUT3R (Wang et al., 2025b) — all of which the paper itself cites in Section 2.2 as viable alternatives with distinct architectural properties (pairwise vs. multi-view processing, recurrent vs. feedforward, different representation formats). The VGGT backbone is treated as a fixed design choice rather than a variable to be ablated.

The consequence. A practitioner considering FALCON cannot know whether VGGT is uniquely suited to this architecture or whether any spatial foundation model of comparable scale would produce similar results. This matters because spatial foundation models differ along axes directly relevant to FALCON’s design claims: DUSt3R processes image pairs rather than multi-view sets, which might alter token quality when only a single view is available (the common case in FALCON’s evaluation). MASt3R improves correspondence estimation, which might produce better-aligned spatial tokens for object tracking across frames. CUT3R’s recurrent formulation trades accuracy for computational efficiency, which would affect the 57 Hz inference claim. Without an ablation over spatial backbones, the paper’s reported performance may be specific to VGGT’s inductive biases. Replacing VGGT with an improved future spatial model might not produce monotonic gains — the action head and adapter were trained on VGGT-specific token distributions, and a spatial backbone with different output statistics could cause the fusion mechanism to produce degraded actions until retrained.

What evidence exists in the paper. The paper provides extensive ablations of modality inputs to the ESM (Table 5, Figure 7), fusion strategies (Table 7, Figure 3), and injection positions (Table 7), but no experiment varies the spatial foundation model itself. The VGGT baseline row in Table 5 (91.33% δ < 1.25, 8.53 Abs. Rel) is provided only to validate that FALCON’s ESM replicates VGGT performance when no additional modalities are used. No alternative backbone is evaluated.

Mitigation status. The paper does not acknowledge this as a limitation. There is no discussion of why VGGT was chosen over alternatives beyond a brief statement in Section 2.2 that VGGT “proposes a multi-view architecture that processes multiple images simultaneously, moving beyond pairwise processing to improve reconstruction consistency and robustness.” The modular architecture — the ESM is a separate component connected to the action head via a lightweight adapter — would in principle support swapping spatial backbones, but no evidence demonstrates that this modularity extends to functional plug-and-play across different foundation models.

---

The Difficulty Estimation Problem: No Mechanism for Knowing When Spatial Priors Will Help

The assumption or constraint. FALCON applies the same architecture — ESM-generated spatial tokens fused via element-wise addition — to every input regardless of whether the task requires precise spatial reasoning or whether the scene’s geometry is accurately captured by the ESM. The architecture has no mechanism to assess, at inference time, whether the spatial tokens are reliable or task-relevant, and no ability to fall back to pure semantic action prediction when spatial information is misleading.

The consequence. The Google Robot Open/Close Drawer task (Table 3) reveals a concrete failure mode: FALCON achieves 39.8% success, substantially below RT-1-X (59.7%) and SpatialVLA (57.4%). This is not a marginal underperformance — it is a 17–20 percentage point gap on one of four evaluated tasks. The paper does not diagnose the root cause, but one plausible mechanism is that the ESM’s spatial tokens, trained primarily for surface geometry reconstruction (per-pixel depth, point maps), do not encode the kinematic structure of articulated objects — the sliding degree of freedom, the thin handle geometry, the contact affordances needed for drawer manipulation. The element-wise addition fusion offers no mechanism to down-weight or ignore spatial tokens when they are uninformative or actively misleading for a particular task type. The semantic action token $\hat{\mathbf{t}}_{\text{act}}$ might encode useful information about how to operate the drawer, but the mandatory additive correction from uninformative spatial tokens could distort the action prediction in unpredictable ways.

More generally, this limitation points to an absence of difficulty-conditioned allocation in the architecture. The paper’s motivation (Section 1) argues that spatial priors are needed for tasks requiring height reasoning, object-scale adaptation, and precise placement. This implies that not all tasks benefit equally. Yet FALCON provides spatial tokens uniformly, with no mechanism to assess at inference time whether the current task falls into the “needs spatial reasoning” category (where the tokens help) or the “spatial tokens are noisy for this geometry” category (where they hurt). A deployment pipeline that routes tasks to different policy configurations based on estimated spatial complexity — analogous to how the reference paper on test-time compute scaling conditioned strategy selection on difficulty quintiles — could address this, but FALCON offers no such routing mechanism.

What evidence exists in the paper. The Open/Close Drawer result (Table 3) is the most direct quantitative evidence. The burden is partially circumstantial: FALCON outperforms baselines on tasks clearly requiring height and placement precision (drawer-and-apple: 41.7% vs. ≤24.1%; spatial understanding tasks in Figure 6) but underperforms on an articulated mechanism task where surface geometry is less informative. This pattern — benefits for some task families, costs for others — is consistent with a model that blindly fuses spatial information rather than adaptively weighting it. The paper does not provide per-task-performance analyses that would confirm or refute this hypothesis more systematically (per-skill CALVIN breakdowns, per-task SimplerEnv ablation with and without spatial tokens for each task individually).

Mitigation status. Not addressed. The paper does not discuss the Open/Close Drawer underperformance in the main text or any appendix. There is no gating mechanism, uncertainty estimation, or adaptive fusion weight that would allow the model to modulate its reliance on spatial tokens based on task context or token quality. The fixed element-wise addition operation provides no learned trade-off parameter between the semantic and spatial streams.

---

Difficulty Estimation Cost Is Unaccounted For and Potentially Dominant

The assumption or constraint. FALCON relies on an Embodied Spatial Model with 1.0B parameters that must execute a full forward pass (DINOv2 visual tokenisation, N cross-attention and self-attention blocks, max-pooling, MLP adapter projection) at every timestep to produce the spatial tokens fed to the action head. The paper reports FALCON’s inference speed as approximately 57 Hz on a single NVIDIA RTX 4090 GPU with 12.8 GB memory usage, but does not decompose this into VLM-only inference time versus the additional ESM overhead.

The consequence. A practitioner deciding between FALCON and a lighter-weight spatial augmentation method (e.g., SpatialVLA’s learned spatial embeddings, which presumably add less computation than a 1.0B-parameter transformer) needs to know the accuracy-vs-latency tradeoff. The paper’s ablation (Table 7) shows that FALCON outperforms SpatialVLA-style VLM injection on CALVIN (4.08 vs. 4.00 Avg. Len.), but at what computational cost? If the ESM forward pass accounts for a substantial fraction of the 57 Hz — say, reducing throughput from a potential 80+ Hz for the VLM-only baseline to 57 Hz — then the decision depends on whether the accuracy gain justifies the latency increase for the target deployment.

This cost is analogous to the “difficulty estimation cost” problem in adaptive test-time compute methods, where estimating which strategy to use consumes a non-trivial fraction of the total budget. Here, the “estimation” — producing spatial tokens — happens on every timestep rather than being amortised over a rollout. For deployments where latency is critical (high-speed pick-and-place, reactive grasping) or where multiple policies must run concurrently on limited GPU resources (edge deployment on embedded GPUs much weaker than an RTX 4090), the ESM overhead might be prohibitive.

What evidence exists in the paper. The paper reports aggregate inference speed (57 Hz) and GPU memory (12.8 GB) in Section 8, but does not provide a VLM-only baseline speed. The ESM architecture is described as 1.0B parameters, approximately 34% of FALCON’s total 2.9B parameters, but parameter count is an imperfect proxy for latency — the ESM uses DINOv2 (a frozen vision transformer) for visual tokenisation, which may itself be computationally heavy. There is no explicit comparison of FALCON’s inference speed against the Kosmos-VLA-2D baseline (the VLM with action head, no ESM) or against competing methods like SpatialVLA or RoboVLM.

Mitigation status. Partially acknowledged. The paper mentions that “this approach not only yields superior success rates but also provides greater training stability and computational efficiency” when justifying element-wise addition over cross-attention and FiLM-gated fusion (Section 3.4), but “computational efficiency” refers narrowly to the fusion mechanism (element-wise addition introduces no extra parameters vs. the others) rather than to the full ESM overhead. The paper does not frame ESM inference cost as a limitation, nor does it propose lighter-weight spatial encoders, distillation strategies, or conditional execution where the ESM is invoked only on timesteps or tasks that require spatial reasoning.

---

Single Robot, Single Camera, Single Model Family

The assumption or constraint. All experiments — simulation and real-world — use a single robot platform (xArm 6 with Robotiq gripper) with a single camera configuration (static side-mounted Intel RealSense D435i, plus optional wrist camera for CALVIN only). The VLM backbone is exclusively Kosmos-2 (1.6B parameters). The spatial backbone is exclusively VGGT. The real-world data is collected via teleoperation with a Spacemouse device at 10 Hz, producing a specific demonstration quality and action distribution.

The consequence. The paper cannot distinguish between findings that generalise across robot embodiments and those specific to this hardware-software stack. The xArm 6 is a 6-DoF industrial-style arm with good repeatability but limited compliance — policies that work well on this platform may fail on more compliant, lower-precision arms (e.g., the WidowX used in SimplerEnv evaluations, which has different kinematics and joint-level dynamics). The fixed side camera at ~0.6 m distance (Appendix 9, Figure 9) may be an optimal configuration for the ESM’s depth estimation range, masking degradation that would occur with different camera placements (overhead, mobile, wrist-only).

More critically, the reliance on Kosmos-2 as the sole VLM backbone means the paper’s central architectural claim — that spatial information should route to the action head rather than the VLM — is validated for exactly one VLM architecture. Kosmos-2 has specific properties (grounding capabilities, a particular tokenisation strategy, a specific pre-training data mixture) that may make it more or less sensitive to embedding drift than other VLMs. A VLM with stronger built-in 3D awareness from its pre-training (e.g., one trained on video data with explicit depth supervision, or a model like LLaVA-OneVision that processes multi-view inputs) might not benefit from action-head injection in the same way. Conversely, a smaller VLM with weaker semantic representations might be disrupted even more severely by spatial token injection than Kosmos-2, making FALCON’s architectural choice relatively more important for smaller models — but this hypothesis is untested.

For the real-world evaluation, all training data comes from a single teleoperation setup using a Spacemouse. Teleoperation quality varies substantially across operators and devices — a Spacemouse produces different trajectory characteristics (smoothness, speed profiles, correction frequency) than a haptic device, a VR controller, or kinesthetic teaching. Policies trained on this specific demonstration distribution may not transfer to data collected with different teleoperation interfaces, limiting the generalisability claims.

What evidence exists in the paper. The breadth of evaluation — CALVIN, two SimplerEnv setups (WidowX and Google Robot, which are simulated replicas of different physical robots), and real-world tasks — provides some cross-embodiment evidence. FALCON outperforms baselines across all three simulation platforms, suggesting the architecture is not uniquely tuned to the xArm 6. However, the WidowX and Google Robot evaluations in SimplerEnv are simulation-based and thus do not capture real-world dynamics differences across embodiments. The paper does not test with an alternative VLM backbone, an alternative camera placement, or data collected with a different teleoperation interface. The real-world evaluation is limited to a single physical robot with a known camera extrinsics.

Mitigation status. Not acknowledged. The paper treats Kosmos-2 as “representative of many contemporary LLMs” (paraphrasing from the VGGT discussion), but this claim is stated about VGGT in the calibration validation, not extended to the VLM choice. There is no discussion of how FALCON’s design principles might or might not transfer to other VLM families (LLaVA, Prismatic-7B as used in OpenVLA, PaLI-X as used in RT-2). The single-robot, single-camera limitation is not discussed.

---

Real-World Evaluation Sample Sizes Are Too Small to Support Quantitative Claims at Per-Task Granularity

The assumption or constraint. The real-world evaluations use small trial counts by the standards of statistical reliability. Base Tasks: 9 task suites × 10 layouts × 10 trials = 90 total rollouts. Few-shot Adaptation: 4 tasks under each of 5 settings × 5 layouts × 2 trials per layout = 200 total rollouts across Tables 9–13, but only 10 trials per (task, setting) pair. Spatial Understanding: 4 tasks × 5 layouts × 2 trials = 40 total rollouts. Per-task success rates are therefore estimates based on 10 or 20 binary trials with no reported confidence intervals.

The consequence. The headline result “FALCON achieves 80% success on open drawer and place tennis ball in the Unseen Object setting, while other models demonstrate near-zero success” (Section 4.2, Few-shot Adaptation) is based on 10 trials (Table 10: “5 different layouts with 2 trials per layout” per Appendix 9). A success rate of 8/10 has a 95% binomial confidence interval of approximately 44–97% — wide enough that the “80%” point estimate could overstate or understate the true success rate by a large margin. The comparison between FALCON (80%) and SpatialVLA (20%) on this task is based on 8 successes vs. 2 successes across 10 trials each — a Fisher’s exact test would yield p ≈ 0.023, which is nominally significant but fragile (one additional failure by FALCON or one additional success by SpatialVLA changes the conclusion).

This matters because the real-world results carry most of the paper’s persuasive weight for the practical value of spatial priors. Simulation results show narrow margins (CALVIN: 4.53 vs. 4.49 Avg. Len.) or mixed outcomes (SimplerEnv: SpatialVLA leads on Stack Block, 29.2% vs. 20.8%). The large claimed margins — 25.6 percentage points in Base Tasks, 27.5 percentage points in Simple Few-shot, 80% vs. near-zero in Unseen Object — come from real-world experiments with the smallest sample sizes. A practitioner basing deployment decisions on these numbers needs to know how stable they are.

Additionally, the paper reports per-setting averages that aggregate across tasks with different difficulties and trial counts. The Figure 5 bar chart labelled “Unseen Average” aggregates Unseen Object, Unseen Background 1, Unseen Background 2, and Unseen Task Description — settings with different expected difficulty levels — and the averaging may obscure that FALCON’s advantage is concentrated in specific settings (e.g., Unseen Object where the gap is largest) rather than uniform.

What evidence exists in the paper. The trial counts are stated in Appendix 9. The paper acknowledges these counts but does not discuss their implications for statistical uncertainty. No confidence intervals, standard errors, or significance tests are reported for any real-world result. By contrast, the CALVIN evaluation with 1000 rollouts provides a large sample that reduces sampling variance to a negligible level — but CALVIN shows the smallest performance gaps between FALCON and baselines, creating a tension where the largest claimed improvements coincide with the least statistical power.

Mitigation status. Not addressed. The paper does not mention sample size as a limitation, does not report uncertainty estimates, and presents real-world results as point estimates in bar charts without error bars. The use of consistent trial counts across methods (all baselines evaluated on the same layouts and trial counts) mitigates comparison bias — any sampling noise affects all methods equally — but does not mitigate estimation uncertainty. A practitioner cannot distinguish a real 25.6 percentage point improvement from a true improvement of 10 points that was inflated by favourable sampling in the 90-trial Base Tasks evaluation.

7. Implications and Future Directions

How This Work Changes the Landscape

FALCON represents a rearchitecting of the VLA design space, not an incremental improvement on existing 3D-enhanced VLAs. The core conceptual shift is reframing where spatial information belongs in a robot policy: not in the vision-language backbone where reasoning happens, but in the action head where motor commands are generated. This inverts a design assumption that has been pervasive across the VLA literature—from PointVLA to SpatialVLA to 3D-VLA—that the VLM should have access to all available modalities to "reason jointly" over them. FALCON demonstrates that this joint-reasoning assumption comes with a measurable cost: the FALCON VLM-tokens ablation in Table 7 shows that injecting the same spatial tokens into the VLM rather than the action head drops zero-shot task-5 completion on CALVIN ABC→D from 62.2% to 57.6%, a 4.6 percentage point degradation from changing only the injection point.

This finding, if it generalises beyond Kosmos-2, changes how researchers should think about adding modalities to pretrained foundation models. The prevailing instinct—"give the model more information and let attention sort it out"—is challenged by evidence that pretrained cross-modal alignments are fragile and that third modalities introduced during fine-tuning cause embedding drift that disproportionately harms generalisation. FALCON's solution—route new modalities to the output stage where they serve as additive corrections to an otherwise untouched pretrained policy—establishes a separation-of-concerns design principle that may apply well beyond spatial tokens. Any VLA that wants to incorporate tactile sensing, force feedback, audio, or proprioceptive history without degrading its language grounding could adopt the same architectural pattern: keep the VLM's input stream fixed to its pretrained modalities, encode the new signal through a dedicated model, and fuse at the action head.

The work also resolves a tension that has split the 3D VLA literature into two camps with opposite failure modes. Methods requiring explicit 3D sensors (PointVLA, GeoVLA, 3D Diffusion Policy) achieve strong geometric awareness when sensors are available but collapse without them—a property the paper names low modality transferability. Methods using weak 3D cues from RGB alone (SpatialVLA, Evo-0) work without specialised hardware but cannot exploit RGB-D cameras or calibrated poses when available, leaving performance on the table. FALCON shows that this tradeoff is unnecessary: stochastic conditioning during training (Equation 4, Section 3.3) produces a single model that extracts strong geometric priors from RGB alone (90.91% δ < 1.25 depth accuracy, Table 5) and improves monotonically when depth or pose are added (99.79% with depth). The same trained policy runs on a cheap RGB-only robot for routine tasks and an RGB-D robot for precision tasks without retraining—a practical unification the field lacked.

Perhaps most consequentially, FALCON demonstrates that spatial foundation models are a drop-in source of geometric priors for VLAs, analogous to how language foundation models provide semantic priors. Just as VLAs inherit syntax and world knowledge from pre-trained LLMs without relearning them from robot data, FALCON shows they can inherit 3D understanding from pre-trained spatial models without needing to learn geometry from manipulation demonstrations. This is a transfer learning insight with compounding benefits: as spatial foundation models improve—and the field is moving rapidly, from DUSt3R to MASt3R to VGGT to future architectures—VLA spatial reasoning improves automatically by swapping the ESM backbone, without retraining the policy or recollecting manipulation data. The modular architecture makes this straightforward in principle, though the paper does not empirically demonstrate plug-and-play across spatial backbones.

The work also makes less-visible but important methodological contributions. The two-stage training paradigm with zero-initialised adapter (Appendix 6.1) provides a template for integrating new modalities into pretrained VLAs without catastrophic forgetting—Stage 1 trains only a lightweight adapter with its final layer zeroed out, ensuring the spatial contribution starts at zero and grows gradually, while Stage 2 unfreezes the VLM only after the adapter has learned to align spatial and semantic features. This phased approach is broadly applicable to any multi-modal fine-tuning problem where the pretrained backbone's representations must be preserved. The paper's choice to standardise training duration rather than using validation-based early stopping (Appendix 7) addresses a practical challenge in robot learning—that validation loss correlates poorly with policy performance due to compounding rollout errors—and establishes an evaluation protocol that future VLA comparisons would benefit from adopting.

Follow-Up Research This Work Enables

Testing whether action-head injection generalises across VLM families. The paper validates its central architectural claim—spatial tokens should route to the action head, not the VLM—using exactly one VLM backbone: Kosmos-2 (1.6B parameters). This leaves open the critical question of whether the finding is VLM-specific or a general principle. A strong follow-up would replicate the FALCON VLM-tokens vs. FALCON action-head ablation (Table 7) across at least three VLM families: a LLaVA-based VLA (e.g., LLaVA-1.5 with action head, following OpenVLA's architecture), a Flamingo-style VLA (e.g., RoboFlamingo with spatial tokens injected), and a larger model (e.g., a 7B-parameter VLM to test whether scale reduces sensitivity to embedding drift). Each would be evaluated on CALVIN ABC→D and ABCD→D with identical ESM, training data, and protocol. If the action-head advantage persists across all backbones, the design principle is validated at the architecture level. If the gap narrows or reverses for certain VLM families (e.g., VLMs with stronger built-in 3D awareness from video pre-training might tolerate VLM injection better), we learn boundary conditions on when the cerebrum-cerebellum separation matters. A negative result—Kosmos-2 being uniquely sensitive, with other VLMs showing no gap—would mean FALCON's architectural innovation is primarily a Kosmos-2-specific optimisation rather than a general principle.

What is the minimal spatial representation needed for the observed gains? FALCON uses VGGT, a 1.0B-parameter spatial foundation model that produces dense per-patch 3D tokens encoding metric depth, point maps, and camera pose. The paper does not establish whether these rich tokens are necessary or whether simpler geometric signals would suffice. A targeted ablation series would replace the ESM's spatial tokens with: (a) pseudo-depth from ZoeDepth or Depth Anything encoded as tokens via the same depth encoder architecture; (b) learned spatial embeddings as in SpatialVLA but fused through FALCON's action head rather than the VLM (to isolate representation quality from injection point); (c) ground-truth depth from the simulator tokenised identically to FALCON's depth encoder (to test whether the policy benefits from any accurate geometric signal or specifically from foundation-model-trained representations); and (d) placeholder random tokens with matching statistics (mean, variance matched to VGGT tokens) to test whether any additional feature dimension helps regardless of content. Each variant would be evaluated on the CALVIN ABC→D split and the real-world spatial understanding tasks from Figure 6. If pseudo-depth tokens approach FALCON's performance, the practical conclusion is that cheap monocular depth estimators are sufficient and the 1.0B-parameter ESM is unnecessary overhead. If ground-truth depth underperforms VGGT tokens, the conclusion is that foundation model pre-training on diverse 3D data produces representations that are useful beyond raw metric accuracy—perhaps encoding surface normals, occlusion boundaries, or object-part segmentation that depth alone lacks.

Combining FALCON's action-head injection with test-time compute scaling for verifier-free spatial adaptation. The paper demonstrates that the ESM's spatial token quality improves when depth and pose are available at inference (Table 5: δ < 1.25 jumps from 90.91% to 99.79%). But the architecture has no mechanism to request additional viewpoints, move the camera, or invoke more expensive spatial processing when the current tokens are likely unreliable. This connects directly to the test-time compute scaling paradigm evaluated in the reference paper: just as some prompts benefit from beam search against a PRM while others benefit from iterative revision, some manipulation scenes might benefit from multi-view spatial aggregation (e.g., capturing the scene from slightly different angles and fusing spatial tokens) while others are adequately served by single-view tokens. A concrete experiment would implement a difficulty estimator that assesses spatial token quality at inference time—perhaps using the variance of depth predictions across multiple stochastic forward passes of the ESM, or the disagreement between ESM-predicted depth and a fast monocular depth estimator—and conditionally invokes additional ESM forward passes from virtual viewpoints (synthesised via the learned camera token perturbation) when uncertainty is high, fusing the resulting spatial tokens before feeding to the action head. Evaluation on the real-world spatial understanding tasks (height variation, scale variation from Figure 6) would test whether adaptive spatial compute improves performance on geometrically challenging scenes without wasting computation on simple ones.

Scaling laws for spatial foundation model size vs. VLA policy performance. FALCON uses a 1.0B-parameter ESM with a 1.6B-parameter VLM. Is this ratio near-optimal, or would the policy benefit substantially from a larger ESM? Conversely, how small can the ESM be before spatial understanding degrades below an acceptable threshold? The paper's modular architecture makes this study tractable without modifying the VLM or action head. A scaling experiment would train FALCON variants with ESM sizes spanning, say, 100M, 300M, 600M, 1B, and 2B parameters (controlled by varying transformer depth and width in the spatial encoder), each pre-trained on the same 3D reconstruction data, and evaluate on CALVIN ABC→D and the real-world spatial understanding tasks. The output would be a scaling curve relating ESM parameter count to policy success rate, analogous to the pretraining scaling laws that inform model size decisions. A flattening of the curve at ~600M would suggest the current 1B ESM is over-parameterised for the manipulation domain; continued improvement to 2B would justify investing in larger spatial backbones. This experiment also tests whether ESM scale matters more or less than VLM scale—a key question for practitioners allocating a fixed parameter budget.

Does FALCON's design benefit long-horizon mobile manipulation beyond tabletop settings? All experiments—CALVIN, SimplerEnv, real-world—operate in tabletop manipulation with a fixed side camera and a stationary robot arm. The architectural principles (ESM for geometric understanding, action-head fusion) are not inherently tabletop-specific, but several design choices are tuned to this setting: the ESM processes a single third-view image (not a sequence from a moving camera), the depth and pose encodings assume known camera extrinsics, and tasks involve object manipulation within a fixed workspace. A stress-test would deploy FALCON on a mobile manipulation benchmark—e.g., a fetch-and-deliver task where the robot navigates between rooms, picks objects from different surfaces, and places them in target locations. The critical adaptation would be replacing the single-image ESM with a multi-view variant that aggregates spatial tokens across a trajectory of camera views (the paper's Appendix 11 already shows that adding wrist camera images to the ESM provides a small CALVIN improvement: Avg. Len. from 4.08 to 4.10, Table 8). The experiment would test whether FALCON's spatial priors help with (a) navigation to target locations (where spatial tokens could encode room geometry), (b) object search (where spatial tokens from multiple viewpoints could resolve occlusions), and (c) precise placement in cluttered target areas. Failure modes on mobile tasks would identify which architectural assumptions break outside the tabletop setting and what adaptations are needed for generalisation.

Characterising and mitigating the articulated-object blind spot. The Google Robot Open/Close Drawer task (Table 3) reveals a clear weakness: FALCON achieves 39.8% success, substantially below RT-1-X (59.7%) and SpatialVLA (57.4%). The paper offers no diagnosis, but a plausible mechanism is that the ESM's spatial tokens—trained for surface geometry reconstruction—do not encode the kinematic structure of mechanisms (hinges, sliders, handles). A diagnostic experiment would analyse the ESM's spatial tokens on drawer images: do the tokens at handle and drawer-edge locations encode useful geometric features (depth discontinuities, surface normal changes) that could inform grasping and pulling? If the tokens lack kinematic-relevant information, the fix might be to add a small set of mechanism-specific training data to the ESM—images of articulated objects with annotations for joint type, axis, and state—or to add an auxiliary loss during ESM fine-tuning that predicts affordance maps (grasp points, push/pull directions) from spatial tokens. A simpler architectural fix would be to add a learned gating weight to the element-wise addition fusion that allows the action head to down-weight spatial tokens when they are uninformative for the current task phase—effectively providing the adaptive fusion that the current fixed-addition design lacks. Evaluation on a suite of articulated object tasks (drawers, cabinets, doors, valves) would quantify the gap and test whether targeted ESM fine-tuning or adaptive fusion closes it.

Practical Applications and Downstream Use Cases

Heterogeneous robot fleets with a single policy. FALCON's stochastic conditioning strategy (Equation 4) means a single trained model can be deployed across a fleet of robots with different sensor configurations—some with RGB-D cameras, some with RGB-only, some with calibrated camera poses, some without—and performance scales naturally with sensor quality. This has direct economic implications for warehouse automation, where a company might deploy high-precision RGB-D cameras on robots handling delicate or variable-sized items (where the 20 percentage point improvement from depth on height-varying tasks in Figure 7 matters) while equipping robots doing routine pick-and-place with cheaper RGB cameras. The same trained FALCON policy serves both, eliminating the engineering cost of maintaining per-hardware policy variants and the risk of deploying the wrong variant on the wrong robot. The paper's demonstration that FALCON with RGB-only achieves performance comparable to a point-cloud-based VLA with RGB-D (Table 4: FALCON RGB-only 4.08 Avg. Len. vs. Kosmos-VLA RGB-D 4.05 on CALVIN ABCD→D) suggests that even the low-cost configuration remains competitive. The 12.8 GB GPU memory requirement and 57 Hz inference speed on a consumer RTX 4090 (Section 8) make this feasible on mid-range edge GPUs, though deployment on embedded hardware (Jetson-class) would require model distillation or quantisation that the paper does not explore.

Data-efficient fine-tuning for custom manipulation tasks. The few-shot adaptation results (Tables 9–10) show FALCON achieving 87.5% average success on real-world tasks with only 20 demonstrations per task (80 total trajectories), while the next best method (RoboVLM) achieves 60.0% with the same data. For small-to-medium enterprises deploying robots for specific manipulation tasks (e.g., a bakery wanting a robot to arrange pastries in packaging, a laboratory wanting a robot to handle sample vials), the ability to achieve high success rates from minimal in-house demonstration data—without requiring depth sensors or calibrated cameras—lowers the barrier to adoption. A technician collects a few dozen teleoperated demonstrations of the specific task using the same RGB camera the robot already has; FALCON inherits geometric understanding from its spatial foundation model pre-training (no need for the technician to collect depth data or calibrate cameras) and semantic understanding from its VLM pre-training (no need for the technician to label objects or write complex task descriptions beyond natural language instructions). The policy is then fine-tuned on the small custom dataset using the same two-stage procedure described in Appendix 6.1, and deployed. The 57 Hz inference speed is sufficient for most non-high-speed manipulation tasks. The key enabling factor is FALCON's spatial priors being sample-efficient—the model already understands 3D geometry from internet-scale pre-training, so the few-shot demonstrations only need to teach task-specific behaviours (which object to grasp, where to place it) rather than teaching geometry from scratch.

Automated spatial reasoning evaluation for VLA benchmarking. The paper's spatial understanding capability evaluations (Figure 6)—testing adaptation to object scale variation (3 cm vs. 5 cm blocks when trained on 4 cm), target height variation (placing a cup on an elevated surface), and spatial-prompt instructions (e.g., "place on the left shelf")—provide a diagnostic toolkit that the VLA community could standardise. Current VLA benchmarks focus on aggregate task success rates that conflate semantic understanding, grasp execution, and spatial reasoning into a single number. FALCON's decomposition into separate evaluation axes (language grounding in cluttered scenes, spatial perception under geometric variation, few-shot robustness to novel objects and backgrounds) enables finer-grained diagnosis of where a particular VLA architecture succeeds or fails. The observation that FALCON substantially outperforms baselines on spatial tasks (Figure 6) but underperforms on articulated mechanisms (Table 3, Open/Close Drawer: 39.8% vs. 57.4–59.7% for baselines) would be invisible in an aggregate benchmark that averages across task types. Adoption of this diagnostic evaluation protocol—even without adopting FALCON's architecture—would accelerate progress by directing research attention to specific capability deficits rather than chasing aggregate benchmark numbers.

Drop-in spatial perception upgrade for existing VLA deployments. FALCON's modular design—the VLM, ESM, and action head are separate components connected through a lightweight adapter—means the spatial perception pipeline can be retrofitted onto existing VLA deployments without replacing the entire policy stack. An organisation that has already invested in fine-tuning a VLA (e.g., OpenVLA, RT-2) on proprietary manipulation data could: (1) keep their existing VLM backbone and action head frozen; (2) attach an ESM (pre-trained VGGT or a future improved spatial model) with a newly trained adapter using FALCON's Stage 1 procedure (train only the adapter, zero-initialised, on the existing manipulation data with the VLM and action head frozen); (3) optionally unfreeze the VLM for Stage 2 joint refinement if additional data is available. The result would be their existing policy augmented with spatial understanding, with the original behaviour preserved as a baseline (since the zero-initialised adapter ensures the spatial contribution starts at zero) and geometric corrections added incrementally. This modular upgrade path is more practical than retraining an entire 3D-enhanced VLA from scratch and avoids the embedding-drift risk that FALCON's VLM-tokens ablation (Table 7) demonstrated for VLM-injection approaches. The 1.0B-parameter ESM adds computational cost, but the paper's 57 Hz inference on a single consumer GPU suggests the overhead is manageable for most non-latency-critical deployments.

When to Prefer This Method

The paper positions FALCON explicitly against two alternative paradigms—explicit 3D input methods (PointVLA, GeoVLA, 3D Diffusion Policy) that require depth sensors or point clouds, and weak 3D cue methods (SpatialVLA, Evo-0) that inject approximate or learned 3D features into the VLM. The choice between these and FALCON is not uniform; it depends on the deployment context and task distribution.

Prefer FALCON over explicit 3D input methods when:

• The deployment hardware includes a mix of sensor configurations (some RGB-only, some RGB-D), and you need a single policy to serve all of them without per-robot retraining. FALCON's stochastic conditioning enables this; PointVLA-style methods break without depth.
• Training data lacks aligned 3D annotations (the common case for large-scale datasets like Open X-Embodiment). FALCON's ESM provides geometric priors from RGB alone, leveraging internet-scale 3D pre-training rather than requiring depth-annotated robot data.
• Tasks involve generalisation to unseen objects and backgrounds where semantic understanding must remain intact. FALCON's action-head injection preserves VLM alignment (Table 7: VLM-tokens degrades zero-shot performance); explicit 3D methods that modify the VLM input space may suffer similar embedding drift.
• However, if all deployment robots have high-quality depth sensors and tasks primarily involve articulated mechanism manipulation (drawers, cabinets, valves), explicit 3D methods may be preferable—FALCON underperforms on Open/Close Drawer (39.8% vs. 59.7% for RT-1-X, Table 3).

Prefer FALCON over weak 3D cue methods when:

• Tasks require metric-level spatial precision—adapting to object scale variation (3–5 cm blocks, Figure 6), height variation (elevated placement targets, Figure 7), or precise positioning relative to 3D structures (drawer-and-apple task: 41.7% vs. 0% for SpatialVLA, Table 3). Weak cues provide relative depth ordering but lack the absolute metric accuracy that foundation model spatial tokens encode.
• Modality transferability matters—you may add depth sensors to the robot fleet later and want the policy to improve immediately without retraining. SpatialVLA-style methods have no mechanism to ingest depth or pose tokens; FALCON's ESM accepts them via the same stochastic conditioning pathway used during training.
• Zero-shot generalisation to unseen environments is critical. The larger gap between FALCON and baselines in zero-shot ABC→D vs. fully-supervised ABCD→D (Table 1) and in few-shot unseen settings vs. simple settings (Tables 9–13) suggests FALCON's spatial priors are particularly valuable when the policy must generalise beyond its training distribution.

Prefer FALCON over both alternatives when:

• Training data is extremely limited (few-shot adaptation with ~20 demonstrations per task). FALCON's 27.5 percentage point improvement over the next-best method in the Simple few-shot setting (87.5% vs. 60.0%, Figure 5) suggests the spatial priors from VGGT pre-training are highly sample-efficient—they provide useful geometric information even when task-specific manipulation data is scarce.
• The deployment involves language-conditioned tasks with spatial referents (e.g., "place on the red plate" where the red plate must be distinguished from a yellow distractor by its spatial location). FALCON's real-world Base Tasks highlight improved instruction following for spatially-grounded commands (Section 4.2).
• You need a single architecture that degrades gracefully across sensor configurations rather than failing catastrophically when expected inputs are missing, and you are willing to pay the ~1.0B-parameter ESM overhead (57 Hz on RTX 4090) for this robustness.