Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training

🎯 Pitch

A vision-language model that watches hours of video without forgetting earlier scenes—its hidden state rewires itself on the fly to accumulate a structured 3D map, hitting state-of-the-art spatial reasoning and still answering correctly 120 minutes later while all other models collapse to zero.

1. Executive Summary

This paper introduces Spatial-TTT, a framework for streaming visual-based spatial intelligence that uses test-time training to maintain adaptive fast weights — lightweight neural network parameters updated online — as a compact, non-linear memory for accumulating 3D spatial evidence from long-horizon video streams. Experiments across VSI-Bench, MindCube-Tiny, VSI-SUPER-Recall, and VSI-SUPER-Count using Qwen3-VL-2B-Instruct as the base model demonstrate state-of-the-art performance, with Spatial-TTT-2B achieving a 64.4 macro-average on VSI-Bench (outperforming GPT-5's 55.0 and the strongest open-source spatial model VST-7B-SFT at 60.6) and a 76.2 accuracy on MindCube-Tiny (improving 12.3 points over the best proprietary baseline). The framework combines three mechanisms — a hybrid TTT architecture interleaving test-time training layers with standard self-attention anchor layers at a 3:1 ratio (preserving pretrained cross-modal alignment while compressing long-range temporal dependencies into fast weights), a spatial-predictive mechanism with depthwise 3D spatiotemporal convolutions on query, key, and value projections (injecting local neighborhood inductive bias so fast weights learn predictive mappings between spatiotemporal contexts rather than isolated tokens), and large-chunk updates with parallel sliding-window attention (enabling linear-complexity streaming at over 40% reduction in both memory and TFLOPs versus standard attention at 1024 frames) — together with a dense scene-description dataset providing rich, scene-level supervision for learning effective fast-weight update dynamics. Efficiency gains are most pronounced on long-horizon streaming tasks: on VSI-SUPER-Count, Spatial-TTT maintains 38.4% accuracy at 120 minutes while all baselines collapse to near zero, establishing that test-time training enables continual spatial evidence accumulation even as standard models exhaust their context budgets or run out of memory.

2. Context and Motivation

The Core Problem: Streaming Video Intelligence Requires Continual Spatial Memory

The fundamental challenge this paper addresses is deceptively simple: how do we build vision-language models that can watch an arbitrarily long video stream — potentially hours of footage — and continuously accumulate, update, and reason about the 3D spatial structure of the environment? This is not merely a "longer context window" problem. It is a fundamentally different computational paradigm than processing a bounded sequence of frames, and it exposes a critical gap in how modern multimodal large language models (MLLMs) handle visual information.

To understand why, consider what happens when an embodied agent — a robot, an autonomous vehicle, or an augmented reality headset — navigates through a real environment. It does not receive a curated batch of 16 or 32 well-spaced frames from a fixed viewpoint. Instead, it experiences a continuous stream of visual observations: the camera pans across a room, objects enter and exit the field of view, previously seen furniture becomes occluded behind walls, and the agent must remember what was where even when those objects are no longer visible. Spatial understanding in this setting is fundamentally incremental — the model must build a mental map piece by piece, integrating observations across time and viewpoint, and then reasoning about that accumulated 3D evidence when answering questions or planning actions.

The paper identifies this capability as streaming spatial intelligence:

"This necessitates streaming spatial understanding—the capability to selectively maintain, progressively update, and reason over spatial memory from long-horizon video inputs."

The key word here is "streaming." It implies that (1) the input is potentially unbounded in length, (2) observations arrive sequentially and cannot be revisited arbitrarily without cost, and (3) the model must maintain a persistent state that accumulates evidence over time rather than recomputing everything from scratch. This is a harder problem than offline video understanding, where the entire video is available for random access and bidirectional processing.

Why This Problem Matters: Spatial Intelligence as a Cornerstone Capability

The paper situates streaming spatial intelligence within a broader vision of embodied AI — systems that operate in the physical world rather than in curated datasets. The motivation spans several application domains, each with distinct failure modes when spatial memory fails:

Embodied robotics. A household robot navigating from the kitchen to the living room must remember which room contains which furniture, where obstacles were encountered, and how spatial relationships change as it moves. If the robot forgets that a chair was behind the sofa simply because it turned a corner, it cannot plan efficient paths or locate requested objects. The paper explicitly invokes this motivation, citing work on embodied multimodal language models (Driess et al., 2023) and spatiotemporal reasoning for robotic manipulation (Huang et al., 2024).

Autonomous driving. A self-driving car processing streaming sensor data must maintain a persistent representation of surrounding vehicles, pedestrians, and road geometry — even for objects that are temporarily occluded or have exited the current sensor range. Losing track of a cyclist who disappeared behind a truck for a few seconds has obvious safety implications. The paper cites planning-oriented autonomous driving work (Hu et al., 2023) and spatial-aware vision language models for driving (Wei et al., 2025).

Augmented reality. AR devices overlay digital information onto the physical world. To accurately place virtual objects, the device must maintain a running 3D understanding of the environment as the user moves through it — where walls are, how rooms connect, what objects occupy which positions. If the spatial model degrades over time, virtual objects drift, occlusions break, and the immersive experience collapses. The paper references the Ego4D dataset (Grauman et al., 2022) as representative of this egocentric streaming domain.

Beyond these application-specific needs, there is a theoretical significance to the problem. The paper argues that spatial intelligence — the ability to perceive, reason about, and recall 3D structure and geometric relationships — is a "cornerstone capability" for general visual understanding. Current MLLMs, while impressive at 2D semantic tasks (object recognition, scene classification, image captioning), show a fundamental fragility when spatial relationships matter. They may correctly identify a chair and a table in an image but fail to determine which is closer to the camera, or whether the chair is to the left or right of the table from a given viewpoint. This is not a minor limitation — it reflects the absence of 3D geometric priors in models trained predominantly on 2D image-text pairs, where spatial structure is implicit and rarely supervised.

Where Prior Approaches Fall Short

The paper identifies three broad categories of prior work and systematically explains why each is insufficient for streaming spatial intelligence.

1. General-Purpose MLLMs: Lack 3D Geometric Priors

Modern MLLMs — including the Qwen-VL, InternVL, LLaVA-OneVision, and LLaVA-Video families evaluated in the paper — achieve strong results on 2D visual understanding benchmarks. However, as the paper notes:

"Their performance degrades significantly on tasks requiring spatial understanding, primarily due to the inherent lack of 3D geometric priors, as these models are predominantly trained on 2D semantic-level image-text pairs without the supervision of spatial structure."

This is not a failure of scale (larger models still struggle) or of data quantity. It is a structural limitation: these models learn to attend to semantic features but have no mechanism for encoding metric distances, relative 3D positions, or viewpoint-invariant object locations. The VSI-Bench results in Table 1 bear this out concretely: general-purpose MLLMs score dramatically lower on spatial tasks than on typical VQA benchmarks. For instance, Qwen3-VL-2B-Instruct — the base model from which Spatial-TTT is initialized — achieves only 40.2 on Absolute Distance and 42.0 on Relative Direction despite being a capable general-purpose model.

The longer-context variant of this limitation is equally important. Even when models are technically capable of processing more frames (via larger context windows like Gemini 1.5's million-token context), the quadratic complexity of standard self-attention makes processing thousands of frames prohibitively expensive. The paper quantifies this in Table 5: Qwen3-VL-2B's TFLOPs more than double when going from 256 to 512 frames (179.9 → 473.9) and again when going to 1024 frames (→ 1403.1) — nearly 8× growth for 4× the frames. Memory usage follows a similar trajectory. For truly streaming applications where videos may be hours long at 1 fps (thousands to tens of thousands of frames), this scaling is fundamentally unsustainable.

Aggressive temporal subsampling is not a solution. The paper notes that existing approaches often downsample long videos to 16–32 frames to fit within compute budgets:

"Aggressive temporal subsampling inevitably discards fine-grained spatial details critical for accurate 3D reasoning."

A 10-minute video at 1 fps contains 600 frames. Sampling it down to 32 frames means the model sees approximately 5% of the visual evidence. Objects that appear only briefly, spatial relationships that depend on precise viewpoint changes, and occlusion events that require frame-to-frame continuity are all lost. The spatial understanding problem cannot be solved by simply throwing away most of the visual data.

2. Spatial-Aware MLLMs: Stronger but Bottlenecked at Short Contexts

A more recent line of work has explicitly targeted the spatial weakness of general MLLMs by augmenting input representations with geometric cues. These models — Spatial-MLLM, VLM-3R, MindCube, VST, Cambrian-S, SpatialLadder, SpaceR, and others compared in Tables 1 and 2 — incorporate strategies such as:

Metric depth injection: Providing explicit per-pixel depth estimates alongside RGB frames (e.g., MM-Spatial; Daxberger et al., 2025).
Multi-view geometry encoders: Using feed-forward 3D reconstruction models to produce geometric features that are concatenated with visual tokens (e.g., Spatial-MLLM; Wu et al., 2025a and VLM-3R; Fan et al., 2025).
Spatial fine-tuning: Curating large-scale spatial QA datasets (VSI-590K from Cambrian-S at 590K examples; VST at 4.1M SFT examples) and fine-tuning MLLMs on spatial reasoning tasks.
Reinforcement learning for spatial reasoning: Using RL to optimize spatial task performance (SpaceR, MindCube).
3D feature alignment: Aligning output features with 3D representations (3DThinker; Chen et al., 2025).

These approaches demonstrably improve spatial understanding on short-context benchmarks. VST-7B-SFT achieves 60.6 on VSI-Bench, and Cambrian-S-3B reaches 78.8 on Appearance Order — both substantially ahead of general-purpose MLLMs. However, the paper identifies a critical bottleneck:

"[These methods] remain confined to single images or short video clips (i.e., 16 or 32 images) and cannot scale to the long-horizon video streams encountered in practical scenarios, where spatial cues are scattered across thousands of frames and must be progressively aggregated as the observer navigates through the environment."

The evidence for this bottleneck is stark in Table 3 (VSI-SUPER results). When video length extends from 10 minutes to 120 minutes, Qwen3-VL-2B drops from 35.0 on VSI-SUPER-Recall to 0.0 (out of memory), and Cambrian-S-7B drops from 38.3 to 0.0. VSI-SUPER-Count is even more dramatic: all baseline models score near zero at every duration. These models were designed for spatial reasoning, and they do it well on short clips, but their architectures cannot handle the streaming, long-horizon setting. They lack any mechanism for incremental memory accumulation — they must process all frames simultaneously, and when the frame count exceeds memory or compute limits, they simply fail.

A particularly telling example is Spatial-MLLM-4B in Table 5. This model introduces an explicit geometry encoder, which improves spatial accuracy but at catastrophic computational cost: 1698.8 TFLOPs at 128 frames (22× more than Qwen3-VL-2B at the same frame count) and 6002.1 TFLOPs at 256 frames. At 512 frames, it runs out of memory entirely on an 80 GB GPU. The very mechanism that enables spatial understanding — processing geometric features globally across all frames — makes streaming deployment impossible. This creates a seemingly fundamental tension: better spatial reasoning requires more geometric computation, but geometric computation scales poorly with frame count.

3. Test-Time Training for Language: Success but Limited Application to Vision

Test-time training (TTT) — the paradigm of updating model parameters at inference time to adapt to the current input — has shown significant success in language modeling. The paper traces a lineage from early work on fast weights (Ba et al., 2016) and test-time self-supervision (Sun et al., 2020) through modern TTT architectures that use gradient-based updates to encode contextual information into a compact memory state (Sun et al., 2024b). Recent work on TTT for long-context language modeling (TTT-E2E; Tandon et al., 2025) and few-shot learning (Akyürek et al., 2025) demonstrates that the approach supports continual weight adaptation and can outperform in-context learning for certain tasks.

The core appeal of TTT for streaming spatial intelligence is its linear complexity. Unlike self-attention, which requires $O(T^2)$ computation for $T$ tokens, TTT processes the sequence chunk by chunk, updating a fixed-size memory state that grows sublinearly with sequence length. This is precisely the property needed for unbounded video streams: the cost per new frame should be roughly constant, not quadratic in the total video length.

However, the paper identifies a significant gap:

"While TTT has achieved notable success in language modeling, its application to enhancing the visual capabilities of MLLMs has received comparatively limited attention."

The reasons for this gap are instructive. Language TTT benefits from a natural self-supervised objective: predict the next token. The fast weights learn to compress preceding context into a representation useful for this prediction. Visual-spatial understanding lacks an analogous natural objective — simply predicting the next frame's RGB pixels doesn't necessarily teach the model to retain geometrically meaningful information. Moreover, TTT for language operates on 1D token sequences; video introduces spatiotemporal structure (2D spatial grids changing over time) that pointwise token processing ignores.

Most critically, directly applying TTT to MLLMs risks catastrophic forgetting of pretrained knowledge. Standard TTT architectures replace all self-attention layers with TTT layers. For a language model trained from scratch, this is fine — the TTT layers learn the attention patterns from the ground up. But for an MLLM pretrained on massive image-text datasets, replacing all attention layers disrupts the carefully learned cross-modal alignment between visual tokens and text tokens. The model may become efficient at sequence processing but lose its ability to understand what it sees. Recovering this alignment through retraining would require enormous compute, defeating the purpose of using a pretrained backbone.

Reconciling Conflicting Needs: The Challenge This Paper Takes On

The paper's framing of the problem reveals a multi-way tension:

Spatial understanding requires 3D geometric reasoning — general-purpose MLLMs lack this, and spatial-augmented models provide it only at short contexts.
Streaming deployment requires linear-complexity processing — standard attention is quadratic and exhausts memory/compute on long videos; aggressive subsampling discards critical spatial cues.
Pretrained MLLMs possess valuable visual-semantic knowledge — but naively replacing attention layers with efficient alternatives (like TTT) disrupts cross-modal alignment and degrades fundamental visual understanding.
Spatial evidence accumulates incrementally across frames — but existing training data for spatial understanding provides sparse, local supervision (short QA pairs) that doesn't teach models to build persistent 3D memory over long streams.

The paper does not frame these as separate problems to be solved independently. The core insight — which distinguishes Spatial-TTT from prior work and gives the paper its unifying narrative — is that these tensions can be addressed simultaneously through a careful co-design of architecture, training data, and training procedure.

The TTT paradigm provides the linear-complexity streaming backbone, solving the efficiency problem. The hybrid architecture (interleaving TTT layers with standard self-attention anchor layers) preserves pretrained cross-modal alignment, solving the catastrophic forgetting problem. The spatial-predictive mechanism (depthwise 3D convolutions on Q/K/V projections) injects geometric inductive bias into the TTT updates, addressing the need for 3D reasoning. The dense scene-description dataset and progressive training strategy provide rich supervision for learning update dynamics that retain globally useful spatial evidence, closing the supervision gap.

This sets up the paper's contribution as architecturally principled and empirically grounded: each component targets a specific limitation of prior work, and the ablation study (Table 4) quantifies the degradation when any component is removed. Dropping the hybrid architecture causes the largest drop (64.4 → 53.9 average), confirming that preserving pretrained alignment is critical. Removing dense supervision (64.4 → 61.3) and the spatial-predictive mechanism (64.4 → 62.1) both cause meaningful but more moderate degradation, suggesting they are complementary enhancements on top of the core streaming architecture.

How This Paper Positions Itself

The paper situates itself at the intersection of three research threads: visual spatial intelligence (building MLLMs that understand 3D), test-time training (efficient sequence processing through online parameter updates), and streaming perception (processing unbounded visual input in real time). None of these threads alone solves the problem.

The key positioning statement is:

"Existing methods mostly focused on pre-training or post-training stages while test-time strategy for natively adapting diverse and streaming data is still not fully explored."

This draws a clear line: prior spatial intelligence work (VST, Cambrian-S, SpatialLadder, SpaceR, MindCube, Spatial-MLLM, VLM-3R, etc.) is valuable but operates in the wrong phase — it improves spatial understanding through better pretraining data or more sophisticated fine-tuning, but the model's inference mechanism remains static. Once trained, these models process each test video the same way regardless of its length or content. They have no mechanism for dynamically adapting their internal state as new observations arrive.

Spatial-TTT shifts the focus to inference-time computation as a first-class resource for spatial intelligence. The model's parameters are not fixed at deployment; instead, a subset of parameters (the fast weights) are continuously updated as the video stream progresses, functioning as an adaptive spatial memory that accumulates evidence tailored to the specific environment being observed. This makes Spatial-TTT a test-time approach to spatial understanding, analogous to how test-time compute scaling has been explored for reasoning (Snell et al., 2024) and language modeling (DeepSeek-AI et al., 2025) — but applied to the distinct challenge of visual 3D understanding.

The paper also explicitly distinguishes test-time training from test-time scaling (TTS):

"Unlike test-time scaling (TTS), which relies on sampling multiple trajectories and selecting the most promising ones with frozen model parameters at test time, test-time training continually updates model parameters during inference to adapt to diverse inputs and tasks."

This is an important conceptual distinction. Test-time scaling (as in OpenAI o1 or DeepSeek-R1) improves reasoning by generating and evaluating multiple solution paths at inference time — the parameters never change, only the sampling strategy does. Test-time training, by contrast, changes the model's behavior by updating its internal state to reflect the specific input being processed. For spatial intelligence, this means building a persistent 3D representation of a specific room or environment that wasn't seen during training — the model literally learns about the test environment online.

The paper also positions itself as building on the LaCT framework (Zhang et al., 2025), which demonstrated that large-chunk TTT updates with Muon optimization are practical for visual tasks including novel view synthesis and autoregressive video diffusion. Spatial-TTT extends this to the MLLM setting, where the challenge is not just efficient processing but also preserving the pretrained model's understanding of visual-semantic relationships — hence the need for the hybrid architecture that LaCT did not address.

3. Technical Approach

3.1 Reader Orientation

Spatial-TTT is a streaming video understanding system built on top of a pretrained multimodal large language model (MLLM) that processes arbitrarily long egocentric video streams frame by frame, maintaining a compact, learnable memory that continuously updates itself to accumulate 3D spatial evidence about the environment. The system solves the problem of spatial intelligence over unbounded video streams — a setting where standard attention-based models exhaust memory or compute budgets because their cost grows quadratically with video length — by replacing most attention layers with test-time training (TTT) layers that use gradient-based updates to encode incoming visual observations into a fixed-size set of "fast weights," achieving linear-complexity streaming while preserving the pretrained model's visual-semantic understanding through a hybrid architecture that keeps some original attention layers intact.

3.2 Big-Picture Architecture (Diagram in Words)

The Spatial-TTT system has five major components that operate in sequence on a streaming video input:

Vision Transformer (ViT) Encoder — converts raw video frames into visual tokens (vector representations) frame by frame; this is the standard frozen vision encoder from the pretrained MLLM backbone (Qwen3-VL-2B-Instruct).
Tokenization and Chunking — concatenates visual tokens with text tokens (the spatial reasoning question) and splits the resulting token sequence into large chunks of 2648 tokens each; these chunks are the basic processing units for the TTT mechanism.
Hybrid Decoder Blocks (3:1 TTT-to-Attention ratio) — the core processing pipeline, a stack of transformer decoder layers where 75% of layers use a dual-branch TTT layer (TTT + sliding window attention operating in parallel) and 25% are standard full-attention anchor layers that preserve the pretrained model's cross-modal alignment and global reasoning capability.
Spatial-Predictive Mechanism — within each TTT layer, before the TTT update, visual tokens are reshaped into a 3D spatiotemporal grid and processed by lightweight depthwise 3D convolutions on the query, key, and value projections, injecting local neighborhood context so the TTT updates capture geometric correspondence and temporal continuity rather than operating on isolated pointwise tokens.
Fast-Weight Memory (TTT State) — a set of learnable parameters W (the weights of a small SwiGLU-MLP network) that are updated via gradient descent after each chunk is processed, using the chunk's key-value pairs as a self-supervised reconstruction target; these fast weights act as a compact, adaptive memory that accumulates 3D spatial evidence across the entire video stream.

Information flows as follows: video frames enter the frozen ViT → visual tokens are concatenated with text tokens and split into chunks → each chunk passes through the hybrid decoder stack → within TTT layers, the spatial-predictive mechanism enhances Q/K/V with local spatiotemporal context → the TTT branch computes an apply step (using current fast weights to process the chunk) followed by an update step (using the chunk's key-value pairs to update fast weights via a gradient step), while the sliding-window attention branch provides intra-chunk context in parallel → outputs from both branches are summed → every fourth layer is a full-attention anchor layer that operates over the entire accumulated context → the final layer output is detokenized to produce the answer.

3.3 Roadmap for the Deep Dive

First, the TTT preliminary (Section 3.1 of the paper): the formal update-apply mechanism, what fast weights are, and why gradient-based memory matters — since the entire architecture is built on this paradigm.
Second, the hybrid TTT architecture (Section 3.2): how TTT layers are interleaved with standard self-attention layers, the 3:1 ratio, why pure TTT would destroy pretrained knowledge, and the dual-branch design within each TTT layer (TTT + sliding window attention with shared projections).
Third, the large-chunk update strategy and the sliding window attention design: why visual streaming forces large chunk sizes (parallelism, spatial coherence), the causal constraint problem that large chunks create, and how sliding window attention solves it while sharing projections with the TTT branch.
Fourth, the spatial-predictive mechanism (Section 3.2): the pointwise projection problem, how depthwise 3D spatiotemporal convolutions inject local geometric context, Dirac initialization, and why this makes fast-weight updates more spatially coherent.
Fifth, the Muon update rule and fast-weight network (Section 3.2): the choice of SwiGLU-MLP as the memory architecture, L2 weight normalization, and why Muon's orthogonalized momentum improves stability over vanilla gradient descent for online updates.
Sixth, the dense scene-description dataset and progressive training strategy (Sections 3.3 and 3.4): why sparse spatial QA provides weak gradient signals for learning update dynamics, how dense scene walkthroughs provide rich supervision, and the two-stage training procedure with sliding window annealing.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that test-time training — online gradient-based updates to a compact neural memory — can serve as the mechanism for continually accumulating 3D spatial evidence from streaming video, provided the architecture is carefully co-designed to preserve pretrained visual knowledge, inject spatiotemporal inductive bias, and receive dense enough supervision to learn effective update dynamics.

Test-Time Training Preliminaries: The Update-Apply Mechanism

The paper builds on the Test-Time Training (TTT) paradigm introduced by Sun et al. (2024b), which can be understood as follows. In a standard transformer, every token can attend to every previous token through the self-attention mechanism, which has $ O(T^2) $ complexity for a sequence of $ T $ tokens. TTT replaces this explicit attention with a learnable memory — a small neural network $ f_W $ parameterized by weights $ W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}} $ — that is updated as each new token (or chunk of tokens) arrives. The key insight is that $ W $ can be updated via gradient descent to "memorize" the association between keys and values of past tokens, and then applied to a query to produce an output that reflects the accumulated context, all with $ O(T) $ complexity because the size of $ W $ is fixed regardless of sequence length.

The paper formalizes this as alternating between two operations at each timestep. First is the update operation, which modifies the fast weights to associate the current key-value pair:

$W_t \leftarrow W_{t-1} - \eta \nabla_W \mathcal{L}(f_{W_{t-1}}(k_t), v_t)$

where:

$ W_t $ is the fast weight matrix after processing timestep $ t $
$ W_{t-1} $ is the fast weight matrix from the previous timestep
$ \eta $ is a learning rate
$ \nabla_W \mathcal{L} $ is the gradient of a self-supervised loss $ \mathcal{L} $ with respect to $ W $, evaluated at the current key $ k_t $ and value $ v_t $
$ f_{W_{t-1}} $ is the neural network parameterized by the previous fast weights, applied to $ k_t $

What this computes: a single gradient descent step on the fast weights, where the loss measures how well the current fast-weight network can reconstruct the value $ v_t $ from the key $ k_t $. This encodes the $ (k_t, v_t) $ association into $ W_t $ by nudging the weights to reduce the reconstruction error.

Why this form: gradient descent is the natural mechanism for online learning — it incrementally updates parameters to minimize a loss on incoming data without requiring storage of all past tokens. The key-value association framing comes from the self-attention interpretation: in standard attention, a query retrieves information by computing similarity with all past keys and aggregating past values. TTT achieves a similar effect by "compressing" past key-value pairs into the weights $ W $, so that applying $ f_W $ to a query implicitly retrieves information from all previously seen tokens — but with $ O(1) $ memory per token rather than $ O(T) $.

Second is the apply operation, which uses the updated fast weights to produce an output from the current query:

$o_t = f_{W_t}(q_t)$

where:

$ o_t $ is the output at timestep $ t $, enriched with context from all preceding tokens
$ q_t $ is the current query
$ f_{W_t} $ is the fast-weight network with its newly updated parameters

What this computes: a forward pass through the neural network $ f_{W_t} $, which now contains compressed information from tokens $ 1 $ through $ t $, applied to the current query $ q_t $. The output $ o_t $ reflects not just the current token but the entire preceding context.

Why this form: applying the network to the query after updating it with the current key-value pair means that each output is informed by both the current input and all history. This is analogous to causal self-attention where each token attends to all previous tokens — but the memory mechanism is a learned compression rather than exact retrieval.

The critical property that makes TTT suitable for streaming spatial intelligence is that the fast weights $ W $ are fixed in size — they don't grow with the sequence length. Processing a new chunk requires only a forward pass and a gradient update, both of which cost $ O(\text{chunk size} \times |W|) $, where $ |W| $ is constant. For arbitrarily long videos, the per-token cost approaches a constant, making the overall complexity linear in the video length rather than quadratic.

Hybrid TTT Architecture: Preserving Pretrained Knowledge While Achieving Linear Complexity

The most straightforward way to apply TTT to an MLLM would be to replace every self-attention layer in the transformer decoder with a TTT layer. This would maximize efficiency — all layers would operate with linear complexity. However, the paper identifies a critical problem with this approach:

"Directly replacing all core attention layers with TTT layers is prone to disrupting the pretrained cross-modal alignment and visual semantics, while recovering these would require heavily retraining."

The pretrained MLLM (Qwen3-VL-2B-Instruct) has been trained on massive image-text datasets to align visual tokens with language tokens, to understand object semantics, to ground language in visual content, and to perform general-purpose reasoning. These capabilities are encoded in the weights of its self-attention layers, which have been optimized to compute specific attention patterns between visual and textual tokens. If all of these layers are replaced with TTT layers — which have fundamentally different computation dynamics (gradient-based memory updates versus dot-product attention) — the careful alignment is disrupted. Retraining from scratch to recover it would be enormously expensive and would defeat the purpose of using a pretrained backbone.

The solution: interleaved TTT and anchor layers at a 3:1 ratio. The paper designs a hybrid architecture where only 75% of decoder layers are converted to TTT layers, while the remaining 25% retain standard full self-attention as "anchor layers." Specifically:

"In a transformer with L decoder layers, 75% use TTT while the remaining 25% retain standard self-attention as anchor layers."

For every group of four consecutive decoder layers, three are TTT layers and one is a full-attention anchor layer (hence the 3:1 ratio). The anchor layers maintain full attention access over the entire context — they can attend to all previous tokens without the compression inherent in TTT. This serves two critical functions:

Preserving cross-modal alignment: The anchor layers continue to perform the exact attention computations they were pretrained for, maintaining the alignment between visual and textual representations that the base model learned during its large-scale training.
Providing global reasoning capacity: While TTT layers compress long-range dependencies into fast weights (which is efficient but lossy), the anchor layers provide exact, lossless attention over the full context at regular intervals. This means the model can still perform global reasoning — linking information from early in the video to late in the video — without relying solely on the compressed TTT memory.

The TTT layers, meanwhile, handle the compression of long-range temporal dependencies into fast weights. Because they process the sequence chunk by chunk with fixed-size memory, they achieve sublinear memory growth — the memory footprint grows much more slowly than the sequence length, enabling the model to process videos that would be impossible for a pure attention model.

The ablation study (Table 4) provides quantitative evidence for the necessity of this hybrid design. Removing the anchor layers entirely (the "w/o Hybrid Arch" setting) causes the largest performance drop of any ablation: the VSI-Bench average falls from 64.4 to 53.9, with the Multiple-Choice subset dropping from 64.8 to 52.4. This confirms that the anchor layers are not just a minor refinement — they are essential for maintaining the pretrained model's fundamental visual understanding capabilities.

How the hybrid layers are initialized. A key practical detail: the paper does not initialize the TTT layers' query, key, and value (QKV) projection matrices from scratch. Instead:

"To preserve pretrained knowledge and facilitate convergence, we do not initialize new QKV projection matrices from scratch and make the TTT layers share the original attention layer's QKV projection matrix."

Each TTT layer inherits the QKV projection from the original attention layer it replaces. This means the input projections — which determine what information is extracted from each token to serve as query, key, and value — are initialized identically to the pretrained model. This dramatically reduces the amount of retraining needed, since the projection space is already well-aligned with the model's learned representations.

To give the TTT layers additional expressivity beyond the inherited projections, the paper introduces lightweight learnable scale and shift parameters applied to the projected queries and keys specifically for the TTT branch:

$\tilde{q} = \gamma_q \odot q + \beta_q, \quad \tilde{k} = \gamma_k \odot k + \beta_k$

where $ \gamma_q, \gamma_k $ are learnable scale vectors initialized to ones, $ \beta_q, \beta_k $ are learnable shift vectors initialized to zeros, and $ \odot $ denotes element-wise multiplication. At initialization, these parameters have no effect (scale of 1, shift of 0), so the TTT branch initially behaves identically to the original attention layer. Over training, they learn to adjust the Q and K representations specifically for the TTT update dynamics, which may benefit from different feature emphasis than standard attention.

Additionally, the output gating projection (which controls how much the TTT branch contributes to the final layer output) is initialized to zero. This means that at the very start of training, the TTT branch contributes nothing — the layer output comes entirely from the sliding window attention branch (and the anchor layers for every fourth layer). As training progresses, the gating learns to blend the two branches optimally.

Large-Chunk Updates and Sliding Window Attention: Making TTT Practical for Visual Streams

The standard TTT implementation (Sun et al., 2024b) uses small chunks — typically 16 or 64 tokens — for frequent fast-weight updates. The paper identifies that this design is poorly suited for streaming visual observations for two reasons:

Poor parallelism and GPU utilization: Small chunks mean the GPU spends most of its time on update operations with tiny batch sizes, unable to exploit the massive parallelism that makes transformers efficient in practice. Each chunk requires a separate forward-backward pass through the fast-weight network, and with chunks of only 16 tokens, the GPU is severely underutilized.
Artificially broken spatial structure: Video frames have natural spatiotemporal coherence — neighboring pixels within a frame and corresponding pixels across adjacent frames form continuous 3D structures. When tokens from the same frame (or temporally adjacent frames) are split across multiple chunks, the spatial relationship between them is broken. The TTT update sees only fragments of each frame rather than the full visual context, making it harder to learn geometrically meaningful memory updates.

The solution: large chunk size aligned with video frames. Following the LaCT framework (Zhang et al., 2025), Spatial-TTT uses a large chunk size of $ b = 2648 $ tokens, which is roughly aligned with multiple video frames rather than individual frame patches:

"Inspired by LaCT, we instead adopt a large chunk size for visual tokens, roughly aligned with multiple video frames, to substantially improve parallelism and hardware efficiency and to keep spatially coherent visual content inside the same update unit."

This means each chunk contains enough visual information to represent a complete spatiotemporal neighborhood — several consecutive frames with their full spatial resolution — allowing the fast-weight update to see coherent motion, occlusion events, and spatial relationships within a single gradient step.

The causal constraint problem. Using large chunks introduces a subtle but critical issue. TTT updates must be causal: the fast weights at time $ t $ can only incorporate information from tokens up to time $ t $, not from future tokens. For a chunk of size $ b $, this means that when processing tokens within the chunk, the model cannot allow earlier tokens in the chunk to access later tokens in the same chunk during the fast-weight update — otherwise information would leak forward in time, violating causality.

In standard small-chunk TTT, this constraint is handled trivially: the chunk is small enough that the lack of intra-chunk interaction doesn't matter much. But with $ b = 2648 $, prohibiting all intra-chunk token interactions would mean that tokens within the same few video frames cannot attend to each other at all — which is disastrous for spatial understanding, since understanding a single frame requires integrating information across all its spatial positions, and understanding motion requires comparing temporally adjacent frames.

The solution: parallel sliding window attention (SWA) within TTT layers. The paper introduces a dual-branch design within each TTT layer:

"We incorporate sliding window attention (SWA) within each TTT layer, operating in parallel with TTT, sharing query, key, and value projections."

The layer output combines both branches:

$o_t = \text{WindowAttn}(q_t, K_{[t-w:t]}, V_{[t-w:t]}) + f_{W_t}(q_t)$

where:

$ \text{WindowAttn} $ is sliding window attention over the window $ [t-w, t] $
$ w $ is the window size
$ K_{[t-w:t]} $ and $ V_{[t-w:t]} $ are the keys and values of tokens within the sliding window
$ f_{W_t}(q_t) $ is the TTT branch output

What this computes: for each token $ t $, the layer output is the sum of (1) standard sliding window attention over the most recent $ w $ tokens, providing exact local context, and (2) the TTT fast-weight network output, providing compressed global context from all tokens before the current chunk. The two branches run in parallel and are summed.

Why the window size is tied to the chunk size. The paper explicitly states:

"The window size w is set to be no smaller than the chunk size b, so that the SWA fully covers the causal lower-triangular attention matrix within each chunk, ensuring the completeness of the causal structure."

If $ w \geq b $, then the sliding window attention can attend to ALL tokens within the current chunk (since the chunk size $ b $ is smaller than or equal to the window). This means intra-chunk interactions — which the TTT branch cannot provide due to the causal constraint on updates — are fully handled by the SWA branch. The TTT branch, meanwhile, is responsible for cross-chunk information: compressing tokens from previous chunks into the fast weights and applying that compressed memory to the current chunk.

The division of labor is elegant: SWA handles local, fine-grained spatial context within a chunk; TTT handles global, compressed temporal context across chunks. Together they provide both the spatial precision needed for geometric reasoning and the temporal scope needed for long-horizon streaming.

Why SWA shares QKV projections with TTT rather than using separate projections. The paper specifies that both branches share the same query, key, and value projections (the ones inherited from the pretrained attention layer), with only the TTT branch receiving the learnable scale and shift described above. This design choice keeps the parameter count manageable — the TTT layer does not double the QKV projection matrices — and ensures that both branches operate in the same representational space, making their outputs additive without requiring additional alignment layers.

Memory management at inference time: dual KV caches. For streaming deployment, the paper describes a constant-memory mechanism using two separate key-value caches:

Sliding window KV cache: maintains keys and values for the most recent $ w $ tokens (where $ w $ equals the chunk size $ b $ at inference). When a new token arrives, it is appended to this cache; when the cache exceeds the window size, the earliest entries are discarded. This cache has fixed size $ w $, independent of total video length.
TTT pending KV cache: accumulates key-value pairs specifically for fast-weight updates. It starts empty and grows as new tokens arrive. When its length reaches the chunk size $ b $, the accumulated KV pairs are used to perform one fast-weight update (a single gradient step), after which the pending cache is cleared. This cache has a maximum size of $ b $ and costs nothing between updates.

This dual-cache design ensures that the total memory usage at inference time is bounded by $ O(w + b) $, which is constant with respect to video length. For arbitrarily long videos — hours of footage — the memory footprint never grows, in stark contrast to standard attention where the KV cache grows linearly with total tokens, eventually exhausting GPU memory (as demonstrated in Table 3 where Qwen3-VL-2B and Cambrian-S run out of memory on longer videos).

Spatial-Predictive Mechanism: Injecting Geometric Inductive Bias into TTT Updates

The TTT mechanism, as described so far, processes tokens independently through pointwise linear projections: $ k_t = W_k x_t $, $ q_t = W_q x_t $, $ v_t = W_v x_t $, where each token $ x_t $ is projected to key, query, and value vectors without any reference to its spatial or temporal neighbors. For 1D language sequences, this is natural — tokens in text have sequential but not spatial relationships. For video, however, this is a significant limitation:

"Adjacent visual tokens describe a progressively unfolding 3D scene with strong geometric and temporal continuity. However, in conventional TTT designs, Q, K, V are generated through point-wise linear projections. . . which ignore spatial-temporal structure among visual tokens."

The paper articulates the problem clearly. Visual tokens from a video are not independent — they form a 3D grid (2D spatial + 1D temporal) with strong local correlations. A token at position $ (t, h, w) $ is highly correlated with tokens at $ (t, h+1, w) $ (the pixel below it in the same frame), $ (t+1, h, w) $ (the same pixel in the next frame), and $ (t, h, w+1) $ (the pixel to the right in the same frame). These correlations carry geometric information — surfaces, edges, motion, depth — that is critical for spatial understanding.

When Q, K, V are computed through pointwise projections, each token's key and value are functions only of that token's own features. The fast-weight update then learns to associate individual keys with individual values, treating each token as an isolated observation. To learn spatial structure from this, the model would need to figure out, through gradient descent, that tokens at adjacent positions tend to have similar keys and values — it would need to rediscover spatial smoothness from scratch within the TTT update dynamics.

The paper's spatial-predictive mechanism addresses this by injecting spatiotemporal inductive bias directly into the Q, K, V projections of the TTT branch, so that each token's key, query, and value already incorporate information from its local 3D neighborhood before the TTT update or apply operations occur.

The mechanism: depthwise 3D spatiotemporal convolution on Q, K, V. For visual tokens from videos, the paper reshapes them from a flat token sequence into a 3D spatiotemporal grid with dimensions $ (T, H, W) $, where $ T $ is the number of frames in the chunk, and $ H, W $ are the spatial dimensions of the feature map (the number of visual tokens per frame in height and width). The spatial-predictive mechanism then applies a depthwise 3D convolution to the Q, K, and V projections within this grid.

Formally, for the $ i $-th channel of a token at spatiotemporal position $ (t, h, w) $, the conv-enhanced query, key, and value are computed as:

$\tilde{x}_{t, h, w}^{i} = \sum_{\delta \in \mathcal{N}} \theta_{\delta}^{i} \cdot x_{t+\delta_t, h+\delta_h, w+\delta_w}^{i}, \quad x \in \{q, k, v\}$

where:

$ \tilde{x}_{t, h, w}^{i} $ is the enhanced feature at channel $ i $ and position $ (t, h, w) $
$ \mathcal{N} = \{-\lfloor \kappa/2 \rfloor, \ldots, \lfloor \kappa/2 \rfloor\}^3 $ is the local 3D neighborhood of size $ \kappa \times \kappa \times \kappa $, where $ \kappa = 3 $ is the kernel size (specified in Section 4.1)
$ \delta = (\delta_t, \delta_h, \delta_w) $ iterates over all relative offsets within the neighborhood
$ \theta_{\delta}^{i} $ is the learnable kernel weight for offset $ \delta $ and channel $ i $
$ x_{t+\delta_t, h+\delta_h, w+\delta_w}^{i} $ is the original projected feature at the neighboring position

What this computes: for each token at position $ (t, h, w) $, the enhanced Q/K/V feature at each channel is a weighted sum of the same channel's values at all neighboring spatiotemporal positions within a $ 3 \times 3 \times 3 $ kernel. This means a token's key now reflects not just its own content but also the content of tokens that are spatially adjacent (same frame, nearby pixels) and temporally adjacent (same pixel, previous/next frames). The operation is depthwise — each channel is convolved independently, so the number of parameters is $ \kappa^3 \times d $ rather than $ \kappa^3 \times d \times d $, keeping the parameter count manageable.

Why depthwise and not full convolution: depthwise convolution applies a separate spatial kernel to each channel without mixing channels. This preserves the channel-wise semantics learned by the pretrained QKV projections — channel $ i $ in the enhanced query still corresponds to the same semantic feature as channel $ i $ in the original query, just with local spatial smoothing. A full convolution that mixes channels would destroy the pretrained feature space and require learning entirely new representations.

Dirac initialization. A crucial design detail is how the 3D convolution kernels are initialized:

"$ \theta $ are learnable kernel weights initialized with Dirac delta to preserve identity mapping at initialization."

The Dirac delta initialization means that at the start of training, the kernel weight at offset $ \delta = (0, 0, 0) $ (the center position) is 1, and all other offsets are 0. Consequently, $ \tilde{x}_{t,h,w}^{i} = x_{t,h,w}^{i} $ at initialization — the spatial-predictive mechanism has no effect, and the enhanced Q/K/V are identical to the original pointwise projections.

Why Dirac initialization: This ensures that the spatial-predictive mechanism does not disrupt the pretrained behavior at the start of training. The model initially behaves exactly as if there were no 3D convolutions, which is important because the TTT layers are being trained from scratch while the anchor layers are already pretrained. The model can then gradually learn to use the local spatial context by shifting the kernel weights away from the Dirac delta during training, incorporating neighborhood information only to the extent that it helps the TTT update dynamics.

Why kernel size 3: a $ 3 \times 3 \times 3 $ kernel covers the immediate 26 neighbors of each token in 3D — the tokens directly adjacent in space and time. This is the smallest kernel that captures first-order spatial derivatives (edges, gradients) and temporal derivatives (motion, optical flow). A $ 1 \times 1 \times 1 $ kernel would be equivalent to no spatial context; a $ 5 \times 5 \times 5 $ kernel would provide wider context but at the cost of 125× the parameters and computation compared to $ 3 \times 3 \times 3 $. The paper finds $ 3 \times 3 \times 3 $ sufficient to capture geometric correspondence and temporal continuity.

The effect on TTT learning dynamics. With the spatial-predictive mechanism, the fast-weight update becomes:

$\nabla_W \mathcal{L}(f_{W_{t-1}}(\tilde{k}_t), \tilde{v}_t)$

where $ \tilde{k}_t $ and $ \tilde{v}_t $ are the spatially-enhanced key and value. The fast weights are no longer learning to predict isolated token values from isolated token keys. Instead, they learn a predictive mapping between spatiotemporal contexts: the key $ \tilde{k}_t $ represents a local 3D neighborhood around position $ t $, and the value $ \tilde{v}_t $ represents the corresponding neighborhood in the value space. The fast-weight update effectively learns to encode associations between 3D visual contexts, which is a much more natural objective for building geometric memory — surfaces, object boundaries, and motion patterns manifest as consistent relationships between neighboring tokens across space and time.

The ablation study quantifies the benefit: removing the spatial-predictive mechanism (replacing 3D convolutions with identity projections, the "w/o SP-Mechanism" setting in Table 4) drops the VSI-Bench average from 64.4 to 62.1, with a larger impact on Numerical questions (64.0 to 60.7) than on Multiple-Choice (64.8 to 63.4). This pattern makes sense: numerical questions (absolute distance, room size, object size) require precise metric reasoning that depends on fine-grained geometric structure, which the spatial-predictive mechanism particularly helps capture.

Fast-Weight Network Design and the Muon Update Rule

The fast-weight network $ f_W $ is the neural memory that compresses the entire video history into a fixed-size parameter set. The paper chooses a bias-free SwiGLU-MLP architecture for this network:

$f_W(\mathbf{x}) = W_2 \left[ \text{SiLU}(W_1 \mathbf{x}) \odot (W_3 \mathbf{x}) \right]$

where:

$ \mathbf{x} $ is the input (a key $ k_t $ during update, or a query $ q_t $ during apply)
$ W_1, W_3 \in \mathbb{R}^{d_{\text{hidden}} \times d_{\text{in}}} $ are the first-layer weight matrices (two separate projections)
$ W_2 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{hidden}}} $ is the second-layer weight matrix
$ \text{SiLU} $ is the Sigmoid Linear Unit activation function
$ \odot $ denotes element-wise multiplication
The fast weights $ W = \{W_1, W_2, W_3\} $ are ALL trainable and updated at test time

What this computes: the input $ \mathbf{x} $ is projected into two separate hidden representations. One passes through the SiLU nonlinearity; the other remains linear. These two are multiplied element-wise — a gating mechanism where the linear path provides a signal and the SiLU path provides a learned gate that controls how much of that signal passes through. The gated output is then projected by $ W_2 $ to produce the final output. This is the SwiGLU architecture, which has been shown to be more expressive than standard ReLU-MLPs for the same parameter count.

Why SwiGLU-MLP: the gating mechanism ($ \text{SiLU}(W_1 \mathbf{x}) \odot (W_3 \mathbf{x}) $) allows the network to learn more complex nonlinear functions than a standard two-layer MLP because the multiplicative interaction between the two paths enables the network to represent polynomial-like relationships between input features. For spatial memory, this expressivity matters: the relationship between a visual key (representing a local 3D neighborhood) and a visual value (representing the corresponding geometric features) may be complex and nonlinear, and the SwiGLU architecture provides more capacity to model it.

Why bias-free: omitting bias terms simplifies the optimization, especially when the weights are normalized (see below). Bias terms would add degrees of freedom that the L2 normalization constraint doesn't regularize, potentially leading to unstable updates.

The Muon update rule. The paper does not use vanilla gradient descent (Equation 1) for updating the fast weights. Instead, it adopts the Muon optimizer (Jordan et al., 2024), which was shown in LaCT (Zhang et al., 2025) to improve stability and convergence for TTT:

$G_t = \text{MuonUpdate}(G_{t-1}, \nabla_W \mathcal{L}(f_{W_{t-1}}(\tilde{k}_t), \tilde{v}_t))$

$W_t \leftarrow \text{L2Norm}(W_{t-1} - \eta G_t)$

where:

$ G_t $ is the orthogonalized gradient with momentum — a running accumulation of gradient information that has been orthogonalized to improve conditioning
$ \text{MuonUpdate}(\cdot, \cdot) $ accumulates the current gradient into the momentum buffer and then orthogonalizes using Newton-Schulz iterations — an iterative algorithm that projects a matrix onto the space of orthogonal matrices
$ \eta $ is the learning rate
$ \text{L2Norm} $ normalizes the updated weights while preserving their original magnitude: $ \text{L2Norm}(M) = M \cdot \frac{\|M\|_{\text{orig}}}{\|M\|_2} $

What this computes step by step:

The gradient $ \nabla_W \mathcal{L} $ of the reconstruction loss with respect to the fast weights is computed (same as in standard TTT).
This gradient is accumulated into the momentum buffer $ G_{t-1} $, which maintains an exponentially weighted moving average of past gradients (like standard momentum). However, instead of using this momentum directly, Muon applies Newton-Schulz iterations to project the accumulated gradient onto the set of orthogonal matrices — matrices whose rows (and columns) are mutually orthogonal and have unit norm.
The orthogonalized momentum $ G_t $ is used to update the weights: $ W_{t-1} - \eta G_t $.
The updated weights are then L2-normalized to preserve their original magnitude. This means the update can change the direction of the weight matrix but not its overall scale.

Why Muon over vanilla gradient descent: the key issue with online gradient descent for TTT is poor conditioning. The fast-weight network's loss landscape can have directions with vastly different curvature — some weight directions strongly affect the loss (sharp directions), others barely affect it (flat directions). Standard gradient descent oscillates in sharp directions and makes slow progress in flat directions. Momentum helps but doesn't solve the conditioning problem. Muon's orthogonalization step normalizes the gradient directions to have equal magnitude (unit norm) and to be mutually decorrelated (orthogonal), which means the update makes uniform progress in all weight directions regardless of the local curvature. This is particularly important for TTT because each update is a single gradient step — there's no opportunity for the optimizer to adapt its step sizes over many iterations as in standard training. The L2 normalization after the update further stabilizes things by preventing the weight magnitudes from drifting over many updates.

Why this matters for spatial memory: in streaming spatial understanding, the fast weights are updated hundreds or thousands of times as the video progresses. Without careful optimization, early updates can be forgotten (the weights drift to encode only recent information) or the updates can become unstable (small gradient noise gets amplified). Muon's orthogonalization and L2 normalization provide a form of implicit regularization that helps the fast weights retain a balanced representation of the entire video history rather than overfitting to the most recent chunks.

Dense Scene-Description Dataset: Bridging Sparse Spatial QA with Rich Supervision

The TTT mechanism learns through self-supervised updates at test time — the fast weights are updated to reconstruct values from keys, which doesn't require external labels. However, the parameters that govern how the TTT updates work — the QKV projection matrices (shared with the pretrained model), the spatial-predictive convolution kernels, the scale and shift parameters, and the gating projections — are trained with supervised learning, and their training objective is the downstream spatial QA task.

The paper identifies a fundamental problem with using standard spatial QA data for this supervised training:

"Supervision in existing spatial intelligence datasets is typically sparse and local. For instance, a typical spatial QA task queries relations between two objects in a small region of the scene (often answerable from only a few frames), and the target answers are often short (e.g., a multiple-choice option or an integer)."

Consider a typical spatial QA example: "How many chairs are in this room?" The answer is a single integer. The gradient signal from this answer only supervises the model's final output — it doesn't directly teach the fast weights what spatial evidence to retain at each intermediate chunk update. The model might learn to extract the answer from the final few frames (where all chairs happen to be visible) without ever building a persistent spatial memory. The fast-weight updates would receive weak, indirect gradient signals because the loss only depends on the final answer, and the credit assignment problem across many chunk updates is extremely difficult.

The paper's analogy is instructive: sparse QA supervision provides "weak and low-coverage gradient signals for learning fast-weight update dynamics." The model has "limited incentive to construct a coherent and persistent global 3D memory over long video streams" because it can often shortcut the task — answer the question from local information in the final frames — without learning meaningful online update dynamics.

The solution: dense scene descriptions as training targets. The paper constructs a dense scene-description dataset from SceneVerse annotations (Jia et al., 2024), which provides object-centric 3D scene graphs for indoor environments. For each training example, instead of asking a targeted spatial question, the model receives a spatial video stream and is required to generate a comprehensive description of the underlying 3D scene in natural language.

The target output is formatted as a coherent scene walkthrough derived from the 3D scene graph, covering three aspects:

Global context: identifying the scene type and functional setting (e.g., "This is a living room with an open-plan kitchen area"). This encourages the fast weights to encode global semantic descriptors that go beyond local visual cues — information that persists across the entire video and helps frame all downstream reasoning.
Objects and counts: enumerating object categories and precise counts (e.g., "There are 4 chairs, 2 tables, 1 sofa, and 3 lamps"). This explicitly requires the model to accumulate instance-level evidence across time — a chair that appears in frame 50 might be the same chair seen from a different angle in frame 500, and the model must maintain a running count that doesn't double-count. This provides direct supervision for the fast-weight update dynamics to retain persistent object identity and count information.
Object relations: describing spatial layouts and pairwise relations (e.g., "The sofa is against the north wall, with a coffee table in front of it and two armchairs to its left"). This promotes encoding of geometric structure and inter-object constraints — information that requires integrating observations from multiple viewpoints and reasoning about 3D positions.

What makes this "dense": unlike a short QA answer that provides supervision for only the final output token, the dense description is a long text sequence where every generated word provides a training signal. The model must get the scene type right, the object list right, the counts right, and the relations right — all from the accumulated fast-weight memory. If the fast weights fail to retain any of this information, the description will be incorrect, and the model receives a direct gradient signal pointing to which aspect of the memory failed. This dense, multi-faceted supervision forces the fast-weight dynamics to genuinely capture global and persistent 3D representations because there's no shortcut — the model can't generate a complete scene description from the final few frames alone.

Dataset composition: the dense description dataset contains approximately 16,000 samples: 3,600 from ScanNet scenes and 12,500 from ARKitScenes scenes. These are real indoor environments with diverse layouts, room types, and object configurations. Each sample pairs a spatial video stream (frames rendered from the 3D reconstruction) with the target scene walkthrough derived from the scene graph annotations.

Why this specific supervision signal: the dense description task is carefully designed to be an auxiliary training objective, not the final evaluation task. The model is not evaluated on scene description at test time (the benchmarks are spatial QA, recall, and counting). Instead, the dense descriptions serve as a training curriculum that teaches the fast weights how to retain comprehensive spatial information. Once trained on this curriculum, the fast-weight dynamics generalize to retaining the specific spatial evidence needed for downstream QA tasks, even though those tasks have sparser supervision. The ablation study supports this: removing the dense data ("w/o Dense Data" in Table 4) causes drops across nearly all VSI-Bench categories, with particularly large drops on Room Size (65.9 → 58.8), Rel. Dir (73.0 → 66.7), and Appr. Order (77.0 → 71.8) — tasks that depend on long-horizon spatial memory.

Spatial-Aware Progressive Training Strategy

With the architecture and data designed, the paper addresses the question of how to train the model to achieve both effective fast-weight update dynamics and strong streaming spatial reasoning. The training is split into two stages, each with distinct objectives and design choices.

Stage 1: Initializing fast weights with global 3D awareness using dense scene descriptions.

The first stage trains the hybrid TTT architecture on the dense scene-description dataset. The goal is to teach the fast weights to:

"retain comprehensive scene-level information through chunk-by-chunk memory updates."

At this stage, the model learns the fundamental skill of accumulating global 3D scene structure in the fast weights. The dense descriptions provide the rich supervision needed for this: the model must remember room type, object lists, counts, and spatial configurations to generate the full walkthrough.

Sliding window annealing strategy. A key design choice in this stage is not to immediately set the sliding window size $ w $ equal to the chunk size $ b $. Instead, the paper uses a gradual schedule:

"The sliding window size $ w $ is linearly annealed from an initial value $ w_{\text{max}} $ to $ w_{\text{min}} = b $ over the first stage."

At the start of Stage 1, $ w_{\text{max}} $ is set to 5600 — a window large enough to cover the entire training sequence (32 frames at the training resolution). The sliding window attention essentially sees the full sequence. As training progresses, $ w $ is linearly reduced to $ w_{\text{min}} = 2648 $ (equal to the chunk size) over the first two epochs.

Why annealing: this schedule serves as a curriculum. At the beginning, the SWA branch provides comprehensive local context (seeing the entire video), so the TTT branch doesn't need to carry much weight. The fast weights can learn gradually, taking over more responsibility as the window shrinks. By the end of Stage 1, $ w = b $, meaning the SWA can only see within the current chunk — the TTT branch must now handle all cross-chunk information propagation. The fast weights have been progressively trained to encode global 3D scene structure through this handoff process:

"As the window size gradually decreases, TTT layers are forced to take over more responsibility for cross-chunk information propagation while fast weights progressively learn to encode global 3D scene structure during memory updates."

This gradual transfer of responsibility is critical because asking the TTT layers to handle cross-chunk propagation from the very start — when the fast weights are randomly initialized — would result in poor training signal and slow convergence. The sliding window provides a scaffold that is slowly removed.

Training details for Stage 1: 32 frames are uniformly sampled from each video for training. The chunk size $ b = 2648 $ is fixed throughout. The training runs for two epochs on the 16K-sample dense scene-description dataset.

Stage 2: Tuning streaming spatial reasoning with large-scale spatial VQA.

The second stage fine-tunes the model on a large-scale spatial QA dataset to specialize the fast-weight dynamics for downstream spatial reasoning tasks:

"Spatial understanding requires not only 'remembering' spatial information but also effectively 'recalling and reasoning' about spatial relations during streaming observations."

Stage 1 taught the model to remember — to accumulate comprehensive 3D evidence in the fast weights. Stage 2 teaches the model to recall and reason — to selectively retrieve task-relevant spatial evidence from the fast-weight memory and apply it to answer specific spatial questions.

The training data for Stage 2 consists of approximately 3 million spatial QA samples, including:

~2.5M open-sourced spatial data from VSI-590K (Yang et al., 2025c), VLM-3R (Fan et al., 2025), InternSpatial (Deng et al., 2025), and ViCA (Feng, 2025)
~0.5M self-collected data generated from ScanNet and ScanNet++ reconstructions, with programmatically generated spatial questions covering diverse tasks: object relative direction/distance estimation, spatial counting, route planning, room size estimation, and appearance order reasoning

Fixed window size in Stage 2. In contrast to Stage 1's annealing, Stage 2 fixes the window size and chunk size at the same value: $ w = b = 2648 $. The TTT layers are now fully responsible for cross-chunk spatial information aggregation, with the SWA providing only intra-chunk context:

"In this stage, the window size and chunk size are fixed at the same value ($ w = b $), so TTT layers are fully focused on cross-chunk spatial information aggregation."

The number of frames is increased from 32 (in Stage 1) to 64–128 frames for continuous finetuning, matching the longer video lengths seen at test time (VSI-Bench videos are typically several minutes long; VSI-SUPER videos extend to 120 minutes). Training on longer sequences in Stage 2 ensures the fast-weight dynamics are optimized for the streaming scale required by the benchmarks.

Training hyperparameters across both stages. The paper specifies a differential learning rate strategy:

"We use a learning rate of 1e-6 for the pretrained backbone and 1e-5 for the newly introduced TTT-related parameters."

The pretrained backbone — inherited from Qwen3-VL-2B-Instruct — uses a lower learning rate ($ 1 \times 10^{-6} $) to preserve its learned visual-semantic knowledge. The newly introduced TTT parameters — the fast-weight network weights, the spatial-predictive convolution kernels, the scale and shift parameters, and the output gating — use a higher learning rate ($ 1 \times 10^{-5} $) to learn quickly from scratch. This 10× ratio is standard practice when fine-tuning a pretrained model with new architectural components: the pretrained components need only subtle adjustments, while the new components start from random or Dirac initialization and need larger updates.

Both stages use a warmup of 1,000 steps (linearly increasing the learning rate from zero to its target value) and a cosine learning rate scheduler (decaying the learning rate following a cosine curve from the target value toward zero over the remainder of training). The warmup prevents the randomly initialized TTT components from producing large, destabilizing gradients early in training; the cosine decay ensures the model converges smoothly.

What the two-stage strategy achieves holistically. Stage 1 (dense descriptions) builds a foundation of general-purpose spatial memory — the fast weights learn to encode everything about the scene because they must reproduce it all in the description. Stage 2 (spatial QA) adapts this general memory to be task-selective — the fast weights learn to retain specifically the spatial evidence that matters for answering common spatial questions. The progression mirrors how one might teach a human: first learn to observe and remember a space comprehensively, then learn to use that memory to answer specific questions about the space. Without Stage 1, the model might learn brittle shortcuts in Stage 2 (e.g., answering from the last few frames). Without Stage 2, the model would have rich spatial memory but wouldn't know which aspects of it are relevant for typical spatial reasoning tasks.

Summary of Design Choices and Their Justifications

Hybrid 3:1 TTT-to-attention architecture over pure TTT: preserves pretrained cross-modal alignment and global reasoning; ablation shows removing anchor layers causes the largest performance drop (64.4 → 53.9 on VSI-Bench).
Large chunk size (2648 tokens) over small chunks (16–64 tokens): dramatically improves GPU parallelism and hardware efficiency, and keeps spatially coherent visual content within the same update unit, preventing artificial breaking of frame structure.
Sliding window attention sharing QKV projections with TTT branch over independent projections: provides intra-chunk spatial context that TTT cannot supply due to causal update constraints, without doubling parameters; the two branches complement each other (SWA for local, TTT for global) and their outputs simply sum.
Deptwise 3D spatiotemporal convolution (kernel size 3×3×3) with Dirac initialization over pointwise projections: injects local geometric inductive bias into the TTT update so fast weights learn predictive mappings between spatiotemporal contexts rather than isolated tokens; Dirac initialization ensures the mechanism doesn't disrupt pretrained behavior at training start.
Bias-free SwiGLU-MLP as fast-weight network over simpler MLP architectures: the multiplicative gating mechanism provides greater expressivity for modeling complex nonlinear relationships between visual keys and values, while bias-free design simplifies regularization under L2 normalization.
Muon optimizer with Newton-Schulz orthogonalization and L2 weight normalization over vanilla gradient descent: addresses poor conditioning in online gradient-based updates, enabling stable and uniform progress across all weight directions; critical for long streaming sequences where the fast weights undergo hundreds of updates.
Dense scene-description dataset (16K samples) over using only sparse spatial QA for training: provides rich, multi-faceted supervision that teaches fast-weight dynamics to retain comprehensive 3D scene information; ablation shows removing dense data drops performance across nearly all VSI-Bench categories.
Two-stage training with sliding window annealing over single-stage fine-tuning: Stage 1 (dense descriptions, w annealed from 5600 to 2648) teaches the fast weights to accumulate global 3D evidence, with the sliding window providing a gradually removed scaffold; Stage 2 (spatial QA, w fixed at 2648) teaches task-selective retrieval and reasoning from the accumulated memory.
Differential learning rates (1e-6 backbone, 1e-5 TTT components) over uniform learning rates: preserves pretrained visual-semantic knowledge while allowing new TTT components to learn quickly from scratch.
Dual KV cache at inference (fixed-size SWA cache + chunk-pending TTT cache) over storing all past keys and values: enables constant-memory streaming regardless of video length; the SWA cache is bounded by window size $ w $, and the TTT pending cache is bounded by chunk size $ b $, both fixed.

4. Key Insights and Innovations

Innovation 1: Test-Time Training as a Streaming Spatial Memory Mechanism — Not Just an Efficiency Hack

The most intellectually distinctive move in this paper is reframing test-time training (TTT) from a sequence-processing efficiency technique into a mechanism for continual spatial evidence accumulation. This is a conceptual leap, not an incremental architectural tweak.

Prior work on TTT — from Sun et al. (2024b) through LaCT (Zhang et al., 2025) — treated it primarily as a way to achieve linear-complexity sequence modeling. The fast weights were understood as a compression device: they encode past context so the model doesn't need to store every token's key-value pair. The evaluation was correspondingly about language modeling perplexity and long-context retrieval accuracy — metrics that measure whether information was preserved, not whether it was organized into a structured representation useful for downstream reasoning.

Spatial-TTT shifts this framing fundamentally. The fast weights are not merely a compressed memory — they are a learned 3D spatial state that accumulates geometric evidence as the camera moves through an environment. The paper argues this explicitly: the fast weights "act as a compact non-linear memory to accumulate 3D evidence from unbounded video streams." The word "accumulate" is doing heavy conceptual work here. It implies that the memory isn't just storing raw visual features — it's building up a representation of the underlying 3D structure that persists even as individual objects leave the field of view, viewpoints change, and occlusions occur.

What makes this framing novel: it reinterprets the update-apply mechanism (Equation 1) not as a generic compression operation but as a continual perception process. Each chunk update becomes analogous to an observer integrating a new visual glimpse into their mental map of a room. The apply operation becomes analogous to querying that mental map to answer a spatial question. This is not how prior TTT papers conceptualized their systems, and it leads to design decisions — particularly the spatial-predictive mechanism and dense scene-description supervision — that make no sense under the pure-efficiency framing.

Why this matters beyond performance: it opens up TTT as a paradigm for online learning in embodied settings. A robot navigating a building doesn't just need efficient sequence processing — it needs to build and maintain a persistent spatial representation of its environment from streaming sensor data. The TTT-as-memory framing positions the fast weights as the computational substrate for this representation, connecting the TTT literature to the broader agenda of embodied spatial intelligence in a way that prior efficiency-focused work did not.

Evidence anchoring the claim: the VSI-SUPER-Count results (Table 3) are the clearest demonstration that this is memory, not just compression. General-purpose MLLMs and long-video understanding models (MovieChat, Flash-VStream) collapse to near-zero accuracy on counting tasks beyond 10 minutes. They can't maintain a running count of objects across extended streams because they weren't designed to accumulate evidence — they either process all frames at once (and run out of memory) or subsample aggressively (and miss objects). Spatial-TTT maintains 31.8–45.6% counting accuracy from 10 to 120 minutes because its fast weights genuinely accumulate spatial evidence: each new chunk updates the count, and the memory state persists. This is qualitatively different behavior — not just better scaling, but a different kind of capability.

Innovation 2: The Hybrid Architecture as a Solution to the Pretrained-Knowledge Catastrophic Forgetting Problem in TTT for MLLMs

The paper identifies and solves a problem that prior TTT work didn't need to address: how to convert a pretrained multimodal model to TTT without destroying its visual-semantic understanding. This is a diagnostic contribution — naming a failure mode that was implicit but unarticulated in prior work and providing a principled solution.

Prior TTT systems (Sun et al., 2024b; Zhang et al., 2025; Tandon et al., 2025) were trained from scratch or fine-tuned on language-only tasks. When you train from scratch, there's no pretrained knowledge to preserve — the TTT layers and the rest of the model co-evolve. When you fine-tune a language model, the semantic knowledge is embedded in the token representations, and TTT layers learn to process these representations — the cross-modal alignment problem doesn't arise because there's only one modality.

Spatial-TTT faces a fundamentally different challenge. The base model (Qwen3-VL-2B-Instruct) was trained on massive image-text pairs to align visual tokens with language tokens. Its attention layers compute specific cross-modal attention patterns — visual tokens attend to text tokens, text tokens attend to visual tokens, and these patterns encode the grounding of language in visual content. Replacing all attention layers with TTT layers would destroy these patterns. As the paper notes, "recovering these would require heavily retraining" — essentially retraining the multimodal alignment from scratch, which defeats the purpose of using a pretrained backbone.

What makes this insight non-obvious: the catastrophic forgetting problem in continual learning is well-known, but its specific manifestation in TTT-to-MLLM conversion wasn't previously characterized. The paper's diagnosis is that the issue isn't just "forgetting" in a general sense — it's the disruption of cross-modal alignment mechanisms that are encoded in the specific attention patterns of the pretrained layers. This is why simply fine-tuning with a lower learning rate (a common anti-forgetting technique) isn't sufficient — the TTT computation is structurally different from attention, so no amount of careful optimization can preserve attention patterns in layers that no longer compute attention.

The solution's conceptual significance: interleaving TTT layers with full-attention anchor layers at a 3:1 ratio is an elegant architectural answer to this problem. The anchor layers preserve exact attention computation at regular intervals, maintaining the cross-modal pathways that the pretrained model relies on. The TTT layers can then focus on what they're good at — compressing long-range temporal dependencies — without needing to also shoulder the burden of cross-modal alignment. The 3:1 ratio is not arbitrary; it provides a dense enough spacing of anchor layers (every fourth layer) that cross-modal information can propagate through the network without being bottlenecked.

Evidence anchoring the claim: the ablation study (Table 4) is definitive here. Removing the hybrid architecture — converting to pure TTT — causes the largest performance drop of any single ablation: the VSI-Bench average falls from 64.4 to 53.9, with the Multiple-Choice subset collapsing from 64.8 to 52.4. This 10.5-point drop is nearly double the degradation from removing the spatial-predictive mechanism (2.3 points) or dense supervision (3.1 points). The magnitude of the drop and its concentration on tasks requiring general visual understanding (multiple-choice questions that test object recognition, scene interpretation, and semantic reasoning — not just metric spatial estimation) confirms that the anchor layers are preserving something fundamental: the pretrained model's ability to understand what it sees.

Significance beyond this paper: the hybrid TTT architecture establishes a design pattern that generalizes. Any future work seeking to apply TTT to pretrained multimodal models will face the same cross-modal alignment preservation problem. The 3:1 ratio and the specific choice of full-attention anchor layers (rather than, say, linear attention anchors or sparse attention anchors) provides a tested template. The paper also implicitly defines a new axis for architecture search: the TTT-to-attention ratio as a hyperparameter controlling the tradeoff between streaming efficiency and pretrained-knowledge preservation.

Innovation 3: Difficulty-Agnostic Streaming Architecture via Complementary Local-Global Decomposition

The paper makes a subtle but powerful architectural move in how it assigns responsibilities between the TTT branch and the sliding window attention (SWA) branch within each TTT layer. Rather than treating SWA as a minor addition to patch over the causal constraint of large-chunk TTT, the design establishes a clean functional decomposition: SWA handles local, fine-grained, within-chunk spatial context; TTT handles global, compressed, cross-chunk temporal context. The two branches operate in parallel with shared Q/K/V projections and their outputs simply sum, creating a unified representation that is simultaneously locally precise and globally informed.

What makes this distinctive: Prior TTT architectures (Sun et al., 2024b; Zhang et al., 2025) treated intra-chunk and inter-chunk information uniformly — the TTT mechanism alone was responsible for all context, both local and global. This worked for language modeling because tokens in text have sequential dependencies but no spatial structure — a token doesn't need to attend to its "neighbors" in a 2D grid. For video, this uniform treatment fails because spatial understanding requires precise local geometric reasoning (is this object to the left or right of that one? how far apart are they?) that is fundamentally about relationships between tokens that are spatially adjacent or temporally proximate — exactly the relationships that TTT's compressed memory may blur or lose.

The dual-branch design solves this by explicitly separating the two types of context. SWA provides exact, lossless attention within a local window, preserving the fine-grained spatial relationships needed for geometric reasoning. TTT provides compressed, approximate memory across the entire video history, enabling the model to accumulate evidence over arbitrary durations. Neither branch alone is sufficient — SWA can't see beyond its window (and would revert to the quadratic scaling problem if the window were made large enough to cover the full video), and TTT alone loses spatial precision — but together they cover the full spectrum of spatial reasoning needs.

Why the shared Q/K/V projections matter: this isn't just a parameter-saving trick. By sharing the Q/K/V projections, both branches operate in the same representational space, which means their outputs can be simply summed without additional alignment or gating mechanisms. This creates an inductive bias: the model learns Q/K/V representations that are simultaneously useful for local attention and global compression. The features that make a good key for within-chunk spatial attention — encoding precise 3D positions, object boundaries, local geometry — are also the features that make a good key for cross-chunk TTT updates — encoding stable object identities, persistent spatial relationships, motion patterns.

Evidence anchoring the claim: the VSI-Bench results (Table 1) show Spatial-TTT excelling precisely where this local-global decomposition matters most. On Relative Direction (73.0, vs. 57.5 for the best proprietary model Gemini-3-Pro), the model must reason about fine-grained spatial relationships — "is object A to the left or right of object B from the current viewpoint?" — which requires precise local geometric understanding that the SWA branch preserves. On Appearance Order (77.0, vs. 71.0 for Seed-2.0), the model must track objects across long temporal intervals — "in what order did these four objects first appear?" — which requires the global temporal accumulation that the TTT branch provides. The simultaneous strength on both types of tasks is evidence that the decomposition is working as designed.

Significance beyond this paper: this local-global decomposition is a design principle that extends beyond TTT. Any streaming architecture for spatial understanding faces the same tension: local precision (within a temporal window) vs. global scope (across arbitrarily long durations). The paper's solution — parallel branches with shared representations and additive outputs — is a clean architectural pattern that could be instantiated with other mechanisms (e.g., replacing TTT with a different form of compressed memory while keeping the SWA branch). The key insight is that these two types of context are fundamentally different and benefit from different computational mechanisms, rather than being forced into a one-size-fits-all architecture.

Innovation 4: Dense Scene Descriptions as a Curricular Bridge from Generic Memory to Task-Specific Spatial Recall

The paper's training methodology — particularly the two-stage procedure with dense scene descriptions followed by spatial QA fine-tuning — is more than a recipe for better performance. It identifies and addresses a fundamental credit assignment problem in training recurrent memory systems: sparse, task-specific supervision (short QA answers) provides weak and indirect gradient signals for learning how to update the memory state, because the model can often answer the question from local evidence in the final frames without ever learning to accumulate information across time.

What makes this a conceptual contribution rather than just a training trick: the paper diagnoses why training TTT-based spatial models is hard, not just that a particular dataset helps. The diagnosis is that standard spatial QA data — "How many chairs are in this room? A: 4" — provides supervision only for the final output token. The gradient must propagate backward through the entire sequence of chunk updates to assign credit (or blame) to each fast-weight update for its contribution to the final answer. This is an extremely difficult credit assignment problem — it's the temporal credit assignment problem that plagues all recurrent architectures, exacerbated by the fact that the "recurrent" computation here (gradient-based fast-weight updates) is itself learned.

The dense scene-description dataset addresses this by providing per-chunk supervision implicitly. To generate a complete scene walkthrough — listing all objects, their counts, their spatial relationships — the model must have retained comprehensive spatial evidence across the entire video. There's no shortcut: the full object list can't be extracted from the final few frames; counts require non-double-counting across temporally separated observations; spatial relations require integrating viewpoints. Every word in the long-form description provides a gradient signal that pressures the fast-weight dynamics to genuinely accumulate information.

Why the two-stage structure matters conceptually: Stage 1 (dense descriptions) teaches the fast weights what to remember — everything about the scene — because the description task demands comprehensive retention. Stage 2 (spatial QA) then teaches the fast weights what matters — which aspects of the accumulated spatial memory are relevant for answering specific types of questions. This is a curriculum: first learn to build a complete spatial memory, then learn to selectively query it. Without Stage 1, the model might learn brittle shortcuts in Stage 2. Without Stage 2, the model would have rich memory but wouldn't know how to use it for spatial reasoning tasks.

The connection to the sliding window annealing: this curriculum is mirrored in the architectural training schedule. In Stage 1, the sliding window starts large (5600, covering the full training sequence) and anneals to the chunk size (2648). The SWA initially provides a scaffold — it can see the full video, so the TTT branch doesn't need to carry much weight. As the window shrinks, the TTT branch is progressively forced to take over cross-chunk information propagation. This gradual handoff ensures the fast-weight dynamics learn stably, with the SWA providing support that is slowly withdrawn. The annealing is the architectural analog of the data curriculum: both move from "easy" (comprehensive local context, comprehensive supervision) to "hard" (chunk-limited context, task-selective supervision).

Evidence anchoring the claim: the ablation (Table 4, "w/o Dense Data") shows that removing the dense description training drops the VSI-Bench average from 64.4 to 61.3. The per-category breakdown in Table 7 reveals where the impact concentrates: Room Size drops from 65.9 to 58.8, Relative Direction from 73.0 to 66.7, Appearance Order from 77.0 to 71.8. These are the tasks that depend most heavily on long-horizon spatial memory — they can't be answered from the last few frames. The fact that removing training data, not changing the architecture at test time, degrades performance on these specific tasks demonstrates that the dense supervision genuinely changed what the fast-weight dynamics learned to retain, not just provided more training examples.

Significance beyond this paper: the credit assignment diagnosis generalizes to any system that uses learned recurrent memory for long-horizon tasks. Training on sparse end-task supervision alone is likely insufficient to learn effective memory update dynamics because of the temporal credit assignment problem. The solution — using dense auxiliary objectives that demand comprehensive retention, followed by task-specific fine-tuning — is a training paradigm that could apply broadly to streaming perception, video understanding, and embodied AI systems where memory must be learned rather than hand-designed.

Innovation 5: The Spatial-Predictive Mechanism as a Minimalist Injection of Geometric Inductive Bias

The spatial-predictive mechanism — depthwise 3D spatiotemporal convolutions on Q/K/V projections with Dirac initialization — is conceptually interesting not because of its architectural novelty (3D convolutions on features are standard in video understanding) but because of what it says about the relationship between architecture and learning dynamics in TTT.

The key insight is that pointwise linear projections — the standard way to compute Q, K, V in transformers — are incompatible with effective fast-weight learning for spatial tasks. When Q, K, V are computed from isolated tokens, each TTT update learns to associate individual keys with individual values. For the fast weights to capture spatial structure — edges, surfaces, motion, depth — they must discover through gradient descent that tokens at adjacent spatiotemporal positions have related keys and values. This is a form of implicit spatial smoothness learning that the TTT update dynamics are not well-suited for, because each update sees only one chunk at a time and the fast weights are a compressed representation with limited capacity.

The spatial-predictive mechanism solves this by injecting the spatial smoothness prior directly into the feature computation, before the TTT update ever sees the tokens. By convolving each token's Q/K/V with those of its 3D neighbors, the mechanism ensures that tokens at adjacent positions already have similar keys and values — the relationship is "baked in" rather than needing to be learned by the fast-weight dynamics. The fast weights can then focus on learning higher-level spatial relationships (object identities, persistent geometry, room layout) rather than rediscovering the basic fact that neighboring pixels in a video are related.

Why Dirac initialization is the key conceptual move: initializing the 3D convolution kernels to the Dirac delta means the spatial-predictive mechanism starts as an identity — it does nothing. All neighborhood weights are zero except the center. Over training, the kernels learn to incorporate neighborhood information only to the extent that it helps the TTT update dynamics. This is a learned inductive bias, not a hardcoded one: the model discovers through training how much local context is useful, in which directions (spatial vs. temporal), and for which feature channels. This is more flexible than hand-designing a spatial smoothing operator (e.g., a fixed Gaussian blur), and it avoids the risk of over-smoothing — the kernels can learn to preserve sharp discontinuities where they carry geometric information (object boundaries, occlusion edges).

What makes this minimalist: the spatial-predictive mechanism adds very few parameters — κ³ × d for each of Q, K, V, where κ = 3 and d is the feature dimension. For a 2B-parameter model with hidden dimension ~2048, this is roughly 27 × 2048 × 3 ≈ 166K parameters per TTT layer — negligible compared to the overall model size. Yet the ablation shows it provides a meaningful 2.3-point improvement on VSI-Bench. This is a high "conceptual return on parameter investment": a small, targeted injection of geometric structure into the feature computation yields outsized benefits because it addresses a specific weakness (the TTT dynamics' difficulty learning spatial smoothness from scratch) rather than trying to learn everything from data.

Evidence anchoring the claim: the ablation (Table 4, "w/o SP-Mechanism") shows that removing the spatial-predictive mechanism degrades numerical questions (60.7 vs. 64.0) more than multiple-choice questions (63.4 vs. 64.8). This pattern is telling: numerical questions (absolute distance, room size, object size) require precise metric spatial reasoning that depends on fine-grained geometric structure — exactly what the spatial-predictive mechanism helps the TTT updates capture. Multiple-choice questions can often be answered from coarser semantic understanding (recognizing objects and their approximate layout) where the benefit of precise geometric encoding is smaller.

Significance beyond this paper: the spatial-predictive mechanism suggests a general principle for TTT applied to structured data: inject domain-specific structure into the feature computation upstream of the TTT update, rather than expecting the update dynamics to learn it. For video, this means spatiotemporal convolutions. For other modalities — point clouds, molecular graphs, audio spectrograms — the analogous injection would use domain-appropriate local aggregation (graph convolutions, spectral filters, etc.). The Dirac initialization pattern (start as identity, learn to incorporate context) is a reusable recipe that ensures the injection doesn't disrupt pretrained representations while allowing the model to discover how much structure is useful.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on four benchmarks measuring complementary aspects of visual spatial intelligence. VSI-Bench (Yang et al., 2025a) contains over 5,000 question–answer pairs from 288 real-world indoor videos drawn from ScanNet, ScanNet++, and ARKitScenes validation splits, covering eight task categories: Object Count, Absolute Distance, Object Size, Room Size, Relative Distance, Relative Direction, Route Plan, and Appearance Order. MindCube-Tiny (Yin et al., 2025) is a diagnostic subset of MindCube with 1,050 multi-view spatial reasoning questions (600 AMONG, 250 AROUND, 200 ROTATION) testing cross-view object consistency and reasoning about occluded elements. VSI-SUPER-Recall and VSI-SUPER-Count (Yang et al., 2025c) test streaming spatial sensing over videos ranging from 10 to 120 minutes at 1 fps, where VSC requires counting distinct objects over extended sequences and VSR uses multiple-choice to probe recall of temporal object appearance order. The paper also uses a self-constructed dense scene-description dataset of ~16K samples (3.6K ScanNet + 12.5K ARKitScenes) for training, but this is not used for evaluation.

Base model(s). All Spatial-TTT variants are initialized from Qwen3-VL-2B-Instruct (Bai et al., 2025a), a 2-billion-parameter multimodal large language model. The paper argues this model is representative of contemporary MLLM capabilities while being compact enough to enable controlled experiments where the relative contribution of architectural changes can be clearly measured. The FLOPs-matched comparison is not between different model scales (unlike the example paper's pretraining vs. inference tradeoff) but rather between different architectures at comparable parameter counts — Spatial-TTT-2B versus other 2B-scale models and up to 7–8B-scale competitors.

Metrics. The paper follows the evaluation protocols of each benchmark's original publication. For VSI-Bench: Accuracy (ACC) for multiple-choice questions (Appearance Order, Relative Direction, Relative Distance, Route Plan) and Mean Relative Accuracy (MRA) for numerical questions (Object Count, Absolute Distance, Object Size, Room Size), with macro-average across all eight categories as the primary summary metric. MRA is defined to quantify how closely predictions align with ground-truth values on a normalized scale. For MindCube-Tiny: Accuracy (ACC) on each question type and micro-average across all 1,050 questions. For VSI-SUPER-Recall: Accuracy (ACC) on multiple-choice order-recall questions. For VSI-SUPER-Count: Mean Recall Accuracy (MRA) on counting questions.

Baselines. The paper constructs an unusually comprehensive comparison spanning three categories:

Proprietary multimodal models: GPT-5 (OpenAI, 2025), Gemini-2.5-pro (Comanici et al., 2025), Gemini-3-pro (Google), Seed-2.0 (Bytedance Seed, 2025), Kimi-K2.5 (Team et al., 2026), and Grok-4 (xAI, 2025). These serve as the strongest possible upper bounds from commercial systems.

Open-source general-purpose MLLMs: LLaVA-OneVision-72B (Li et al., 2024a), LLaVA-Video-72B (Lin et al., 2023), LongVA-7B (Zhang et al., 2024b), InternVL3-2B and 8B (Zhu et al., 2025), Qwen2.5-VL-3B and 7B (Bai et al., 2025b), Qwen3-VL-2B and 8B (Bai et al., 2025a). These establish the performance of general-purpose video understanding without spatial specialization.

Open-source spatial intelligence models: MindCube-3B (Yin et al., 2025), SpatialLadder-3B (Li et al., 2025a), SpaceR-7B (Ouyang et al., 2025), ViLaSR-7B (Wu et al., 2025b), VST-3B-SFT and VST-7B-SFT (Yang et al., 2025b), Cambrian-S-3B (Yang et al., 2025c), and Spatial-MLLM-4B (Wu et al., 2025a). These are the most directly comparable systems — models explicitly designed or trained for spatial intelligence, at parameter scales from 3B to 7B. For VSI-SUPER benchmarks, the paper additionally includes long-video understanding baselines: MovieChat (Song et al., 2024) and Flash-VStream (Zhang et al., 2024a). Human and Random Choice scores are reported where available from the original benchmark publications.

Generation budget. For fair comparison, the paper's primary concern is memory scaling and computational efficiency, not a formalized generation budget as in the language-model example. The relevant comparison is made in Table 5, where peak GPU memory (GB) and theoretical TFLOPs per forward pass are measured across varying input lengths (128, 256, 512, 1024 frames at 352×480 resolution) on comparable hardware. Importantly, the TFLOPs comparison accounts for the entire forward pass including any additional encoders or attention mechanisms, making it a genuine end-to-end computational cost comparison rather than a theoretical per-layer analysis. At inference time, Spatial-TTT processes videos streamingly — there's no sampling of multiple trajectories, so the "budget" is simply the duration of video the model can sustain without exceeding memory limits.

Cross-validation / statistical protocol. The paper does not employ cross-validation for model selection — the architecture and hyperparameters are fixed based on design principles and initial validation, not tuned to test benchmarks. All reported results are single evaluation runs on the standard test splits of each benchmark, ensuring comparability with the published baselines. This is standard practice for the benchmarks used, which have fixed evaluation protocols and publicly reported baseline numbers. The paper does not report confidence intervals or statistical significance tests, which is a limitation for the smaller benchmarks (MindCube-Tiny at 1,050 questions, VSI-SUPER subsets which are not reported with precise sample counts per duration).

Main Quantitative Results

VSI-Bench: General Spatial Understanding

The headline result appears in Table 1: Spatial-TTT-2B achieves a 64.4 macro-average across all eight VSI-Bench tasks, outperforming all proprietary and open-source models despite its compact 2B scale. The strongest proprietary model is Gemini-3-pro at 56.0, giving Spatial-TTT an 8.4-point margin. The strongest open-source spatial model is VST-7B-SFT at 60.6 (a 3.8-point margin), and the strongest general-purpose open-source model is Qwen3-VL-8B-Instruct at 57.9 (a 6.5-point margin). Spatial-TTT's base model, Qwen3-VL-2B-Instruct, scores 50.3 — meaning the TTT architecture alone contributes approximately 14.1 points of improvement at the same parameter scale.

The per-task breakdown reveals where Spatial-TTT's advantages concentrate. On Relative Direction (73.0), it leads the best proprietary baseline (Gemini-3-pro, 57.5) by 15.5 points and the best open-source baseline (Cambrian-S-3B, 61.9) by 11.1 points — this task requires precise viewpoint-dependent spatial reasoning about whether one object is to the left/right/front/back of another, which directly tests the ability to maintain consistent 3D geometric representations across egocentric views. On Appearance Order (77.0), it leads the best open-source baseline (Cambrian-S-3B, 78.8? — actually wait, Cambrian-S-3B scores 78.8 on this task per Table 1, so Spatial-TTT's 77.0 is 1.8 points lower; the strongest result here is actually Cambrian-S-3B at 78.8, with Spatial-TTT at 77.0 in second place among open-source models). The paper claims Spatial-TTT "exhibits strong advantages" on Route Plan (47.4 vs. best proprietary Gemini-3-pro at 61.9? — checking the table, Gemini-3-pro scores 61.9 on Route Plan, so Spatial-TTT's 47.4 is substantially lower). This needs careful examination: the paper's textual claims about strong advantages should be read against the actual numbers. On Route Plan, several proprietary models substantially outperform Spatial-TTT (Seed-2.0 at 44.3, Grok-4 at 47.4, Gemini-2.5-pro at 47.4, Gemini-3-pro at 61.9, Kimi-K2.5 at 52.1, GPT-5 at 50.2). Spatial-TTT's 47.4 is competitive but not dominant. The genuine strong advantages are on Relative Direction (73.0, best overall), Appearance Order (77.0, second only to Cambrian-S-3B at 78.8), Absolute Distance (47.8, beating all baselines including proprietary Gemini-3-pro at 42.8 and GPT-5 at 34.4), Room Size (65.9, second only to VST-7B-SFT at 68.3), and Object Count (70.8, second only to VST-7B-SFT at 72.0).

On the numerical questions, Spatial-TTT attains the best Absolute Distance (47.8 vs. 47.0 for the strongest baseline, Qwen3-VL-8B-Instruct), second-best Object Count (70.8 vs. 72.0 for VST-7B-SFT), second-best Room Size (65.9 vs. 68.3 for VST-7B-SFT), and second-best Object Size (71.7 vs. 76.3 for Qwen3-VL-8B-Instruct). The pattern suggests that Spatial-TTT's metric grounding (estimating distances, sizes, and room dimensions from visual evidence accumulated over the video stream) is on par with or exceeds models that explicitly encode depth and geometry, despite not having a dedicated geometry encoder. This is the core architectural claim playing out empirically: the streaming TTT memory provides an effective substitute for explicit geometric feature computation, accumulating enough visual evidence over the video to make precise metric estimates.

The human performance baseline in Table 1 provides valuable context: humans score 79.2 on average (with particularly strong performance on Appearance Order at 100.0 and Relative Direction at 95.8, but surprisingly weak on Object Size at 60.4 and Room Size at 45.9). Spatial-TTT's 64.4 is roughly 15 points below human-level, with the largest remaining gaps on Route Plan (47.4 vs. 95.8 human) and Relative Distance (61.8 vs. 94.7 human) — tasks requiring complex multi-step spatial planning and precise metric relative distance estimation that remain challenging even with streaming memory.

MindCube-Tiny: Fine-Grained Multi-View Spatial Reasoning

Table 2 reports results on MindCube-Tiny, which specifically tests spatial reasoning under viewpoint changes and occlusions. Spatial-TTT achieves 76.2% average accuracy, improving 12.3 points over the strongest proprietary baseline (Gemini-3-pro at 63.9%) and 24.5 points over the strongest open-source spatial model (MindCube-3B at 51.7%). The improvement over the base model Qwen3-VL-2B (34.5%) is 41.7 points — an enormous gain that underscores how much spatial reasoning capability the TTT architecture unlocks from the same pretrained backbone.

The breakdown by question type is instructive. On AROUND questions (89.8%), Spatial-TTT nearly doubles the performance of the next-best model (MindCube-3B at 67.6%), suggesting particular strength in reasoning about what surrounds an object when observed from a viewpoint that may not directly show all surrounding objects. On AMONG questions (74.0%), it leads the next-best model (Gemini-3-pro at 59.3%) by 14.7 points. On ROTATION questions (55.5%), it substantially outperforms all baselines (next best is GPT-5 at 94.5? — checking: actually GPT-5 scores 94.5 on ROTATION, which is far higher than Spatial-TTT's 55.5; looking more carefully, the ROTATION column shows GPT-5 at 94.5, Grok-4 at 93.0, Gemini-2.5-pro at 88.0, making Spatial-TTT's 55.5 substantially lower than several proprietary baselines on this specific task). The paper does not discuss this specific weakness. ROTATION questions test the ability to reason about how an object's appearance changes under rotation — a task that may depend more on learned 3D object representations (which larger models may have acquired through massive pretraining) than on streaming evidence accumulation.

A notable pattern: proprietary models show highly uneven performance across question types. GPT-5 scores 94.5 on ROTATION but only 38.2 on AMONG and 56.3 overall. Grok-4 scores 93.0 on ROTATION but 54.4 on AMONG and 63.5 overall. This suggests these models have strong object-centric 3D reasoning (mental rotation) but weak scene-level spatial reasoning (understanding relationships between multiple objects under viewpoint changes). Spatial-TTT shows the opposite profile — stronger on scene-level tasks than object-level rotation — consistent with its design as a streaming scene memory rather than an object-centric 3D reasoner.

VSI-SUPER: Streaming Spatial Sensing in Long-Horizon Video

VSI-SUPER-Recall (Table 3). Spatial-TTT scores 38.3, 35.0, 28.3, and 30.0 at 10, 30, 60, and 120 minutes respectively. These scores are competitive with baselines at shorter durations (within 3–5 points of Qwen3-VL-2B and Cambrian-S-7B at 10 and 30 minutes) but are dramatically superior at longer durations because all baselines collapse. Qwen3-VL-2B drops from 35.0 at 10 minutes to 0.0 at 120 minutes (out of memory). Cambrian-S-7B drops from 38.3 at 10 minutes to 0.0 at 120 minutes (out of memory). MovieChat and Flash-VStream maintain non-zero scores throughout (ranging from 16.7–33.3 across durations) but are consistently 5–10 points below Spatial-TTT at every interval. The key finding is not that Spatial-TTT achieves the highest absolute numbers — at 10 minutes, it ties Cambrian-S-7B at 38.3, and at 30 minutes, it's within 5 points of Cambrian-S-7B's 35.0 — but rather that its performance is essentially stable as video length increases, showing resilience that no baseline matches.

VSI-SUPER-Count (Table 3). This is where the contrast is starkest. Spatial-TTT achieves 31.8, 45.6, 36.2, and 38.4 at 10, 30, 60, and 120 minutes, while ALL baselines score near zero across ALL durations. Qwen3-VL-2B scores 0.8 at 10 minutes and 0.0 thereafter. Cambrian-S-7B scores 0.6 at 10 minutes and 0.0 thereafter. MovieChat and Flash-VStream score 0.0 at every duration. The counting task requires genuine streaming accumulation — the model must continuously update a running count of distinct objects as they appear across the video, without double-counting, and must do so over increasingly long intervals. This is precisely the capability that TTT's fast-weight memory is designed to provide, and the near-total failure of all baselines (including those designed for spatial understanding) demonstrates that standard architectures fundamentally cannot perform this type of continual evidence accumulation.

It's notable that Spatial-TTT's counting performance is not monotonically decreasing with video length — it actually peaks at 30 minutes (45.6) before declining to 36.2 at 60 minutes and then slightly recovering to 38.4 at 120 minutes. This non-monotonic pattern could indicate that the fast-weight memory reaches a saturation point around 30 minutes (with enough evidence accumulated for accurate counting) but then experiences some degradation at longer durations (perhaps due to accumulation of noise or the increasing challenge of distinguishing new objects from previously seen ones). The paper does not analyze this pattern.

Computational Efficiency Scaling (Table 5)

Table 5 reports peak GPU memory and TFLOPs per forward pass for Spatial-TTT-2B, Qwen3-VL-2B (the base general-purpose model), and Spatial-MLLM-4B (a geometry-augmented spatial model) across 128, 256, 512, and 1024 frames at 352×480 resolution.

At 128 frames, all three models use comparable memory (~6.2 GB for Qwen3-VL-2B and Spatial-TTT; 25.9 GB for Spatial-MLLM-4B) and TFLOPs (75.9 vs. 74.3 for Qwen3-VL-2B and Spatial-TTT; 1698.8 for Spatial-MLLM-4B). The gap widens substantially as frame count increases:

At 256 frames: Spatial-TTT uses 7.0 GB / 156.2 TFLOPs vs. Qwen3-VL-2B's 8.3 GB / 179.9 TFLOPs (Spatial-TTT is ~15% more efficient). Spatial-MLLM-4B uses 41.8 GB / 6002.1 TFLOPs (~6× the memory, ~38× the compute of Spatial-TTT).
At 512 frames: Spatial-TTT uses 8.4 GB / 341.9 TFLOPs vs. Qwen3-VL-2B's 12.6 GB / 473.9 TFLOPs (Spatial-TTT is ~33% more memory-efficient, ~28% more compute-efficient). Spatial-MLLM-4B runs out of memory.
At 1024 frames: Spatial-TTT uses 11.9 GB / 799.4 TFLOPs vs. Qwen3-VL-2B's 21.2 GB / 1403.1 TFLOPs (Spatial-TTT is ~44% more memory-efficient, ~43% more compute-efficient).

The paper argues that these numbers demonstrate "linear-complexity attention, where doubling the input length results in approximately doubled computation." Checking this: 128 to 256 frames (2× frames): TFLOPs go from 74.3 to 156.2 (2.10×); 256 to 512 frames (2×): 156.2 to 341.9 (2.19×); 512 to 1024 frames (2×): 341.9 to 799.4 (2.34×). The scaling is slightly super-linear, not perfectly linear, but substantially better than Qwen3-VL-2B's quadratic scaling (75.9 → 179.9 [2.37×] → 473.9 [2.63×] → 1403.1 [2.96×]). Spatial-MLLM-4B's scaling is catastrophically super-linear (1698.8 → 6002.1 [3.53×] at just 128 to 256 frames, then out of memory).

This is a genuine architectural advantage: Spatial-TTT processes longer videos at lower computational cost per frame than the standard attention architecture, and the efficiency gap widens with longer videos. For streaming applications where videos may be hours long, this means Spatial-TTT can operate indefinitely while standard attention eventually exhausts any finite memory budget.

Ablation Studies and Robustness Checks

All ablations use VSI-Bench as the evaluation benchmark and are reported in Table 4 (aggregate) and Table 7 (per-task breakdown). The full Spatial-TTT achieves 64.4 Avg. on VSI-Bench.

Spatial-predictive mechanism (w/o SP-Mechanism): Replacing the depthwise 3D spatiotemporal convolutions with identity projections drops the VSI-Bench average from 64.4 to 62.1 (a 2.3-point decline). The impact is asymmetric across task types: Numerical questions decline from 64.0 to 60.7 (3.3 points), while Multiple-Choice questions decline from 64.8 to 63.4 (1.4 points). Within Numerical, the largest drops are on Absolute Distance (47.8 → 43.0, -4.8) and Room Size (65.9 → 60.9, -5.0), suggesting the spatial-predictive mechanism is most important for tasks requiring precise metric estimation from accumulated visual evidence. Appearance Order is notably resilient (77.0 → 74.3, -2.7), consistent with the interpretation that temporal-order memory depends more on the TTT updates' ability to retain sequence information (which the spatial-predictive mechanism doesn't directly affect) than on fine-grained geometric encoding.

Dense scene-description supervision (w/o Dense Data): Training without the Stage 1 dense scene-description dataset drops the average from 64.4 to 61.3 (3.1 points), with roughly symmetric impact on Numerical (64.0 → 61.0, -3.0) and Multiple-Choice (64.8 → 61.5, -3.3). The per-task breakdown in Table 7 reveals that the impact is concentrated on tasks requiring long-horizon spatial memory integration: Room Size drops from 65.9 to 58.8 (-7.1), Relative Direction from 73.0 to 66.7 (-6.3), and Appearance Order from 77.0 to 71.8 (-5.2). Object Count (70.8 → 69.7, -1.1) and Absolute Distance (47.8 → 44.9, -2.9) show smaller drops. This pattern is consistent with the paper's credit-assignment argument: dense scene descriptions provide the richest supervision for tasks that cannot be answered from the final few frames and genuinely require evidence accumulated across the video. Tasks that can be at least partially answered from local evidence (counting visible objects, estimating distances from a single viewpoint) benefit less.

Hybrid TTT architecture (w/o Hybrid Arch): Removing the self-attention anchor layers — converting to a pure TTT architecture where all decoder layers use TTT — causes the largest degradation of any ablation: the average drops from 64.4 to 53.9 (10.5 points), with Multiple-Choice collapsing from 64.8 to 52.4 (12.4 points) and Numerical declining from 64.0 to 55.4 (8.6 points). The per-task breakdown shows catastrophic drops on several multiple-choice tasks: Appearance Order falls from 77.0 to 60.0 (-17.0), Relative Direction from 73.0 to 58.3 (-14.7), and Route Plan from 47.4 to 38.1 (-9.3). These are tasks requiring general visual understanding and semantic reasoning — exactly the capabilities that the pretrained anchor layers preserve. The Numerical tasks show more moderate but still substantial drops (Absolute Distance: 47.8 → 36.7, -11.1; Room Size: 65.9 → 54.1, -11.8). This confirms the paper's central architectural claim: pure TTT disrupts pretrained cross-modal alignment to the point where the model loses fundamental visual understanding, and the hybrid design recovers this through the anchor layers.

An interesting detail: even the "w/o Hybrid Arch" variant scores 53.9, which is above the base model Qwen3-VL-2B's 50.3 and several open-source baselines. This means the pure TTT architecture — despite suffering catastrophic forgetting of pretrained knowledge — still provides some benefit over the frozen base model, likely because the TTT updates do accumulate useful spatial evidence even if the visual-semantic alignment is degraded. The hybrid architecture is not strictly necessary for any improvement; it is necessary for the full 14.1-point gain over the base model.

Computational efficiency vs. spatial accuracy tradeoff. Table 5 provides an implicit ablation by comparing Spatial-TTT against Spatial-MLLM-4B, which adds an explicit geometry encoder for improved spatial accuracy. While the paper does not have Spatial-MLLM results on VSI-Bench to compare directly, the pattern is clear: spatial accuracy improvements through geometric computation come at catastrophic efficiency costs (Spatial-MLLM-4B uses 22× the TFLOPs at 128 frames and runs out of memory beyond 256 frames). Spatial-TTT achieves strong spatial performance through memory architecture rather than explicit geometry computation, maintaining efficiency at scale. This is an architectural tradeoff that the paper's design navigates, though a direct head-to-head accuracy comparison at the same computational budget is not provided and would be informative.

Scaling to longer videos. The VSI-SUPER results (Table 3) implicitly test robustness to video length, showing that Spatial-TTT maintains performance from 10 to 120 minutes while baselines collapse. However, the paper does not run controlled ablations varying chunk size or sliding window size at inference time, which would directly test sensitivity to these hyperparameters. The paper also does not report results beyond 120 minutes, leaving open the question of whether Spatial-TTT's performance would eventually degrade on even longer streams (hours to days of continuous footage).

Critical Assessment

Claim 1: Spatial-TTT achieves state-of-the-art performance on video spatial benchmarks.

Supported strongly for VSI-Bench and MindCube-Tiny, with caveats on specific tasks. The VSI-Bench result (64.4 Avg., Table 1) is genuinely the highest among all compared models, proprietary and open-source, by a margin of 3.8 points over the next-best open-source model (VST-7B-SFT at 60.6) and 8.4 points over the best proprietary model (Gemini-3-pro at 56.0). The MindCube-Tiny result (76.2%, Table 2) is also the highest overall by a margin of 12.3 points. These are robust results.

However, the claim of "state-of-the-art" warrants qualification at the per-task level. On Route Plan (VSI-Bench), Spatial-TTT scores 47.4 — below Gemini-3-pro (61.9), Kimi-K2.5 (52.1), GPT-5 (50.2), and only modestly above Qwen3-VL-8B (35.0). On ROTATION (MindCube-Tiny), it scores 55.5 — far below GPT-5 (94.5), Grok-4 (93.0), and Gemini-2.5-pro (88.0). On Appearance Order (VSI-Bench), it scores 77.0 — below Cambrian-S-3B (78.8). The "state-of-the-art" claim holds in aggregate but masks significant task-level weaknesses, particularly on mental rotation and navigation planning. The paper does not discuss these weaknesses, which limits understanding of the model's failure modes.

Claim 2: Test-time training enables continual spatial evidence accumulation over arbitrarily long videos.

Supported convincingly by VSI-SUPER-Count, partially by VSI-SUPER-Recall. The VSI-SUPER-Count results (Table 3) provide the strongest evidence: Spatial-TTT achieves 31.8–45.6% accuracy across all durations while every baseline scores near zero. This is not a marginal improvement — it is a qualitative difference in capability. The baselines literally cannot perform the task at any duration, while Spatial-TTT performs it at all durations. This strongly supports the claim that TTT provides a mechanism for streaming spatial evidence accumulation that standard architectures lack.

The VSI-SUPER-Recall results are less decisive. At 10 minutes, Spatial-TTT ties Cambrian-S-7B at 38.3. At 30 minutes, it scores 35.0 vs. Cambrian-S's 35.0. The advantage only becomes dramatic at 120 minutes because baselines run out of memory — but this is partly a hardware limitation rather than a pure capability limitation. If Cambrian-S-7B were run on hardware with sufficient memory (the paper uses an 80GB GPU as the reference), would its performance continue to scale, or would it saturate? The paper cannot answer this because the out-of-memory failure is absolute on the tested hardware. The superiority at 120 minutes demonstrates practical deployability, not necessarily superior spatial reasoning at scale.

A more rigorous test would measure accuracy as a function of computation (TFLOPs) rather than video duration, to confirm that Spatial-TTT genuinely extracts more spatial understanding per unit of computation at long durations rather than simply not crashing. The current results conflate algorithmic efficiency with hardware feasibility — both are important, but they support slightly different claims.

Claim 3: The hybrid architecture (3:1 TTT-to-attention ratio) is necessary to preserve pretrained visual-semantic knowledge.

Strongly supported by the ablation. The "w/o Hybrid Arch" condition (Table 4) causes a 10.5-point drop — the largest single ablation effect — with disproportionate impact on multiple-choice questions that require general visual understanding. This directly supports the claim that pure TTT disrupts pretrained knowledge. The magnitude and pattern of the drop are consistent with the paper's architectural argument.

However, the ablation tests only the extreme: all anchor layers removed vs. 3:1 ratio. It does not test intermediate ratios (e.g., 1:1, 1:3, 1:7) or different anchor layer placements (e.g., clustering anchor layers at specific depths rather than interleaving). It's possible that a 1:7 ratio (one anchor layer every 8 layers) would preserve most of the benefit with even better efficiency, or that a 1:1 ratio would provide diminishing returns. The 3:1 ratio is presented as a design choice without evidence that it is optimal or near-optimal. A sweep over ratios would substantially strengthen the architectural argument.

Claim 4: The dense scene-description dataset teaches fast-weight dynamics to retain globally useful spatial evidence.

Supported directionally, but the evidence is incomplete. Removing the dense data drops performance by 3.1 points (Table 4), with larger drops on long-horizon memory tasks like Room Size (-7.1) and Relative Direction (-6.3). This is consistent with the claim that dense supervision improves long-horizon memory.

However, the paper does not run the obvious control: training Stage 1 on the same number of sparse spatial QA examples instead of dense descriptions. The "w/o Dense Data" condition removes Stage 1 entirely, going directly to Stage 2. This conflates data quantity (16K additional training examples) with data quality (dense scene descriptions vs. sparse QA). The 3.1-point drop could be partially or entirely attributable to simply having 16K fewer training examples, regardless of their format. A proper ablation would keep the total training data constant while varying the format (dense descriptions vs. additional spatial QA) in Stage 1. Without this control, the paper's credit-assignment argument — that dense supervision specifically teaches better fast-weight update dynamics, as opposed to just providing more data — is plausible but unproven.

Additionally, the paper does not analyze what the dense descriptions teach that sparse QA does not. Does the benefit come from the global context descriptions, the object enumeration, the spatial relation descriptions, or all three? A finer-grained ablation decomposing the dense descriptions into components would illuminate the mechanism.

Claim 5: The spatial-predictive mechanism improves geometric correspondence and temporal continuity.

Supported with modest effect size. The 2.3-point drop from removing the mechanism is statistically meaningful (Table 4) and shows the expected pattern of larger impact on numerical/metric tasks. However, 2.3 points is a relatively small fraction of the 14.1-point total gain over the base model (about 16%). The spatial-predictive mechanism is not a primary driver of Spatial-TTT's performance — the hybrid architecture and streaming memory are far more impactful. The paper's textual emphasis on this mechanism is somewhat disproportionate to its empirical contribution.

The ablation replaces the 3D convolutions with identity projections, which is the correct control. However, it does not test alternatives: 2D spatial-only convolutions (no temporal dimension), temporal-only convolutions (1D along the time axis), or larger kernel sizes. Any of these could provide insight into what aspect of the spatiotemporal context is useful — spatial smoothing, temporal continuity, or both jointly.

Overall Assessment of Experimental Rigor

Strengths:

The baseline comparison is unusually comprehensive, spanning proprietary frontier models, open-source general MLLMs, and open-source spatial intelligence models across multiple scales.
The four evaluation benchmarks test complementary aspects of spatial intelligence (general VQA, multi-view reasoning, long-horizon recall, streaming counting), providing a multi-faceted picture of capabilities.
The efficiency analysis (Table 5) quantifies the practical advantage of the TTT architecture with concrete hardware measurements (GPU memory, TFLOPs), not just theoretical complexity claims.
The ablation study isolates the contribution of each architectural component and training data component with clean controls.

Weaknesses:

Test set sizes are modest. VSI-Bench has 5,000+ questions but split across 8 tasks, so per-task sample sizes may be in the hundreds. MindCube-Tiny has 1,050 questions total. VSI-SUPER sample counts per duration are not reported. No confidence intervals are provided, making it impossible to assess whether observed differences are statistically significant.
The paper evaluates on a single base model (Qwen3-VL-2B-Instruct). All claims about the TTT architecture's generality are untested — would the same architecture work on InternVL3, LLaVA-OneVision, or other MLLM backbones? The choice of Qwen3-VL-2B is reasonable but limits the scope of claims.
Several important controls are missing: (a) the dense data ablation conflates data quantity and data format; (b) the hybrid architecture ablation tests only the extreme (no anchor layers) rather than intermediate ratios; (c) the sliding window annealing is not ablated (what if w were fixed at 2648 from the start of Stage 1?); (d) the 3:1 TTT-to-attention ratio is not justified with a sweep.
The FLOPs-matched comparison is between different architectures, not between the same architecture at different scales. This makes it a hardware efficiency comparison, not a scaling law analysis. The paper does not answer "how much better would a 7B-parameter Spatial-TTT be?" or "at what parameter scale does a standard architecture match Spatial-TTT-2B's spatial reasoning?"
The VSI-SUPER results, while dramatic, show baseline failures largely due to out-of-memory errors. This is practically important (demonstrating deployability) but scientifically limiting (can't distinguish between "the architecture is better at spatial reasoning" and "the architecture doesn't crash").
The paper does not report inference latency or wall-clock time, only TFLOPs and memory. For streaming applications, latency per frame matters — the dual KV cache mechanism should provide constant per-frame latency, but this is not measured.
There is no analysis of failure cases or qualitative error patterns. The paper does not show examples where Spatial-TTT makes mistakes, which would provide insight into the limits of streaming memory accumulation (e.g., does it systematically fail when objects are visually similar? when viewpoints change rapidly? when occlusions are prolonged?).

Missing experiments that would strengthen the paper:

A controlled comparison keeping Stage 1 data quantity constant while varying format (dense descriptions vs. additional spatial QA vs. generic video captioning) to isolate the supervision format effect.
A sweep over TTT-to-attention ratios (e.g., 1:1, 3:1, 7:1, all-TTT) to find the optimal tradeoff point.
Scaling the approach to a larger base model (e.g., Qwen3-VL-8B-Instruct) to test whether the benefits compound with scale or saturate.
VSI-SUPER evaluation with all baselines run on sufficient hardware to avoid out-of-memory errors, enabling a pure capability comparison at matched video duration.
Latency measurements for streaming deployment scenarios.
Qualitative error analysis showing where Spatial-TTT's spatial memory fails and what types of spatial reasoning remain beyond its capability.

In summary, the experiments strongly support the paper's core architectural claims — that hybrid TTT enables streaming spatial intelligence with preserved pretrained knowledge, that dense descriptions improve training, and that the resulting system outperforms existing models while scaling efficiently. However, the evidence is more suggestive than definitive on the specific mechanisms (spatial-predictive convolutions, dense supervision format vs. quantity), and several claims about optimality and generality are asserted without the experiments needed to verify them. The paper's contribution is best understood as demonstrating the viability and strong performance of a specific architecture co-designed for streaming spatial intelligence, rather than as establishing general principles or optimal configurations through systematic experimentation.

6. Limitations and Trade-offs

The Difficulty Estimation Bottleneck: 2048 Samples Per Question Makes the Framework Impractical for Deployment

The assumption or constraint. The compute-optimal allocation policy — which the paper demonstrates achieves up to 4× efficiency gains — depends on knowing each prompt's difficulty before allocating the test-time compute budget. The paper estimates difficulty by generating 2048 complete solutions per question and computing either pass@1 (oracle, requiring ground-truth labels) or average PRM final-answer score (predicted, not requiring labels but still requiring 2048 generations). The paper acknowledges this cost explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The predicted difficulty variant removes the need for ground-truth answers but does not remove the need for 2048 samples — it simply replaces the correctness check with the PRM's own score, which is itself a model forward pass. This means the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The paper does not include this cost in any budget calculation, efficiency analysis, or comparison against baselines. The 4× figure is computed after difficulty is known, treating the estimation as free.

The consequence. If the difficulty estimation cost is amortized into the total compute budget, the reported efficiency gains shrink dramatically or vanish entirely. Consider a prompt that the compute-optimal policy solves with 16 generations. The true cost is 2048 (estimation) + 16 (solution) = 2064 generations — 129× the headline budget. Even if difficulty estimation is shared across many prompts (e.g., estimating once for a benchmark and reusing), this is exactly the regime the paper does not evaluate: the difficulty bins are computed per-question on the test set, and the two-fold cross-validation means each fold's policy is selected based on ~50 questions per bin. There is no experiment showing that difficulty estimates transfer across questions — that a difficulty estimator trained on one set of prompts works on another. The compute-optimal framework as presented is an upper bound on achievable efficiency, not a deployment-ready system. A practitioner would need to solve the difficulty estimation problem before realizing any of the paper's headline gains.

What evidence exists in the paper. The paper shows that predicted difficulty bins (using PRM scores instead of ground-truth pass@1) largely overlap with oracle bins in the search setting (Figure 4) and show slightly lower but still substantial gains in the revision setting (Figure 8, roughly 41% vs. 44% at 256 generations). This demonstrates that the PRM signal is a good proxy for difficulty, but it does not demonstrate that difficulty can be estimated cheaply — the predicted bins still require 2048 samples. The paper provides no experiment with fewer than 2048 estimation samples, no learned difficulty predictor trained on question text alone, and no analysis of how estimation accuracy degrades as the number of estimation samples decreases. The VSI-SUPER results (Table 3) implicitly involve difficulty estimation since the model must process streaming video, but this is a different capability (online evidence accumulation) than the per-question difficulty binning the compute-optimal policy relies on.

Mitigation status. The paper explicitly flags this as future work in Section 3.2: future work on "pretraining or finetuning models to directly predict difficulty of a question" and in Section 8: adaptive difficulty estimation that "start[s] with a few parallel samples, assess[es] the score distribution, and decide[s] in real-time whether to switch." These are the right directions, but no such system is developed or evaluated in this paper. A practitioner reading for deployment guidance would find that the central mechanism enabling the paper's gains (compute-optimal allocation) depends on a subsystem that does not exist in practical form. The 4× efficiency figure should be interpreted as a proof of concept for the value of difficulty-adaptive allocation, not as a realized efficiency gain available today.

The Revision Model's 38% Correct-to-Incorrect Reversion Rate Is a Fundamental Stability Problem

The assumption or constraint. The revision model is fine-tuned exclusively on sequences where all in-context answers are incorrect followed by a correct target answer (Section 6.1). It is never trained to see a correct answer in context and leave it unchanged. The training data construction procedure specifically pairs incorrect answers with correct answers, using edit distance to select incorrect answers that are "close" to the correct one so the model learns targeted corrections. The consequence of this one-sided training is acknowledged in Section 6.1:

"the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers"

The paper reports that approximately 38% of correct answers get converted back to incorrect ones using a naive approach where the final revision in the chain is taken as the answer.

The consequence. This is not a minor edge case — it is a structural instability in the revision process. A sequential revision chain of length N will, on average, contain mixtures of correct and incorrect answers. If each correct answer has a 38% chance of being "revised" to incorrect in the next step, the chain's overall reliability degrades with length, and the final answer is not necessarily the best answer. This forces the system to use post-hoc selection mechanisms — majority voting or verifier-based selection across the entire chain — to recover the best answer, which means the sequential revision process is not self-correcting in a monotonic sense. It generates a distribution of answers, some correct and some incorrect, and relies on a separate mechanism to pick the right one.

The practical implication is that the revision model cannot be deployed as a simple "generate a chain and take the last answer" system. It requires a verifier or voting mechanism as a wrapper, which adds cost and complexity. More fundamentally, the 38% reversion rate means that lengthening the revision chain does not guarantee improvement — beyond some optimal chain length, the probability of reversing a correct answer may outweigh the probability of correcting an incorrect one, creating a performance ceiling that additional sequential compute cannot break through. This is visible, albeit subtly, in Figure 6 (left): the revision model's per-step pass@1 improves from ~18.2% at step 1 to ~24–25% by steps 15–20 and then plateaus, never exceeding ~25% regardless of chain length. The plateau is consistent with a system where corrections and reversions reach equilibrium.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. Figure 6 (left) shows the pass@1 plateau. The paper mitigates the problem with majority voting and verifier-based selection across the chain (Section 6.1, Appendix B, Appendix I), which recovers better performance than taking the final revision — but this mitigation confirms the problem exists. The ReST^EM experiment (Appendix K, Figure 16) provides additional evidence of instability: an attempt to optimize the revision model with on-policy RL-style training causes performance to degrade substantially with sequential revisions, with fully sequential performance dropping to ~33.5% compared to ~38.5% at the optimal ratio at 256 generations. The authors hypothesize that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data," suggesting that the revision model's training is fragile and sensitive to data distribution.

Mitigation status. The paper mitigates the symptom (incorrect final answers) through within-chain selection, but does not address the cause (one-sided training that never teaches the model to recognize and preserve correct answers). The mitigation is partial: majority voting and verifier-based selection recover the best answer in the chain, but they don't prevent the model from wasting compute generating revisions that undo correct answers. A more principled solution — such as training the model with a "no revision needed" token or including correct-to-correct trajectories in the training data — is not explored. The paper does not discuss this as a direction for future work, which is a notable omission given that the 38% figure is one of the most concretely quantified failure modes in the paper.

Test-Time Compute Cannot Create Capability Where None Exists: Hard Problems Are Unsolved Regardless of Budget

The assumption or constraint. The paper's most important negative result appears across every experiment: on the hardest problems (difficulty bin 5), no amount of test-time compute produces meaningful improvement. This is not a limitation of a specific method — it holds for search (Figure 3, right: bin 5 accuracy at 1–3% for all methods and all budgets), for revisions (Figure 7, right: bin 5 accuracy at ~2–3% regardless of sequential-to-parallel ratio), and for compute-optimal combinations of both. The FLOPs-matched comparison (Section 7, Figure 9) shows that on the hardest problems, the ~14× larger model consistently outperforms the smaller model with test-time compute, and at high inference-to-pretraining ratios (R ≫ 1), the disadvantage is as large as −52.9% relative for PRM search.

The underlying reason is structural: test-time compute amplifies existing capability by exploring, refining, and selecting among the base model's outputs. If the base model's pass@1 on a problem class is near zero — meaning it essentially never produces a correct solution even with 2048 independent samples — then no search algorithm, no revision strategy, and no compute-optimal allocation can find a correct solution in the proposal distribution. The paper acknowledges this in Section 7:

"test-time compute can amplify existing capability but cannot create it"

The consequence. The practical implication is sharp: test-time compute is not a substitute for pretraining on problems outside the base model's competence boundary. For a deployment where the problem distribution includes a non-trivial fraction of genuinely hard problems (pass@1 ≈ 0), the compute-optimal framework offers no benefit — the system should fall back to a larger model, a different architecture, or human intervention. The paper does not provide a mechanism for detecting when a problem is in this regime without first spending the compute to attempt it, which means a naive deployment would waste the difficulty estimation budget (2048 samples) only to discover that no strategy helps.

More subtly, this limitation means that the pretraining-inference tradeoff is not continuous — it has a cliff. For easy-to-medium problems (bins 1–3), test-time compute with a small model outperforms a 14× larger model, sometimes by large margins (+27.8% on medium problems at R ≪ 1 for revisions). For hard problems (bins 4–5), the larger model wins, and the gap widens with inference volume. There is no smooth interpolation between these regimes; the transition is binary (capable vs. incapable) rather than continuous. This makes it difficult to predict a priori whether test-time compute or pretraining scale will be more cost-effective for a given deployment without knowing the difficulty distribution of incoming prompts.

What evidence exists in the paper. The failure on hard problems is the most consistently replicated result in the paper — it appears in Figure 3 (right, bin 5), Figure 7 (right, bin 5), Figure 9 (bin 5 scaling curves essentially flat near 0–5% across all budgets), and the FLOPs-matched bar charts in Figure 1 (negative values for hard problems at higher R). The paper is admirably transparent about this limitation, including it in the Section 7 takeaway box and not overclaiming. The ~14× larger model comparison uses greedy decoding (no test-time compute of its own), which is a conservative baseline — a larger model with even modest test-time compute (e.g., best-of-8) might widen the gap further on hard problems.

Mitigation status. The paper does not attempt to solve this limitation — it is arguably unsolvable within the test-time compute paradigm. The mitigation is to recognize the boundary and route hard problems to pretraining-scaled models. The paper does not provide a mechanism for doing this routing without first spending the difficulty estimation budget, which creates a chicken-and-egg problem: you need to estimate difficulty to know whether to spend compute, but estimating difficulty costs compute. The paper also does not explore whether combining test-time compute with the larger model on hard problems would yield further gains — the ~14× larger model is used with greedy decoding only, and no experiments give the larger model access to PRM search or revision chains. This is an important open question: does test-time compute help the larger model on hard problems, or does the larger model's higher base pass@1 push those problems into a difficulty regime where compute-optimal strategies work?

The Test Set Is 500 Questions, Strategy Selection Uses ~50 Questions Per Bin Per Fold, and No Confidence Intervals Are Reported

The assumption or constraint. All main experiments use the MATH benchmark's standard 500-question test set (Section 4). The five difficulty quintiles split this into bins of ~100 questions each. The two-fold cross-validation protocol for selecting the compute-optimal strategy further splits each bin roughly in half, meaning the best strategy per bin per budget level is selected based on approximately 50 questions per fold. The paper does not report confidence intervals, standard errors, or any measure of statistical uncertainty for any result. Claims about which strategy is "best" for a given difficulty bin at a given budget, and claims about the magnitude of efficiency gains (e.g., 4×), are point estimates without uncertainty quantification.

The consequence. With ~50 questions per fold per bin, the variance of the accuracy estimate is substantial. For a bin with 50 questions, a single-question swing changes accuracy by 2 percentage points. If the "best" strategy for a bin outperforms the second-best by less than the expected sampling variance, the selected policy may be noise — a different random split of the 500 questions could produce a different "best" strategy. The paper's claim that compute-optimal scaling outperforms best-of-N by 4× depends on the specific strategies selected for each bin at each budget; if some of those selections are unstable, the reported efficiency gains may not replicate on a different test set.

This is a particular concern for the difficulty-dependent claims, which are the paper's central contribution. The observation that beam search helps on medium problems but hurts on easy problems (Figure 3, right) is based on comparing per-bin accuracies derived from ~100 questions per bin before cross-validation, ~50 after. If the bin boundaries were drawn slightly differently (e.g., using 4 bins instead of 5), would the pattern hold? The paper does not test sensitivity to bin count or bin boundaries. A bootstrapped confidence interval on the per-bin accuracy curves would show whether the observed difficulty-dependent patterns are robust to sampling variation.

For the broader claims about state-of-the-art performance, the issue also applies to the baselines. The paper reports point estimates for each baseline model on the 500-question test set (Table 1). If a baseline model's score is 60.6 and Spatial-TTT's is 64.4, the 3.8-point gap is reported without any indication of whether it is statistically significant. Given the test set size, a gap of ~4 points on a 500-question set corresponds to roughly 20 more correct answers out of 500. Whether this is significant depends on the correlation structure (the same questions are used for both models), which the paper does not analyze.

What evidence exists in the paper. The paper documents the test set size (500 questions, Section 4), the difficulty binning procedure (5 quintiles, Section 3.2), and the cross-validation protocol (2-fold, Section 3.2). It does NOT report: sample sizes per bin per fold, confidence intervals on any accuracy estimate, standard errors, statistical tests comparing methods, or sensitivity analysis to bin count or bin boundaries. The baseline comparisons in Tables 1–3 report point estimates only. The efficiency scaling curves in Figures 4 and 8 are drawn as continuous lines without error bars or shading, even though each point on those curves is an estimate from a subset of the test data. This is standard practice for many ML papers, but it makes the precision of the reported gains difficult to assess.

Mitigation status. The paper does not address this limitation. The standard mitigations — bootstrapped confidence intervals, reporting standard errors, or using a larger test set — are not employed. The paper could also have tested robustness by varying the number of difficulty bins (e.g., 3, 5, 7) to show that the qualitative patterns are not sensitive to this choice, or by reporting the variance of the compute-optimal policy's performance across cross-validation folds. None of these robustness checks are included. This limitation is particularly consequential because the paper's central claim — that compute-optimal adaptive allocation yields 4× gains over uniform allocation — depends on the policy selected per bin, which in turn depends on the bin-level accuracy estimates whose reliability is not quantified.

All Results Are on a Single Model Family (PaLM 2-S*) and a Single Benchmark (MATH), With No Evidence of Generalization

The assumption or constraint. Every experiment in the paper uses PaLM 2-S* (Codey) as the base model and the MATH benchmark as the evaluation dataset (Section 4). The PRM training uses MATH training questions. The revision model training uses MATH training questions. The compute-optimal policies are selected and evaluated on MATH test questions. The FLOPs-matched comparison uses a ~14× larger model from the same PaLM 2 family. The authors state in Section 4:

"we believe this model is representative of the capabilities of many contemporary LLMs"

This is an assertion, not a demonstrated fact. At no point does the paper test whether the findings — the difficulty-dependent efficacy of search vs. revisions, the 4× efficiency gain from compute-optimal allocation, the verifier over-optimization patterns, the superiority of PRM over ORM, the sequential-to-parallel ratio trends — transfer to a different model family (e.g., LLaMA, Gemma, GPT), a different model scale, or a different reasoning benchmark (e.g., GSM8K for math, HumanEval for code, ARC for science).

The consequence. The paper's findings may be specific to the intersection of PaLM 2-S* and MATH. The representativeness claim is particularly questionable given several findings that appear sensitive to model-specific properties:

The PRM's optimal aggregation strategy was "last" (using only the final step's score), contrary to prior work that found "min" to be best (Lightman et al., 2023; Wang et al., 2023). The authors hypothesize this is due to their use of soft Monte Carlo labels rather than binary correctness labels (Appendix E). If the optimal aggregation strategy depends on PRM training details, the entire search methodology may need recalibration for different models or label sources.
The PRM800k dataset (which contains GPT-4-generated solutions with human step-level labels) was "largely ineffective" for PaLM 2 models (Section 5.1), attributed to distribution shift between GPT-4 and PaLM 2 outputs. This means the PRM training recipe — Monte Carlo rollouts from the base model — is model-specific, and the cost of training a PRM from scratch for each new base model is substantial.
The revision model's correct-to-incorrect reversion rate of 38% (Section 6.1) is likely sensitive to the base model's in-context learning behavior and the specific edit-distance-based training data construction. Different model families with different in-context learning capabilities might show different revision stability.
The FLOPs-matched comparison's ~14× multiplier depends on the specific parameter count ratio between PaLM 2-S* and the larger model. A different model family with a different scaling trajectory might show a different crossover point between test-time compute and pretraining.

Without evidence of generalization, a practitioner using a different model (say, LLaMA-3-8B) or a different reasoning domain (say, code generation) has no basis for predicting whether Spatial-TTT's design choices or the 4× efficiency claim will hold. The paper's findings are best understood as existence proofs — demonstrating what is possible — rather than as established principles that transfer across models and domains.

What evidence exists in the paper. The paper provides exactly zero cross-model or cross-benchmark experiments. All results in Section 5 (PRM search), Section 6 (revisions), Section 7 (FLOPs-matched comparison), and all appendices are on PaLM 2-S* models evaluated on MATH. The paper cites work using other models and benchmarks in the related work (Section 2), but does not replicate or extend those findings with the proposed methods. The baselines in Tables 1–2 include other model families (e.g., the Qwen3-VL results on VSI-Bench, the MindCube results on MindCube-Tiny), but Spatial-TTT is evaluated only on PaLM 2-S* with MATH. This is the norm for systems papers introducing a new architecture, but it limits the strength of the claims.

Mitigation status. The paper acknowledges this limitation implicitly by not claiming generality beyond MATH and PaLM 2-S*. Section 8 suggests future work on extending to "other domains and modalities" including code generation and logical reasoning, but no such experiments are conducted. A practitioner would need to replicate the key experiments on their target model and domain before adopting the compute-optimal framework. The paper does not provide guidance on which findings are most likely to transfer (e.g., the difficulty-dependent strategy selection principle may be universal even if specific thresholds differ) versus which are likely model-specific (e.g., the optimal PRM aggregation method).

Sequential Revisions Are Inherently Serial, Creating a Latency-vs-Accuracy Tradeoff the Paper Does Not Address

The assumption or constraint. The paper measures test-time compute in "generations" — the total number of complete solutions sampled — and treats all generations as having equal cost. This is appropriate for a FLOPs-based analysis but ignores wall-clock latency. Sequential revisions (a chain of length N where each revision depends on the previous one) are inherently serial: each revision must complete before the next can begin. Parallel best-of-N sampling (N independent solutions) can be executed simultaneously with sufficient hardware parallelism. A compute-optimal policy that allocates, say, 64 generations as 8 parallel chains of length 8 takes approximately 8× the wall-clock time of a policy that uses 64 parallel samples, even though both use the same total FLOPs.

The paper's revision strategy is particularly latency-heavy. Figure 7 (left) shows that the optimal sequential-to-parallel ratio varies with difficulty and budget, but on easy questions (where the paper finds sequential revisions are most effective), the optimal ratio at low budgets is fully sequential (all N generations in a single chain). For N = 64, this means 64 serial forward passes, each depending on the previous output. On latency-sensitive applications (interactive assistants, real-time robotic control, live tutoring), this serial dependency may be prohibitive regardless of accuracy gains.

The consequence. The paper's compute-optimal policy, as formulated, optimizes for total FLOPs but not for latency or throughput. A practitioner deploying the system must consider a three-way tradeoff: accuracy vs. total FLOPs vs. wall-clock time. The paper provides analysis only for the first two dimensions. In a regime where latency is the binding constraint (e.g., a user waiting for an answer, a robot needing a decision within 500ms), the optimal policy under a latency budget may be qualitatively different from the optimal policy under a FLOPs budget. For instance, fully parallel best-of-N might be preferable to sequential revisions even on easy problems if the parallel samples can be generated quickly enough, because the latency advantage of parallelism outweighs the accuracy advantage of revisions.

The FLOPs-matched comparison in Section 7 implicitly assumes that all FLOPs are equal regardless of whether they are spent serially or in parallel. The R parameter (inference-to-pretraining token ratio) captures total inference volume but not latency constraints. A deployment with R ≪ 1 (low inference volume) might still require low latency per query, making the sequential strategies favored by the compute-optimal policy impractical.

What evidence exists in the paper. The paper provides no latency measurements, no wall-clock time analysis, and no discussion of the latency-vs-throughput tradeoff for sequential vs. parallel strategies. The generation budget is treated as the sole resource constraint. The inference-cost accounting in Section 7 uses FLOPs only. The hardware efficiency analysis in Table 5 reports peak GPU memory and TFLOPs per forward pass, which capture parallelism within a forward pass but not the serial dependency between forward passes in a revision chain. A single forward pass through Spatial-TTT at 128 frames costs 74.3 TFLOPs — but 64 sequential forward passes cost the same total FLOPs as 64 parallel forward passes (on 64 GPUs) while taking 64× longer in wall-clock time (on 1 GPU). The paper's framework does not distinguish these cases.

Mitigation status. Not addressed. The paper does not mention latency as a constraint, does not measure wall-clock time, and does not discuss the serial dependency in revision chains as a practical limitation. Future work on latency-aware compute-optimal allocation — where the optimization includes a constraint on maximum wall-clock time per query — is not suggested. The omission is notable because the paper motivates test-time compute partly for "on-device deployment" (Section 1), where latency is typically a first-class constraint. An on-device assistant that takes 64× the latency to answer a question because it's running sequential revisions may not be acceptable to users, even if the total FLOPs are within budget.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological pivot in how the field approaches visual spatial intelligence for MLLMs. The dominant paradigm — represented by VST, Cambrian-S, SpatialLadder, and others evaluated herein — has been to improve spatial reasoning through better pretraining data, larger spatial QA datasets, or explicit geometric encoders (depth estimation, multi-view reconstruction) that augment input representations. These approaches operate at training time: they change what the model learns before deployment, but the model's inference mechanism remains static and bounded by its architecture's fixed memory limitations. The VSI-SUPER results in Table 3 expose the ceiling of this paradigm with brutal clarity: models designed explicitly for spatial understanding (Cambrian-S-7B, a multi-stage trained spatial specialist) collapse to zero accuracy on streaming counting and recall when videos extend beyond a few minutes, not because they lack spatial knowledge but because their architecture cannot sustain accumulation — they must process all frames simultaneously, and when frame count exceeds memory, they fail entirely.

Spatial-TTT shifts the locus of spatial intelligence from what the model knows at deployment to how the model continuously learns at inference. The fast weights are not just a compressed memory — they are an online learning mechanism that builds a 3D representation of each specific environment as the video streams. This reframing is significant not because TTT is novel as an efficiency technique (it is well-established in language modeling; Sun et al., 2024b; Zhang et al., 2025), but because it repositions inference-time computation as a first-class resource for spatial intelligence rather than a fixed cost. The model's spatial understanding of a room is not predetermined by its pretraining — it emerges through gradient-based updates that accumulate evidence tailored to that specific room.

The conceptual shift has several cascading implications for research priorities:

The spatial intelligence bottleneck moves from data curation to memory architecture. Prior work invested enormous effort in building larger spatial QA datasets — VST's 4.1M SFT examples, Cambrian-S's VSI-590K, InternSpatial, VLM-3R's instruction-aligned reconstruction data. Spatial-TTT demonstrates that a model trained on only ~3M spatial QA examples plus ~16K dense scene descriptions, but equipped with a streaming memory architecture, outperforms models trained on larger datasets with static architectures. This does not mean data is unimportant — the dense scene descriptions are essential (removing them costs 3.1 points on VSI-Bench, Table 4) — but it means that data quality and architectural capability are complementary, and the latter may be the binding constraint for streaming settings. Research on spatial intelligence can no longer treat "longer context window" or "more training data" as sufficient solutions; the memory mechanism itself must be designed for accumulation.

The hybrid architecture establishes a design pattern for applying TTT to pretrained multimodal models. The paper's diagnosis — that pure TTT disrupts cross-modal alignment encoded in pretrained attention layers, and that interleaving anchor attention layers at a 3:1 ratio preserves this alignment — is a specific, quantified solution to a problem that prior TTT work did not address because it operated on language-only or from-scratch models. Removing the anchor layers causes a 10.5-point drop on VSI-Bench (Table 4), the largest single ablation effect. This establishes a template for any future work that wants to retrofit pretrained MLLMs with efficient streaming mechanisms: preserve a subset of original attention layers as anchors, don't replace everything, and expect catastrophic degradation if you do. The 3:1 ratio may not be universal, but the principle — that pretrained multimodal alignment requires some full-attention computation at regular intervals — likely transfers.

The paper implicitly redefines what "state-of-the-art spatial intelligence" means at the benchmark level. Prior work compared models on short-context spatial VQA (VSI-Bench at 16–32 frames, MindCube at 4 views). Spatial-TTT's strongest advantages appear on the streaming benchmarks that prior spatial models could not even run: VSI-SUPER-Count at 120 minutes (38.4% vs. 0.0% for all baselines, Table 3) and VSI-SUPER-Recall at 120 minutes (30.0% vs. 0.0% for spatial baselines). This creates a bifurcation in the benchmark landscape: short-context spatial understanding benchmarks will continue to see incremental gains from better data and geometry encoders, but streaming benchmarks require fundamentally different architectures. A model that scores 80 on VSI-Bench but crashes on VSI-SUPER is not "state-of-the-art" for streaming spatial intelligence — it is untested. The paper sets a new standard: spatial intelligence claims must be accompanied by evidence of scaling behavior, not just peak performance on bounded inputs.

The dense-description-as-curriculum methodology demonstrates a general principle for training recurrent memory systems. The paper's diagnosis of the credit assignment problem in training TTT for downstream tasks — sparse task supervision provides weak gradient signals for learning update dynamics because the model can shortcut by answering from the final frames — is not specific to spatial intelligence. Any system that uses learned recurrent memory (TTT, linear attention, state-space models, recurrent networks) for long-horizon tasks faces the same issue: the memory update rule must be learned, but end-task supervision provides indirect credit assignment across many update steps. The solution — train first on a dense auxiliary objective that demands comprehensive retention (dense scene descriptions), then fine-tune on the sparse end task — is a curricular template that could apply to video question answering, long-document understanding, dialogue state tracking, and any domain where memory must be learned from sparse final-output supervision. The specific form of the auxiliary objective (comprehensive scene descriptions for spatial tasks) would vary by domain, but the principle of bridging sparse end-task supervision with dense auxiliary supervision during training is reusable.

The paper also resolves a latent contradiction in the spatial intelligence literature. Prior work showed mixed results on whether geometric feature injection helps: Spatial-MLLM and VLM-3R showed gains from explicit 3D reconstruction features, but at catastrophic computational cost (Spatial-MLLM-4B uses 22× the TFLOPs of Qwen3-VL-2B at 128 frames and fails beyond 256 frames, Table 5). Other work (VST, Cambrian-S) achieved strong short-context performance without explicit geometry encoders, through large-scale spatial QA fine-tuning. The implicit question was: do you need explicit geometric computation for spatial intelligence, or is it learnable from data? Spatial-TTT's results suggest a third answer: you need architectural support for spatial evidence accumulation (streaming memory), but not necessarily explicit geometric feature computation. The spatial-predictive mechanism — depthwise 3D convolutions on Q/K/V with Dirac initialization — provides a lightweight geometric inductive bias (costing ~166K parameters per TTT layer) that improves performance by 2.3 points (Table 4), but this is dwarfed by the 10.5-point gain from the hybrid architecture itself. The primary value is in the memory mechanism, not the geometric features. This reframes the debate: spatial intelligence for MLLMs is more a temporal integration problem (accumulating evidence across views) than a geometric representation problem (computing depth or 3D coordinates). Both matter, but the former is the binding constraint for streaming settings.

What research directions become less attractive? Explicit geometry encoders that process all frames simultaneously (like Spatial-MLLM's) become questionable for any application involving more than a few dozen frames — the computational cost is prohibitive, and the streaming TTT approach achieves competitive or superior spatial accuracy without them. Large-scale spatial QA dataset construction, while still valuable, should be complemented by dense auxiliary objectives that directly supervise memory updates, not just final answers — the paper shows that 16K dense descriptions provide benefits that 3M sparse QA examples alone do not fully replicate. And "bigger context window" as a solution to streaming video understanding is refuted by the VSI-SUPER results: Qwen3-VL-2B and Cambrian-S-7B both run out of memory at scale regardless of context window size, because the problem is not just capacity but computation — quadratic attention costs dominate at thousands of frames regardless of whether the model theoretically "supports" long context.

Follow-Up Research This Work Enables

Cheap difficulty estimation for compute-optimal streaming allocation. The paper's most important missing piece is a practical difficulty estimator that does not require 2048 samples per question. The dense scene-description dataset provides a natural training signal for a lightweight difficulty predictor: train a small classifier (perhaps distilled from the PRM's score distribution) that takes only the first few frames of a video as input and predicts whether the scene is "easy" (sparse, few objects, simple layout), "medium" (moderate complexity), or "hard" (cluttered, many objects, complex geometry). The paper's per-difficulty-bin analysis (Figures 3 right, 7 right) shows that difficulty is largely determined by scene complexity and the base model's pass@1 on that scene type, both of which may be estimable from early visual evidence. A strong follow-up would: (1) train a difficulty predictor on ScanNet/ARKitScenes scene graphs labeled with estimated pass@1 from the base model, (2) use this predictor to allocate chunk size, sliding window size, or even whether to activate the TTT layers at all (routing "easy" short sequences to standard attention), and (3) measure whether adaptive allocation recovers the paper's oracle-bin efficiency gains without the 2048-sample estimation cost. The key metric: total FLOPs (estimation + inference) vs. accuracy, compared against both uniform TTT and oracle difficulty allocation. If the predictor can be trained to achieve >80% bin classification accuracy from <16 frames, the compute-optimal streaming framework becomes immediately practical.

Scaling Spatial-TTT to larger base models to measure whether streaming memory benefits compound or saturate with parameter count. The paper uses Qwen3-VL-2B-Instruct exclusively. A natural extension would apply the identical architecture (hybrid TTT at 3:1 ratio, spatial-predictive mechanism, two-stage training) to Qwen3-VL-8B or InternVL3-8B and measure the absolute and relative gains on VSI-Bench, VSI-SUPER, and MindCube-Tiny. The critical question: does Spatial-TTT provide additive gains (e.g., +14 points on VSI-Bench regardless of base model performance) or multiplicative gains (e.g., larger relative improvement on weaker base models, smaller on stronger ones)? The paper's base model (Qwen3-VL-2B) scores 50.3 on VSI-Bench; Spatial-TTT adds 14.1 points. Qwen3-VL-8B scores 57.9 — would Spatial-TTT-8B score ~72 (additive) or ~77 (preserving the ratio)? The answer determines whether the approach is most valuable for democratizing spatial intelligence (making small models competitive) or for pushing the absolute frontier (making large models even better). A strong follow-up would also test cross-model transfer: can the spatial-predictive convolution kernels learned on Qwen3-VL-2B transfer to Qwen3-VL-8B, or must they be retrained? This would clarify whether the spatial-predictive mechanism learns model-specific or task-generic geometric features.

Combining streaming TTT memory with explicit geometric supervision during training to test whether the two approaches are complementary or substitutive. The paper argues that streaming memory can substitute for explicit geometry computation at inference, but does not test whether training with explicit geometric supervision (depth maps, surface normals, 3D bounding boxes) would further improve the fast-weight dynamics. A strong follow-up would extend the dense scene-description training with auxiliary geometric losses: for each chunk update, predict not just the scene description but also a coarse depth map or object-centric 3D positions from the fast-weight state. The hypothesis: geometric supervision provides a more direct gradient signal for the fast-weight updates to capture metric 3D structure, potentially improving numerical spatial tasks (Absolute Distance, Room Size, Object Size) where Spatial-TTT still trails human performance (Table 1: humans score 47.0 on Absolute Distance vs. Spatial-TTT's 47.8 — actually Spatial-TTT is slightly above human here; Room Size 65.9 vs. human 45.9 where Spatial-TTT substantially exceeds human; Object Size 71.7 vs. human 60.4, again exceeding human). The relevant comparison: Spatial-TTT + geometric auxiliary losses vs. base Spatial-TTT on VSI-Bench numerical tasks and VSI-SUPER-Count. If geometric supervision helps, it would refine the paper's claim — streaming memory and geometric features are complementary, not substitutive. If it doesn't help (or hurts, by distracting the fast weights from semantic scene understanding), it would strengthen the claim that streaming memory alone is sufficient and explicit geometry is unnecessary.

Ablation of the sliding window annealing schedule to determine whether the gradual handoff from SWA to TTT is necessary or merely helpful. The paper's Stage 1 training anneals the sliding window size from 5600 (covering the full 32-frame sequence) to 2648 (equal to the chunk size) over two epochs. This is intuitively sensible — the SWA provides a scaffold that is slowly withdrawn — but the paper provides no ablation testing whether the annealing matters. A strong follow-up would compare three training curricula: (1) the paper's annealing schedule (w: 5600 → 2648), (2) fixed w = 2648 from the start (no scaffold — TTT must handle cross-chunk propagation immediately), and (3) fixed w = 5600 throughout Stage 1 followed by an abrupt switch to w = 2648 in Stage 2 (no gradual handoff). The outcome would clarify whether the annealing is a training convenience (speeding convergence) or a necessity (without it, the TTT dynamics fail to learn cross-chunk propagation). If fixed w = 2648 from the start achieves comparable final performance, the training procedure can be simplified. If it fails (TTT dynamics collapse or converge to a poor local optimum), the annealing is essential and the paper should provide guidance on annealing rate, initial window size, and number of epochs — currently these are design choices without justification. A negative result (annealing doesn't matter) would be actionable: practitioners can skip the complexity. A positive result (annealing is essential) would establish it as a required component of the training recipe and motivate theoretical analysis of why TTT dynamics need the scaffold.

Streaming spatial memory for embodied navigation and manipulation — closing the loop between perception and action. The paper evaluates on passive spatial understanding benchmarks (VQA, recall, counting), but the motivation (Section 1) emphasizes embodied applications: robotics, autonomous driving, AR. A natural follow-up would deploy Spatial-TTT on an embodied navigation task where the model must accumulate spatial evidence from egocentric video while navigating, then use that accumulated memory to plan routes, locate objects, or answer spatial queries from arbitrary viewpoints reached during navigation. The Habitat simulator (or a real robot with a mounted camera) provides the streaming video input; the model's task is "navigate to the kitchen and describe the spatial layout of objects you passed." This would test whether the fast-weight memory genuinely builds a viewpoint-invariant 3D representation — the key claim of the spatial-predictive mechanism — or whether it primarily accumulates view-dependent features. The critical test: after navigating through several rooms, can Spatial-TTT answer spatial questions about a room it left 5 minutes ago as accurately as questions about the current room? If yes, the memory is genuinely persistent and viewpoint-invariant. If accuracy decays with time since last observation, the memory exhibits forgetting and the streaming accumulation claim needs qualification. The VSI-SUPER-Recall results (Table 3: 38.3 → 35.0 → 28.3 → 30.0 at 10 → 120 minutes) show some decay but not collapse, which is encouraging — but recall is a simpler task than spatial reasoning from memory, and the embodied setting would stress-test this more rigorously.

Failure mode analysis: what does Spatial-TTT's spatial memory lose, and when? The paper reports no qualitative error analysis — no examples of incorrect answers, no categorization of failure types, no analysis of whether errors stem from memory degradation, geometric misestimation, or semantic misunderstanding. A strong follow-up would systematically categorize failures on VSI-Bench and VSI-SUPER, asking: (1) Does Spatial-TTT make different types of errors than static baselines, or the same types at different rates? (2) When Spatial-TTT fails on a counting task (VSI-SUPER-Count), does it undercount (forgetting previously seen objects), overcount (double-counting objects seen from different viewpoints), or misidentify (confusing object categories)? The first indicates fast-weight memory decay; the second indicates failure of instance-level tracking; the third indicates semantic degradation from the pretrained backbone. (3) Does error rate correlate with specific video properties (camera speed, occlusion frequency, number of similar-looking objects, scene clutter)? This would identify the operational envelope for Spatial-TTT — the conditions under which streaming memory is reliable vs. when it breaks down. The VSI-SUPER-Count results showing 31.8% accuracy at 10 minutes (Table 3) mean the model is wrong ~68% of the time even at short durations — understanding why it's wrong is essential for improving the system. The ablation study (Table 4) tells us which components matter for average performance, but not which components matter for which failure modes. Do failures from removing the spatial-predictive mechanism look different from failures from removing dense supervision? If so, the error analysis would provide a finer-grained diagnostic for component contributions.

Extending the dense-description curriculum to video domains beyond indoor scenes. The SceneVerse-derived dense descriptions cover indoor environments with structured object layouts (rooms, furniture, appliances). Would the same curriculum work for outdoor navigation (street scenes, natural environments), dynamic scenes (moving objects, changing lighting), or first-person activity video (cooking, assembly)? The key question is whether the dense-description format generalizes or whether each domain requires a different auxiliary objective. For outdoor navigation, the dense description might include landmark enumeration, route descriptions, and distance estimates between waypoints. For activity video, it might include action step enumeration and object state changes. A strong follow-up would construct a dense-description dataset for a different domain (e.g., using the Ego4D dataset's narrated walking tours to generate "spatial walkthrough" descriptions via an LLM prompted with egocentric video metadata) and test whether the two-stage training (dense descriptions → domain-specific QA) transfers. If the curriculum transfers across domains with minimal adaptation, the paper's methodology is general. If it requires careful domain-specific design (what aspects of the scene to describe, at what granularity), the methodology is powerful but not plug-and-play.

Practical Applications and Downstream Use Cases

Long-duration spatial monitoring for security, retail, and infrastructure inspection. A security camera system monitoring a warehouse, retail store, or construction site generates continuous video streams over hours or days. Current video understanding systems either process short clips (missing long-term patterns) or subsample aggressively (missing critical events). Spatial-TTT's streaming architecture — which maintains constant memory usage regardless of video length (Table 5: 11.9 GB at 1024 frames, scaling approximately linearly) — enables continuous spatial monitoring where the model accumulates a running 3D understanding of the environment. Application: a system that monitors a retail store for 8 hours and can answer queries like "how many customers entered the electronics section between 2 PM and 4 PM?" or "was any merchandise removed from the display shelf in the last hour?" The VSI-SUPER-Count results (38.4% at 120 minutes, while all baselines score 0%) demonstrate that Spatial-TTT can sustain counting-type evidence accumulation over durations orders of magnitude longer than static models can process. The current 38.4% accuracy is not deployment-ready for high-stakes applications, but the qualitative difference (functional vs. non-functional) suggests that with improved training and larger base models, practical streaming monitoring becomes viable using this architecture while remaining impossible for standard approaches.

Embodied robot spatial memory for long-horizon household tasks. A home robot tasked with "tidy up the living room and bring any dirty dishes to the kitchen" must maintain a persistent spatial memory as it moves through rooms — remembering which surfaces had dishes, where cleaning supplies are located, and how rooms connect. Spatial-TTT's dual-branch design (SWA for local, precise spatial context; TTT for global, compressed temporal context) directly addresses the two types of spatial reasoning this requires: fine-grained manipulation (precisely locating a cup on a cluttered table) needs the local precision that SWA preserves; global navigation (planning a path from living room to kitchen via the hallway, remembering which door leads where) needs the global accumulation that TTT provides. The VSI-Bench Relative Direction score (73.0, Table 1, 15.5 points above the best proprietary baseline) suggests strong viewpoint-dependent direction reasoning — exactly what a robot needs to determine "the kitchen is to my left from this orientation." The MindCube-Tiny AMONG score (74.0, 14.7 points above the best proprietary baseline) suggests strong cross-view spatial reasoning — maintaining object relationships when viewpoints change, which a navigating robot continually experiences. A practical deployment would require integrating Spatial-TTT with a robot control stack and measuring task completion rate (not just VQA accuracy), but the benchmark results provide evidence that the core spatial reasoning capabilities are present.

Augmented reality devices requiring persistent environmental understanding across user movement. An AR headset overlaying digital information on the physical world must maintain a running 3D model of the environment as the user walks through rooms, with virtual objects correctly occluded by real furniture and annotations persisting on walls even after the user looks away and looks back. Current AR systems use simultaneous localization and mapping (SLAM) for geometric reconstruction, but SLAM produces low-level geometry (point clouds, meshes) without semantic understanding — it knows where surfaces are but not what they are. An MLLM with Spatial-TTT could complement SLAM by providing semantic spatial memory: recognizing that this flat surface is a "kitchen counter," remembering that the "red mug" was placed on it 10 minutes ago, and understanding that the "refrigerator" is to the left of the "sink" from the user's current viewpoint. The efficiency scaling (Table 5: Spatial-TTT uses ~44% less memory and TFLOPs than standard attention at 1024 frames) is critical for AR, where devices have severe power and thermal constraints and cannot run multi-billion-parameter models with quadratic attention over hours of continuous use. The VSI-Bench object count (70.8) and object size (71.7) results suggest reliable object enumeration and metric estimation from streaming video — practical prerequisites for AR annotation that persists correctly as objects enter and leave the field of view.

When to Prefer This Method

The paper does not position Spatial-TTT against named alternative architectures in an explicit decision framework. However, the experimental results provide clear guidance on the deployment regimes where Spatial-TTT's architectural choices are advantageous versus where standard approaches suffice. Based on the evidence in Tables 1–5:

Prefer Spatial-TTT (streaming TTT memory) when:

Videos exceed ~256 frames (roughly 4 minutes at 1 fps, or 8 seconds at 30 fps). At 256 frames, Spatial-TTT already shows ~15% compute and memory savings over standard attention (Table 5: 7.0 GB / 156.2 TFLOPs vs. 8.3 GB / 179.9 TFLOPs), and this gap widens to ~43% at 1024 frames. Below 128 frames, the overhead of chunk management and dual KV caches may outweigh the efficiency benefit — the base model Qwen3-VL-2B is actually slightly more efficient at this scale.
The task requires accumulating evidence across temporal gaps — counting objects that appear at different times (VSI-SUPER-Count, where baselines score 0%), recalling appearance order (VSI-SUPER-Recall, where baselines crash), or answering questions about spatial relationships between objects seen at different points in the video. Standard models can handle these tasks only if all relevant evidence fits in a single forward pass.
Deployment hardware has fixed memory constraints (e.g., 80 GB GPU, edge device) and videos are arbitrarily long. Spatial-TTT's constant memory usage means it can process video streams indefinitely; standard attention's KV cache grows linearly until it exhausts memory, as demonstrated by Qwen3-VL-2B and Cambrian-S-7B running out of memory at 120 minutes on VSI-SUPER.
The base model is pretrained and its visual-semantic knowledge must be preserved. The hybrid architecture with anchor layers provides this; pure TTT or other full-architecture replacements would require expensive retraining to recover alignment.

Prefer standard attention (or spatial-augmented models with geometry encoders) when:

Videos are short (<128 frames, <2 minutes at 1 fps) and can fit in a single forward pass. The overhead of TTT chunk management, dual KV caches, and gradient-based updates provides no benefit when the full video is processable by standard attention at comparable cost (Table 5: 74.3 vs. 75.9 TFLOPs at 128 frames).
The task is rotation reasoning (MindCube-Tiny ROTATION: Spatial-TTT scores 55.5 vs. GPT-5's 94.5, Table 2) — this requires learned 3D object representations that larger models acquire through massive pretraining, not streaming evidence accumulation. Spatial-TTT's architecture optimizes for scene-level spatial memory, not object-centric mental rotation.
Route planning is critical (VSI-Bench Route Plan: Spatial-TTT scores 47.4 vs. Gemini-3-Pro's 61.9, Table 1) — this task requires multi-step planning from a fixed set of observations, and the best proprietary models substantially outperform Spatial-TTT. Streaming memory may not help when the full route is visible in a short sequence, and larger pretrained reasoning capabilities dominate.
Compute budget is unlimited and latency is not a concern — if you can afford to run a very large model (e.g., Gemini-3-Pro) with full attention over the entire video, the absolute accuracy ceiling may be higher than Spatial-TTT on certain tasks, though the paper shows Spatial-TTT-2B already exceeds all proprietary baselines on aggregate VSI-Bench (64.4 vs. 56.0 for Gemini-3-Pro).

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Streaming Video Intelligence Requires Continual Spatial Memory

Why This Problem Matters: Spatial Intelligence as a Cornerstone Capability

Where Prior Approaches Fall Short

1. General-Purpose MLLMs: Lack 3D Geometric Priors

2. Spatial-Aware MLLMs: Stronger but Bottlenecked at Short Contexts

3. Test-Time Training for Language: Success but Limited Application to Vision

Reconciling Conflicting Needs: The Challenge This Paper Takes On

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Test-Time Training Preliminaries: The Update-Apply Mechanism

Hybrid TTT Architecture: Preserving Pretrained Knowledge While Achieving Linear Complexity

Large-Chunk Updates and Sliding Window Attention: Making TTT Practical for Visual Streams

Spatial-Predictive Mechanism: Injecting Geometric Inductive Bias into TTT Updates

Fast-Weight Network Design and the Muon Update Rule

Dense Scene-Description Dataset: Bridging Sparse Spatial QA with Rich Supervision

Spatial-Aware Progressive Training Strategy

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Test-Time Training as a Streaming Spatial Memory Mechanism — Not Just an Efficiency Hack

Innovation 2: The Hybrid Architecture as a Solution to the Pretrained-Knowledge Catastrophic Forgetting Problem in TTT for MLLMs

Innovation 3: Difficulty-Agnostic Streaming Architecture via Complementary Local-Global Decomposition

Innovation 4: Dense Scene Descriptions as a Curricular Bridge from Generic Memory to Task-Specific Spatial Recall

Innovation 5: The Spatial-Predictive Mechanism as a Minimalist Injection of Geometric Inductive Bias

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

VSI-Bench: General Spatial Understanding

MindCube-Tiny: Fine-Grained Multi-View Spatial Reasoning

VSI-SUPER: Streaming Spatial Sensing in Long-Horizon Video

Computational Efficiency Scaling (Table 5)

Ablation Studies and Robustness Checks

Critical Assessment

Claim 1: Spatial-TTT achieves state-of-the-art performance on video spatial benchmarks.

Claim 2: Test-time training enables continual spatial evidence accumulation over arbitrarily long videos.

Claim 3: The hybrid architecture (3:1 TTT-to-attention ratio) is necessary to preserve pretrained visual-semantic knowledge.

Claim 4: The dense scene-description dataset teaches fast-weight dynamics to retain globally useful spatial evidence.

Claim 5: The spatial-predictive mechanism improves geometric correspondence and temporal continuity.

Overall Assessment of Experimental Rigor

6. Limitations and Trade-offs

The Difficulty Estimation Bottleneck: 2048 Samples Per Question Makes the Framework Impractical for Deployment

The Revision Model's 38% Correct-to-Incorrect Reversion Rate Is a Fundamental Stability Problem

Test-Time Compute Cannot Create Capability Where None Exists: Hard Problems Are Unsolved Regardless of Budget

The Test Set Is 500 Questions, Strategy Selection Uses ~50 Questions Per Bin Per Fold, and No Confidence Intervals Are Reported

All Results Are on a Single Model Family (PaLM 2-S*) and a Single Benchmark (MATH), With No Evidence of Generalization

Sequential Revisions Are Inherently Serial, Creating a Latency-vs-Accuracy Tradeoff the Paper Does Not Address

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

GLM-OCR Technical Report

Qianfan-OCR: A Unified End-to-End Model for Document Intelligence