VGG-T$^3$: Offline Feed-Forward 3D Reconstruction at Scale

🎯 Pitch

By compressing a Transformer’s quadratic-cost key-value scene memory into a small MLP optimized at test time, VGG-T3 turns multi-view 3D reconstruction into a linear-time operation—rebuilding Rome's Colosseum from 1,000 tourist photos in under a minute, 11× faster than the teacher model.

1. Executive Summary

This paper introduces VGG-T³ (Visual Geometry Grounded Test Time Training), a method that converts a pre-trained multi-view 3D reconstruction model with quadratic attention complexity into a linear-time alternative by replacing the variable-length Key-Value (KV) scene representation with a fixed-size MLP optimized via test-time training (compressing the K→V mapping into compact MLP weights). Evaluated against the VGGT architecture on standard benchmarks (NRGBD, DTU, ETH3D, 7scenes) as well as large-scale reconstructions of up to 2,000 images, the approach achieves an 11.6× speedup at 1k input views (reconstructing them in 58 seconds versus VGGT's 11 minutes) while outperforming the only other linear-time baseline, TTT3R, by 2–2.5× on pointmap estimation error. The method further enables feed-forward visual localization by freezing the optimized MLP and querying it with novel views, establishing that linear-time global reconstruction with a compressed implicit scene representation is feasible, though a gap to softmax attention persists on complex wide-baseline scenes.

2. Context and Motivation

The Core Problem: Quadratic Attention Bottleneck in Feed-Forward 3D Reconstruction

The fundamental problem VGG-T³ tackles is deceptively simple: how do we reconstruct 3D geometry from large, unordered image collections without the computational cost exploding as we add more images? The paper focuses on feed-forward methods—neural networks that directly predict 3D structure from images in a single forward pass, without iterative optimization like classical structure-from-motion (SfM) pipelines. These feed-forward approaches have recently achieved remarkable accuracy, matching or exceeding classical methods like COLMAP while being more robust to challenging conditions such as rapid camera motion and low visual overlap (Section 1).

However, this accuracy comes at a steep price. The state-of-the-art multi-view feed-forward architectures—exemplified by VGGT (Visual Geometry Grounded Transformer), π³, and Fast3R—all rely on global self-attention to aggregate information across all input views simultaneously. This is what gives them their power: every image token can attend to every other image token, allowing the model to reason about the full scene geometry holistically. But the mechanism that enables this global reasoning is also the bottleneck. As the paper explains in Section 1:

"This bottleneck originates from the implicit scene-level memory stored in the Key-Value (KV) space of the global self-attention layer. This KV space, projected from all input image tokens, functions as the dense, variable-length scene representation queried for 3D attribute prediction."

The problem is that querying this KV space requires $O(n^2)$ operations relative to the number of input images $n$ . Every token must compute attention scores against every other token. For modest image collections (say, 24 images, which is what VGGT was trained on), this is manageable. But for realistic large-scale reconstruction—think "Building Rome in a Day" scenarios with thousands of tourist photos of landmarks—the quadratic scaling becomes prohibitive.

The paper quantifies this starkly: reconstructing 1,000 images takes VGGT approximately 11 minutes, while for 2,000 images the runtime balloons to 27 minutes. This is not a constant-factor issue; it's an asymptotic wall. You cannot simply throw more compute at it and expect linear returns.

Why This Matters: The Gap Between Capability and Practicality

The significance of this bottleneck extends beyond academic benchmarking into several real-world deployment scenarios:

Tourism-scale reconstruction. The paper's teaser figure (Figure 0) shows reconstructions of Roman landmarks—the Colosseum, Castel Sant'Angelo, the Pantheon, and the Trevi Fountain—from tourist-sourced images. These are exactly the kind of large, unordered image collections that feed-forward methods should excel at processing. The images come from different cameras, different viewpoints, different lighting conditions, and different decades. There is no natural ordering. Classical SfM pipelines can handle these collections, but they involve multi-stage processing (feature extraction, matching, incremental reconstruction, bundle adjustment) that is brittle to failures in any single stage and can take hours or days. Feed-forward methods promise to collapse this into a single network forward pass, but only if the network can actually process thousands of images without running out of memory or time.

Autonomous driving and robotics. Long video sequences from self-driving cars or drones can easily span thousands of frames covering kilometer-scale routes. While these are ordered sequences rather than unordered collections, the underlying computational challenge is the same: global reasoning across all frames improves consistency and reduces drift, but quadratic attention makes this impractical for sequences beyond a few hundred frames. Methods like VGGT-Long and SLAM3R address this by processing in chunks with sliding windows, but this decouples the global state and introduces drift—exactly the problem that global attention was supposed to solve.

Edge deployment and cost constraints. Quadratic scaling also has hardware implications. Even with large GPUs (80GB A100s), the paper notes that VGGT with FlashAttention still requires all q, k, v vectors to be in GPU memory simultaneously, which "even for large GPUs, leads quickly to out-of-memory errors when processing larger image collections" (Section 3.3). This makes feed-forward reconstruction inaccessible on consumer hardware and expensive in cloud deployments, limiting its practical reach.

Where Prior Approaches Fall Short

The paper identifies four categories of prior work and explains why each fails to fully solve the scalability problem:

1. Sparse attention variants (FastVGGT, SparseVGGT) reduce the constant factor but not the asymptotics.

These methods compress the KV representation before applying attention. FastVGGT uses token merging—combining similar tokens to reduce the effective sequence length. SparseVGGT employs block-sparse attention, attending only within local neighborhoods. As the paper explains (Section 2):

"While reducing the constant factor $O(n^2) \to O(n/r^2)$ where $r$ is the token down-sampling ratio, the asymptotic complexity of both approaches remains quadratic."

In other words, they make the model faster by a constant factor (the paper reports FastVGGT takes about 4.3 minutes for 1k images versus VGGT's 11 minutes), but the curve still bends upward quadratically. Double the images, and the runtime more than doubles. These are valuable optimizations, but they don't change the fundamental scaling law.

A deeper issue is that these methods rely on heuristics about which tokens are "similar" and can be merged safely. The paper notes they assume "tokens close in image space share similar scene features." This is often true, but when it fails—for instance, when two distant images happen to capture the same object from opposite sides—the compression loses information that global attention would have preserved.

2. Chunked/sliding window methods (VGGT-Long, SLAM3R, VGGT-SLAM) sacrifice global consistency.

These approaches process long sequences by breaking them into overlapping chunks, running local attention within each chunk, and then aligning the chunks post-hoc. This is effective for video data where frames are temporally ordered, but the paper identifies a fundamental limitation (Section 2):

"this decouples the global scene state, making them prone to drift and unsuitable for unordered image sets."

The drift problem is well-known in SLAM literature: small errors in chunk alignment accumulate over long sequences, causing the reconstruction to warp or diverge. For unordered image collections (like tourist photos of a landmark), there is no natural chunking order, making these methods inapplicable without additional heuristics to determine which images should be processed together. The paper's goal is a method that works on both ordered and unordered inputs without this limitation.

3. Online autoregressive methods (CUT3R, Must3R, TTT3R, StreamVGGT) process sequentially but either scale quadratically or lose accuracy.

Several recent methods process images one at a time, maintaining a persistent state that represents the scene seen so far. StreamVGGT and Stream3R convert pre-trained VGGT models to causal models, but the paper notes they still "scale quadratically w.r.t. $n$ and require memory-intensive KV caching to accelerate inference" (Section 2). They've changed the processing order but not the underlying attention mechanism.

Others, like CUT3R and TTT3R, use fixed-size implicit memory updated iteratively. These genuinely achieve linear complexity, but the paper's experiments show they pay a significant accuracy penalty. On the NRGBD dataset, TTT3R—the concurrent method most similar to VGG-T³ in spirit—shows 2–2.5× higher Chamfer distance error compared to VGG-T³ (Table 1). On video depth estimation, TTT3R performs substantially worse on two of three datasets (Table 2). The qualitative comparisons in Figure 5 are particularly revealing: TTT3R's reconstructions show visible degradation and inconsistency, while VGG-T³ produces coherent 3D structure approaching VGGT's quality.

The fundamental tradeoff is that autoregressive methods can only attend backward in time (to previously processed frames). They cannot look ahead to future frames for additional context. This is fine for streaming applications but suboptimal for offline reconstruction where all images are available upfront. The paper's key insight is that offline, bi-directional global reasoning can be achieved in linear time by compressing the scene representation rather than restricting the attention pattern.

4. Classical SfM pipelines work but are slow and brittle.

The paper acknowledges that established SfM methods (COLMAP, GLOMAP, Bundler) can handle large image collections, given sufficient visual overlap and connectivity. However, their multi-stage nature makes them vulnerable to cascading failures: if feature matching produces insufficient correspondences between a critical subset of images, the entire reconstruction can fragment. Feed-forward methods are empirically more robust because they learn to handle such ambiguities directly from data, rather than relying on explicit geometric heuristics.

The Paper's Position: Compression Over Approximation

The paper positions its contribution at a specific point in the design space. Rather than:

Approximating attention (FastVGGT, SparseVGGT)—which keeps the quadratic structure but reduces the constant, or
Restricting attention to local or causal windows (chunked methods, autoregressive methods)—which achieves linear complexity but sacrifices global context, or
Training from scratch with linear attention mechanisms—which the paper shows gets "stuck in a local optimum" (Table 6, ablation i),

VGG-T³ proposes a fundamentally different approach: compress the variable-length KV scene representation into a fixed-size MLP via test-time optimization. The paper frames this through a conceptual analogy to DeepSDF (Section 1):

"Intuitively, the fixed-state decoder learns rich geometric priors, while a small latent code encodes instance-specific details through test-time optimization. In this work, we revisit this core principle in the context of feed-forward multi-view 3D reconstruction."

In DeepSDF, a pre-trained shape decoder takes a compact latent code (optimized at test time) and reconstructs a specific shape. The decoder contains general knowledge about shapes; the latent code captures what makes this particular shape unique. VGG-T³ applies the same principle: the pre-trained VGGT model (encoder, per-image attention, prediction heads) contains general knowledge about 3D reconstruction from images; the MLP weights optimized at test time compress the specific scene geometry from the observed views.

This is a critical distinction from other linearization approaches in the language model literature (T2R, LoLCats, Mamba) which the paper acknowledges as related work (Section 2). Those methods typically replace attention with recurrent state updates during a forward pass. VGG-T³ instead performs an explicit optimization at test time, treating the MLP weights as "fast weights" (in the terminology of Hinton & Plaut, 1987) that are learned specifically for each scene. This gives the method more expressive power than a fixed recurrence because the optimization can take multiple steps to fit the scene representation.

The paper draws a direct connection to Sun et al.'s TTT framework (2025), which showed that test-time training provides "a strictly more general framework" that "recover[s] linear attention and SSMs as special cases while improving adaptability across domains." VGG-T³ extends this insight from language and video modeling to multi-view 3D reconstruction, demonstrating that the TTT mechanism can compress bi-directional transformer KV states as effectively as it can handle sequential recurrences.

Reconciling a Tension: Global Reasoning vs. Linear Complexity

At a meta-level, the paper addresses what might seem like a contradiction: how can you have global reasoning (every image influences every other image) with linear complexity? The answer lies in separating when the global information is aggregated.

In standard softmax attention, global aggregation happens at query time: every output token is computed as a weighted sum over all input tokens' values, with weights determined by query-key similarity. The complexity is $O(n^2)$ because each of the $n$ output positions requires computing attention over all $n$ input positions.

In VGG-T³, global aggregation happens at test-time optimization time: the MLP is trained to map keys to values across all tokens simultaneously. The optimization sees all tokens and compresses the global mapping into its weights. Then, at query time, producing the output for a single token is just an MLP forward pass—an $O(1)$ operation per token, or $O(n)$ for the whole sequence.

The paper expresses this in Equations 3 and 4:

$\operatorname*{arg\,min}_{\theta} \sum_i L_t(T_\theta(k_i) - v_i) \quad \text{(optimization)}$

$o_i = T_\theta(q_i) \quad \text{(query)}$

Both operations are linear in $n$ . The optimization step sums over all $i$ tokens to compute the gradient (which can be done in minibatches, as Section 3.3 exploits for distributed inference), and the query step is just $n$ independent MLP forward passes. The "global" nature of the computation is preserved because the MLP's weights encode information from all tokens—but the per-token cost is constant.

This framing positions VGG-T³ not as a compromise (sacrificing global reasoning for speed) but as a representation change: the same information is stored differently, in a way that is cheaper to access. The paper's ablation studies (Table 6) show this change does incur an accuracy cost compared to softmax attention—the MLP representation is less expressive than the full KV cache—but the gap narrows with increasing image count, and the speedup is transformative (33× faster at 2k images).

The Broader Vision: Unified Mapping and Localization

Beyond the scalability contribution, the paper positions its MLP-based scene representation as enabling a unified framework for reconstruction and visual localization. Traditionally, these are separate tasks requiring separate solutions: first reconstruct the scene (via SfM or feed-forward methods), then build a separate localization pipeline (feature matching, PnP solving, or scene coordinate regression) to locate new query images within that reconstruction.

Because VGG-T³'s MLP stores a compressed scene representation, it can be frozen after optimization and queried with novel views. The paper shows this works in Section 4.3: the same model that reconstructed the scene can localize new images within it, simply by running a forward pass through the frozen MLPs without updating them. This is a proof-of-concept demonstration—the localization accuracy doesn't match specialized methods like Reloc3R (which achieves $e_r = 1.02^\circ, e_t = 0.04\text{m}$ on 7scenes)—but it establishes that the representation is general enough to support query-time generalization, a capability that the quadratic-attention baselines cannot offer without additional mechanisms.

3. Technical Approach

3.1 Reader Orientation

VGG-T³ is a system that takes a pre-trained multi-view 3D reconstruction model (VGGT) and surgically replaces its quadratic-complexity global attention layers with a linear-time alternative based on test-time training, without retraining the entire model from scratch. The problem it solves is that VGGT's global attention over all image tokens scales as $O(n^2)$ with the number of input views, making it impractical for large image collections—VGG-T³ compresses the variable-length Key-Value scene representation into a fixed-size MLP whose weights are optimized per-scene at inference time, reducing complexity to $O(n)$ while preserving the ability to reason globally across all views.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

Pre-trained VGGT backbone (frozen) — the image tokenizer, per-image self-attention layers, and prediction heads from the original VGGT model. These are kept frozen; they convert input images into token sequences and decode output tokens into depth maps, camera poses, and intrinsics.
Global attention replacement (trained) — the quadratic softmax attention layers in VGGT's global attention blocks are replaced with TTT layers. Only the QKV projection matrices, output projections, and newly introduced TTT module parameters are fine-tuned; everything else stays frozen.
Test-time training (TTT) module — a SwiGLU MLP with learnable parameters $\theta$ that is optimized at inference time to map keys $K$ to values $V$ across all input tokens. This MLP constitutes the compressed, fixed-size scene representation.
ShortConv2D spatial mixing — a lightweight 2D convolution applied to the Value tokens before the TTT optimization, which aggregates local spatial context and breaks the linear dependency between keys and values, forcing the MLP to learn a more expressive mapping.
Inference strategies — mechanisms for flexibly deploying the optimized model: single-GPU processing via CPU offloading of minibatches, multi-GPU distributed inference via gradient synchronization, and query-mode inference for visual localization (freeze MLP weights, apply only the query step of TTT to novel views).

Information flows as follows: input images → frozen VGGT tokenizer (produces token sequences) → alternating per-image self-attention (frozen) and TTT-based global aggregation (active, with MLP optimization) → frozen VGGT prediction heads → per-image depth maps, camera poses, intrinsics. During the forward pass, each global attention layer executes a two-stage cycle: (1) Update — project tokens to QKV, optimize the TTT MLP on the current layer's $(K, V)$ pairs to compress the scene information, and (2) Apply — query the optimized MLP with $Q$ to produce output tokens for downstream layers.

3.3 Roadmap for the Deep Dive

First, the VGGT attention mechanism in detail (Equations 1–2), because VGG-T³'s architecture is defined by what it replaces, and understanding the quadratic bottleneck requires seeing exactly how softmax attention creates it.
Second, the TTT framework from Sun et al. (2025) (Equations 3–4), the mathematical foundation that VGG-T³ adapts, establishing how learning a $K \to V$ mapping can replace attention and why it yields linear complexity.
Third, the linearization strategy — how the pre-trained VGGT is converted to a TTT model, including the critical removal of LayerNorm, the $L_2$ normalization substitution, and why post-training linearization empirically dominates training from scratch.
Fourth, ShortConv2D spatial mixing — the mechanism that prevents the TTT objective from degenerating into a trivial linear mapping, implemented as a 2D convolution on Value tokens before optimization.
Fifth, test-time scaling via optimizer steps — the empirical finding that out-of-distribution sequence lengths require more optimization steps, and how this enables constant scaling to arbitrary image counts.
Sixth, the MLP architecture and optimization details — the SwiGLU MLP, dot product loss, Muon optimizer, and training hyperparameters that make the system work.
Seventh, inference strategies for scale — the minibatch gradient decomposition (Equation 5) that enables single-GPU CPU offloading and distributed multi-GPU inference, along with the query-mode mechanism for visual localization.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and representation paper whose core idea is that the global attention operation in multi-view reconstruction transformers can be replaced by a per-layer, per-scene test-time optimization of a compact MLP, converting quadratic complexity to linear while preserving global information flow.

VGGT's Global Attention Mechanism (What We Are Replacing)

The starting point is VGGT's alternating attention architecture, which is essential to understand because VGG-T³ surgically replaces one specific component while keeping everything else intact.

Tokenization. Each input image $\mathcal{I}_i \in \mathbb{R}^{H \times W \times 3}$ is processed by a frozen image tokenizer (a Vision Transformer encoder) that converts it into a sequence of patch tokens. With a patch size $p = 14$ and input resolution $518 \times 518$ , each image produces approximately $(518/14)^2 = 1369$ tokens. For $N$ images, the total token count is $N \times 1369$ , plus additional tokens (camera tokens, register tokens) appended for specific tasks. These tokens form the input sequence that flows through the transformer layers.

Alternating attention blocks. VGGT's transformer consists of blocks that alternate between two types of attention (Figure 1a):

Per-image self-attention: tokens within each image attend only to other tokens from the same image. This captures local geometric structure within individual views—edges, surfaces, object boundaries.
Global self-attention: tokens from all images attend to tokens from all other images indiscriminately. This is where cross-view reasoning happens—the model learns that a feature in image A corresponding to a window corner should relate to a feature in image B corresponding to the same window corner seen from a different angle.

It is the global self-attention layers that create the quadratic bottleneck and are the target of replacement. The per-image attention layers are left completely untouched because they operate independently per image and already scale linearly.

QKV projection with normalization. Within each global attention layer, every input token $x_i$ (representing a patch from some image) is projected into query, key, and value vectors:

$q_i = \text{LN}_q(W_q x_i), \quad k_i = \text{LN}_k(W_k x_i), \quad v_i = W_v x_i$

where $W_q$ , $W_k$ , and $W_v$ are learned linear projection matrices, and $\text{LN}_q$ and $\text{LN}_k$ are LayerNorm operations performing QK normalization.

What this equation computes: three distinct vector representations for each token. The query $q_i$ represents "what information this token is looking for," the key $k_i$ represents "what information this token contains that might be relevant to others," and the value $v_i$ represents "the actual content this token will contribute if attended to." The LayerNorm operations on queries and keys normalize their distributions, which the paper cites (Dehghani et al., 2023; Henry et al., 2020) as being important for stabilizing softmax attention training. The value projection notably does NOT apply LayerNorm—it goes through only the linear transformation.

Why this form: the separation into Q, K, V is standard transformer design (Vaswani et al., 2017). QK normalization is added because in deep vision transformers with many layers and tokens, the dot products $q_i^T k_j$ can grow in magnitude, causing the softmax to saturate (producing near-one-hot attention distributions). LayerNorm keeps the magnitudes controlled so the softmax operates in a regime where it can produce graded attention weights. The asymmetry—normalizing Q and K but not V—is because the value vectors need to preserve magnitude information for the weighted sum; normalizing them would discard scale information about feature importance.

Softmax attention. The per-head output $o_i$ for token $i$ is computed as:

$o_i = \sum_j \text{softmax}_j\left(\frac{q_i^T k_j}{\sqrt{d}}\right) v_j$

where the softmax is taken over all tokens $j$ in the sequence, and $d$ is the dimensionality of the key vectors (the scaling factor $\sqrt{d}$ prevents dot products from growing with dimensionality).

What this equation computes: for a given query token $i$ , compute its dot-product similarity with every key $k_j$ in the entire sequence (all tokens from all images), scale by $1/\sqrt{d}$ , convert to a probability distribution via softmax, and then produce $o_i$ as the weighted sum of all value vectors, with weights proportional to how relevant each key is to the query. This is fundamentally a content-based retrieval operation: the query says "I'm looking for information about X," the keys advertise "I contain information about Y," the dot product measures how well X matches Y, and the output retrieves the values from the best-matching keys.

Why this form: the softmax over all tokens is what makes this "global" attention—every token can influence every other token's output. The $\sqrt{d}$ scaling (originally from Vaswani et al., 2017) is a variance-stabilizing factor: if the components of $q$ and $k$ have unit variance and are independent, then $q_i^T k_j$ has variance $d$ , so dividing by $\sqrt{d}$ keeps the input to the softmax at a scale where the exponential doesn't blow up or collapse to zero. Without this scaling, attention would become either too peaked (for large $d$ ) or too uniform (for small $d$ ).

The quadratic bottleneck. The critical issue is the summation over $j$ : for each of the $n$ output positions, the softmax must be computed over all $n$ input positions, requiring $O(n)$ operations per output position, or $O(n^2)$ total. The Key and Value tensors for the entire sequence must be stored in memory because every query needs access to every key and value. For $N$ images at resolution $518 \times 518$ with patch size 14, $n \approx 1369N$ (plus overhead tokens), and the attention cost grows as $(1369N)^2$ . At 24 images (VGGT's training sequence length), $n \approx 32,856$ , which is manageable. At 1000 images, $n \approx 1.37$ million, and the attention cost is roughly $(1.37 \times 10^6)^2 / (3.29 \times 10^4)^2 \approx 1736$ times higher.

The paper also notes a subtle detail: VGGT appends a dedicated camera token to the image tokens immediately before the attention layer. This token participates in the global attention but has a special role—it aggregates information for downstream camera pose prediction. This heterogeneous structure (image patch tokens vs. camera tokens as two different "modalities" within the same attention operation) becomes a challenge for the MLP in VGG-T³, as discussed in Section 4.1's camera pose estimation results.

The Test-Time Training (TTT) Framework Adapted from Sun et al. (2025)

The mathematical core that VGG-T³ adopts is Sun et al.'s reformulation of attention as a learning problem. Understanding this reformulation is essential because it is the exact mechanism VGG-T³ uses to replace softmax attention.

The key insight. In standard attention (Equation 2), the relationship between keys and values is implicit: the attention weights determine which values are retrieved for each query, but the keys and values themselves are just vectors in a shared space. Sun et al. observed that you can make this relationship explicit by learning a function that directly maps keys to values. If such a function exists, then querying it with a query $q_i$ (which lives in the same space as keys, since both are projections of the same input token $x_i$ ) should retrieve the appropriate value.

The TTT optimization objective. The framework replaces the softmax attention operation with a two-stage process. First, an optimization step learns a function $T_\theta$ that maps keys to values:

$\operatorname*{arg\,min}_{\theta} \sum_i L_t(T_\theta(k_i) - v_i)$

where $L_t$ is a loss function measuring the discrepancy between the MLP's prediction $T_\theta(k_i)$ and the true value $v_i$ , and the sum runs over all tokens $i$ in the sequence.

What this equation computes: given all $(k_i, v_i)$ pairs from the current layer and current input sequence, find parameters $\theta$ of a neural network $T_\theta$ (an MLP) that minimize the total loss across all tokens when the network tries to predict each token's value vector from its key vector. This is a supervised regression problem where the keys are inputs and the values are targets. The optimization is performed at test time, on the specific input being processed—hence "test-time training."

Why this form: the sum over $i$ means that the MLP sees ALL tokens during optimization, so it can learn a global mapping that captures relationships across the entire scene. The form is chosen to be decomposable as a sum of per-token losses because this property enables the minibatch and distributed inference strategies (Equation 5, discussed later). If the objective were not a sum over independent per-token terms, gradient accumulation across minibatches would not be straightforward.

The query step. Once $\theta$ is optimized, retrieving the output for any query token is a simple forward pass:

$o_i = T_\theta(q_i)$

What this equation computes: for each token, apply the optimized MLP to the token's query vector to produce the output that would have been obtained through the weighted sum of values in standard attention. The MLP has internalized the mapping from the key space to the value space, and since queries and keys live in the same space (both are linear projections of the same input token $x_i$ ), querying with $q_i$ retrieves what the MLP learned about tokens similar to $q_i$ .

Why this form: both the optimization (Equation 3) and the query (Equation 4) are $O(1)$ per token with respect to the sequence length. The MLP forward pass takes constant time regardless of how many tokens were used to train it. This is the fundamental source of linear complexity: training the MLP sees all $n$ tokens (cost $O(n)$ ), and querying with all $n$ tokens also costs $O(n)$ , but neither step involves pairwise interactions between tokens.

The dot product loss. The paper specifies that VGG-T³ uses a dot product loss for $L_t$ :

$L_t(T_\theta(k_i), v_i) = T_\theta(k_i)^T v_i$

This is a maximization objective (the paper's arg min formulation implies a negative sign, or equivalently maximizing the dot product). The MLP is trained to produce an output that has high dot product with the true value vector.

What this computes: the cosine similarity (up to magnitude) between the MLP's prediction and the ground-truth value. Maximizing this encourages the MLP to produce vectors pointing in the same direction as the true values.

Why this form: the paper follows Zhang et al. (LaCT, 2025) in using dot product loss rather than, say, MSE. MSE would force the MLP to match both the direction and magnitude of the value vectors exactly, which is a harder optimization problem and may not be necessary—what matters for downstream layers is primarily the direction of the output tokens, as LayerNorm will renormalize magnitudes anyway. The dot product loss is also computationally simpler (no squared differences, just a dot product and a sign flip for minimization).

Connection to fast weights. The paper explicitly connects $\theta$ to the concept of "fast weights" introduced by Hinton & Plaut (1987): "weights that are optimized at train and test time." The key distinction from standard neural network weights ("slow weights") is that fast weights are transient—they are computed specifically for the current input and discarded afterward, rather than being learned across many training examples and stored permanently. In VGG-T³, every new scene gets its own set of MLP weights optimized from scratch during the forward pass.

Linearizing the Pre-Trained VGGT Model

The paper does not train a linear-time model from scratch. Instead, it converts a pre-trained VGGT checkpoint into a linear-time model through a fine-tuning process that leverages the existing weights. This decision is both practical (12% of the cost of training VGGT from scratch) and empirical (training from scratch with TTT "gets stuck in a local optimum," per Table 6).

Strategy overview. The conversion process has three phases: (1) architectural replacement—swap global softmax attention layers for TTT layers, (2) weight initialization—transfer QKV projection matrices from the pre-trained model into the new architecture, and (3) fine-tuning—train only the parameters in the global attention layers (QKV projections, output projections, TTT MLPs) while keeping all other parameters frozen, using a dataset comparable to VGGT's original training data for 100k steps on 8 NVIDIA A100-80GB GPUs.

What is frozen and what is trained. The frozen components include: the image tokenizer (ViT encoder), all per-image self-attention layers, and all prediction heads (depth, camera pose, intrinsics). The trained components are: the $W_q$ , $W_k$ , $W_v$ projection matrices in the global attention layers, the output projection matrices that follow the attention operation, and the newly introduced SwiGLU MLPs (one per global attention layer) that implement $T_\theta$ .

Why freeze most of the model: the tokenizer, per-image attention, and prediction heads represent general visual knowledge and geometric reasoning capabilities learned during VGGT's pretraining on diverse multi-view data. These capabilities are not specific to the attention mechanism—they would be equally useful regardless of how cross-view information is aggregated. Freezing them preserves this knowledge and dramatically reduces the number of parameters that need to be trained (and the amount of training data required). The 12% cost figure comes from training only ~100k steps versus the full VGGT training, which presumably requires ~800k+ steps.

The LayerNorm removal (critical insight). The paper identifies a specific obstacle to naive linearization: the LayerNorm operations in the QK projections (Equation 1). Section 3.2 states:

"LayerNorm involves additional learnable parameters that distort the input space that the MLP is trying to learn at test time."

The problem is subtle. LayerNorm normalizes its input to have zero mean and unit variance, then applies a learned affine transformation (scale $\gamma$ and shift $\beta$ ). This means that the keys $k_i$ that the TTT MLP sees during optimization are not simply linear projections of the input tokens—they have been normalized and transformed by $\gamma$ and $\beta$ , which are learned parameters from the pre-trained model. The MLP is supposed to learn the mapping from the key space to the value space, but the key space has been "warped" by LayerNorm's learned parameters in ways that may not be stable or meaningful for the TTT optimization objective.

The solution is to remove LayerNorm from the QK projections and instead apply $L_2$ normalization. $L_2$ normalization simply scales each vector to unit length without any learned parameters: $\hat{k}_i = k_i / \|k_i\|_2$ . This preserves the directional information in the keys (which is what matters for the mapping to values) without introducing parameterized distortions. The paper reports that this change "unlock[s] fast convergence from pre-trained weights."

Why $L_2$ normalization specifically: it is the simplest normalization that removes scale variation without introducing learned parameters. Alternatives like batch normalization would require running statistics, which is ill-defined at test time when processing a single scene. LayerNorm without learned parameters (just the normalization step, no affine transform) would also work but is essentially equivalent to $L_2$ norm on the directional component. $L_2$ normalization has the additional property that it projects keys onto the unit hypersphere, which may help the MLP learn a smoother mapping since all inputs have the same norm.

Why post-training linearization beats training from scratch. Table 6 reports that training a model with TTT from scratch (variant i) achieves substantially worse performance than linearizing a softmax-pretrained model. The paper's interpretation is that softmax attention training produces better internal representations (QKV spaces) than TTT training from scratch, because softmax attention is more expressive during training—it can form arbitrary pairwise relationships between tokens, while TTT is constrained to learn a fixed-parameter function. By training with softmax first, the model learns to organize its QKV spaces in a way that captures rich cross-view relationships. The TTT linearization then learns to approximate this already-well-organized mapping, which is an easier problem than discovering the organization from scratch.

This parallels findings in the language model linearization literature (Kasai et al., 2021; Mercat et al., 2024), where converting pre-trained transformers to linear-complexity variants consistently outperforms training linear models from scratch. The pre-trained model acts as a "teacher" that has already solved the hard representation learning problem; the linearized model only needs to learn to approximate the teacher's behavior.

Training dataset and sampling. The fine-tuning uses a dataset comparable to VGGT's original training data (Table 7 lists 24 datasets including ScanNet, Hypersim, Replica, MatrixCity, MegaDepth, TartanAir, DL3DV-10K, and others). Image collections are sampled using a greedy co-visibility algorithm:

Pick a random first image.
Uniformly sample the next image from the set of images with co-visibility greater than 0.3 with any image already in the collection.
Repeat until the desired collection size is reached.

Co-visibility is pre-computed via a depth consistency check using LoFTR (Sun et al., 2021). This ensures that collections contain overlapping views, which is necessary for multi-view reconstruction to be possible.

Training hyperparameters. The paper provides specific values (Appendix A):

Optimizer: AdamW with learning rate $10^{-4}$ , weight decay 0.05, $\beta_1 = 0.9$ , $\beta_2 = 0.95$
Learning rate schedule: linear warmup for 1,000 steps (increasing by factor of 10), followed by cosine decay to $10^{-6}$
Batch size: adaptive, with image collections of 2–24 images while keeping total images per GPU at approximately 48
Image resolution: longer side resized to 518, aspect ratio uniformly sampled from $[0.5, 2.0]$
Data augmentation: independent color jitter per image (brightness and contrast)
Inner optimizer for TTT: Muon (Jordan et al., 2024) with 5 Newton-Schulz iterations, learning rate 0.1, 1 optimizer step during training
TTT MLP dimensions: input and output 1024 (matching VGGT hidden state), hidden layers project to $4\times$ input dimension

Why Muon for the inner optimizer: Muon is a recently proposed optimizer designed specifically for hidden layers in neural networks, using Newton-Schulz iterations to approximate orthogonal matrix updates. The paper likely chose it because the TTT optimization needs to converge in very few steps (1–2 during inference) on variable-length sequences, requiring an optimizer with good single-step convergence properties and robustness to varying data scales. Standard SGD with momentum might require more steps to converge, negating the speed advantage.

Non-Linear Spatial Mixing via ShortConv2D

This is arguably the most subtle and innovative component of the architecture. The paper identifies a fundamental mathematical degeneracy in the TTT objective when applied to transformer QKV projections and introduces a clever mechanism to break it.

The linear dependency problem. Section 3.2 explains:

"Recall that we are learning a mapping from Key to Value space $K \to V$ . However, both $K$ and $V$ are derived from same token $x$ via linear projections $K = W_k x$ and $V = W_v x$ , and the relationship between them is linear ( $V = W_v W_k^{-1} K$ , assuming $W_k$ is invertible). Therefore, simply optimizing Equation 3 can yield a trivial solution."

This is a critical observation. If keys and values are both linear projections of the same underlying token, then there exists an exact linear transformation from keys to values: $V = (W_v W_k^{-1}) K$ . An MLP that learned this linear transformation would achieve zero loss on the TTT objective without learning anything about the specific scene—it would simply be recovering the fixed linear relationship between the projection matrices.

In practice, $W_k$ may not be invertible (it projects from a higher-dimensional token space to a lower-dimensional key space), but the general point stands: there is a data-independent component to the $K \to V$ mapping that the MLP can memorize without paying attention to scene-specific information. The MLP's capacity would be partially wasted on recovering this projection matrix relationship rather than encoding scene geometry.

How the degeneracy manifests. Without mitigation, the TTT objective would encourage the MLP to learn a mapping that works well on average across all tokens (capturing the linear component) rather than one that captures the deviations from this linear relationship that encode scene-specific information. The result would be an MLP that produces bland, scene-independent outputs, failing to propagate the cross-view geometric reasoning that makes global attention valuable.

The ShortConv2D solution. The paper introduces a spatial mixing operation applied to the Value tokens before they are used as targets for the TTT optimization. The procedure has three steps:

Reshape: The 1D sequence of value vectors $[v_1, \dots, v_{N \times H/p \times W/p}]$ , each $v_i \in \mathbb{R}^d$ , is reshaped into a 2D grid of shape $(N, H/p, W/p, d)$ , where $p$ is the patch size. This recovers the spatial structure that was flattened when the image was tokenized—each value vector now sits at its original position in the image grid.
Convolve: A single-layer 2D convolution (ShortConv2D) with $3 \times 3$ filter is applied to this grid. For each spatial position, the convolution aggregates the value vectors from the $3 \times 3$ neighborhood, producing a context-aware target vector $\mathbf{V}'$ that contains information from surrounding patches.
Flatten: $\mathbf{V}'$ is reshaped back to a 1D sequence, and the TTT optimization target becomes mapping keys to these context-enriched values: $K \to V'$ instead of $K \to V$ .

What this accomplishes mechanically: the convolution mixes information across spatial neighborhoods. After ShortConv2D, the value vector at position $(h, w)$ in image $i$ no longer represents just the content of that specific patch—it represents a weighted combination of that patch and its eight immediate neighbors. This means the value contains local geometric context (edges, corners, surface orientation) that a single isolated patch token would not have.

Why this breaks the degeneracy: the key $k_i$ for token $i$ is still computed from the single token $x_i$ (no spatial mixing on keys). But the target value $v_i'$ now contains information from neighboring tokens that is NOT present in $k_i$ alone. The MLP cannot achieve zero loss by learning the linear relationship $W_v W_k^{-1}$ because this would only predict the unmixed value $v_i$ , not the context-enriched $v_i'$ . To predict $v_i'$ , the MLP must infer what the neighboring patches contain based on the current patch's features—which requires understanding the geometric structure of the scene. This is exactly the kind of cross-patch reasoning that global attention performs implicitly through the attention weights.

The paper's interpretation. Section 3.2 explains:

"This incentivizes the fast weights optimization to distill a robust geometric scene representation via a stronger self-supervised objective as the MLP must now predict a neighborhood $\mathbf{V}'$ from a single token's feature $\mathbf{K}$ ."

The MLP is being trained on an inherently harder task (predict a neighborhood aggregate from a single point), which forces it to learn a more sophisticated internal representation. The key insight is that making the objective harder for the MLP (by removing the trivial linear solution) leads to better learned representations, which improve downstream reconstruction quality.

Ablation on filter configurations (Appendix C, Table 9). The paper tests three configurations:

$V\text{-}3$ : $3 \times 3$ convolution on values only (the main configuration)
$V\text{-}5$ : $5 \times 5$ convolution on values only
$KV\text{-}3$ : $3 \times 3$ convolution on both keys and values

Increasing filter size from 3 to 5 does not further improve performance, suggesting that a $3 \times 3$ neighborhood provides sufficient local context and larger neighborhoods add redundancy without additional useful information. Applying ShortConv2D to both keys and values ( $KV\text{-}3$ ) actually decreases performance. The paper explains: "applying the same spatial mixing does not break the dependency between keys and values." If both $K$ and $V$ receive the same spatial mixing, the linear relationship between them might be partially preserved (both are convolved versions of their original projections), reducing the benefit.

Connection to concurrent work. The paper notes that ViT³ (Han et al., 2025) "successfully employs convolutions directly in the inner model for the classification task," suggesting that spatial mixing in TTT architectures is a broadly useful principle beyond 3D reconstruction.

Implementation detail (Appendix A). ShortConv2D is only applied to values projected from image patch tokens. The camera tokens and register tokens (special tokens appended for downstream tasks) are passed through without spatial mixing. This makes sense because camera tokens don't have a meaningful spatial position in the image grid—they represent global properties of the view, not local patches.

Test-Time Scaling via Optimizer Steps

A critical empirical finding is that the number of optimizer steps used in the TTT objective must adapt to the sequence length. The training procedure uses exactly 1 optimizer step (because training sequences are 2–24 images, within the in-distribution range), but inference on larger collections requires more steps.

The sequence length generalization problem. The paper identifies a severe degradation when processing out-of-distribution sequence lengths with the training-time number of steps. Section 3.2 reports:

"For example, the reconstruction error increases about $5\times$ when extending from $N = 100$ to $N = 1k$ images of the same scene."

This means that if you train with 1 optimizer step on 2–24 image collections and then try to process 1000 images with 1 step at inference time, the reconstruction quality collapses. The MLP trained with 1 step cannot compress the information from 1000 images' worth of tokens into its fixed-size weights.

Diagnosis via optimizer step sweep. The paper logs the reconstruction quality as a function of the number of optimizer steps for different sequence lengths. Figure 2a shows the key finding:

For in-distribution sequences (~20 images), 1 optimizer step is optimal. Adding more steps does not improve (and can slightly degrade) performance.
For out-of-distribution sequences (~1000 images), performance peaks at 2 optimizer steps, and 1 step is clearly insufficient.

Why more steps help for longer sequences. The TTT objective is a sum over all tokens (Equation 3). With more tokens, the loss landscape becomes more complex—the MLP must simultaneously satisfy constraints from many more $(k_i, v_i)$ pairs. A single gradient step can only move the parameters a fixed distance in the direction of the average gradient. With many tokens, this average gradient becomes a blurry compromise that doesn't adequately capture the fine-grained structure. Additional steps allow the optimizer to refine the solution, moving from the coarse average to a more precise fit.

The paper demonstrates this empirically in Appendix C (Figure 4) with a sweep across 1–4 optimizer steps for various image collection sizes. Two steps achieve the best performance across a wide range, with 3–4 steps showing comparable or slightly worse results. The diminishing returns suggest that the MLP capacity (fixed at $1024 \to 4096 \to 1024$ ) becomes the bottleneck—more optimization steps cannot extract more information than the MLP can represent.

Practical configuration. The paper states: "we perform 2 steps unless otherwise noted." This is the sweet spot that generalizes well from small to large collections. It represents a form of "test-time scaling" analogous to chain-of-thought reasoning in language models (the paper cites DeepSeek-R1 in this context)—spending additional computation at inference time to handle harder problems.

A crucial design consequence. Because the number of optimizer steps is small (2), and each step requires only a forward and backward pass through a compact MLP (not the full transformer), the additional cost is negligible compared to the speedup gained from linear complexity. At 1000 images, going from 1 to 2 steps adds perhaps 10–20% to the TTT optimization cost, but the alternative (1 step with degraded quality or quadratic attention with 11.6× longer runtime) is far worse.

TTT MLP Architecture and Inner Optimization

The MLP that implements $T_\theta$ is a specific architectural choice with carefully chosen hyperparameters.

SwiGLU MLP. The paper follows Zhang et al. (LaCT, 2025) in using a SwiGLU MLP. SwiGLU (Shazeer, 2020) is a gated variant of the standard MLP where the activation function is applied in a multiplicative gating structure:

$\text{SwiGLU}(x) = (xW_1 + b_1) \otimes \text{Swish}(xW_2 + b_2)$

where $\otimes$ is element-wise multiplication and Swish is the activation $x \cdot \sigma(x)$ . The gating mechanism allows the network to selectively pass or suppress information, which is useful for learning which key features are predictive of which value components.

Dimensions. The MLP takes inputs of dimension 1024 (matching VGGT's hidden state size for token representations) and produces outputs of dimension 1024 (matching the value dimension). The hidden layers project to $4 \times 1024 = 4096$ dimensions. This is a standard expansion ratio for transformer MLPs.

Muon optimizer for inner loop. The paper uses Muon (Jordan et al., 2024) as the optimizer for the TTT inner loop, with 5 Newton-Schulz iterations and learning rate 0.1. Muon is designed to produce approximately orthogonal weight updates by using Newton-Schulz iterations to compute matrix square roots for preconditioning. In the context of test-time training with very few steps (1–2), having an optimizer that makes well-conditioned updates is crucial—a poorly conditioned update (e.g., one that overshoots in some directions and undershoots in others) cannot be corrected in the next step because there may not be a next step.

Why learning rate 0.1: this is relatively high for an inner optimizer (outer training uses $10^{-4}$ ), reflecting that the inner optimization needs to make significant progress in very few steps. The high learning rate is possible because Muon's orthogonal updates prevent the kind of divergence that would occur with standard SGD at such high learning rates.

One MLP per global attention layer. Each global attention layer gets its own independent MLP with its own parameters $\theta$ . During the forward pass, each layer's MLP is optimized separately on that layer's $(k_i, v_i)$ pairs before being queried with $q_i$ to produce the layer's output. This means that information flows through the network in a staged fashion: layer 1's MLP compresses the scene geometry at a low level, layer 2's MLP compresses higher-level relationships, and so on. The MLPs do not share information across layers—each one sees only the keys and values at its specific depth.

Why per-layer MLPs rather than a shared MLP: different transformer layers represent different levels of abstraction. The key-value relationships at layer 1 (low-level features) are qualitatively different from those at layer 12 (high-level semantic features). A shared MLP would need to handle all these different mappings with the same parameters, which would require much larger capacity. Per-layer MLPs can specialize.

Training of the TTT modules. During the fine-tuning phase, the TTT MLP parameters are trained alongside the QKV projections. However, the MLP parameters are not directly optimized by the outer training loop on the reconstruction loss—instead, they are optimized by the inner TTT loop on each training batch, and the outer loop only updates the QKV projections and output projections. The MLP "learns to learn" the $K \to V$ mapping efficiently from few steps and few examples.

Large-Scale Inference Strategies

The linear complexity of TTT does more than just reduce computation—it enables inference strategies that are impossible with quadratic attention.

Minibatch gradient decomposition (Equation 5). The core property that enables flexible inference is that the gradient of the TTT objective with respect to the MLP parameters is a sum of per-token gradients:

$\frac{dL_{\text{total}}}{d\theta} = \sum_i \frac{d}{d\theta} L(k_i, v_i) = \sum_s \left(\sum_{i \in s} \frac{d}{d\theta} L(k_i, v_i)\right)$

where $s$ indexes minibatches (subsets of tokens).

What this equation computes: the total gradient is exactly the sum of gradients computed independently on each minibatch. This means you can process the sequence in chunks, accumulate gradients, and apply the update—the result is mathematically identical to processing the entire sequence at once.

Why this matters for single-GPU inference (CPU offloading): the tokens for a 2000-image collection with patch size 14 produce approximately $2000 \times 1369 \approx 2.74$ million tokens, each of dimension 1024. Storing all QKV vectors in GPU memory simultaneously would require tens of gigabytes, potentially exceeding GPU memory. With the decomposition property, VGG-T³ can:

Load a minibatch of tokens onto the GPU.
Compute the per-minibatch gradient of the TTT loss.
Accumulate the gradient in the MLP parameters (which are tiny—a few million parameters at most).
Offload the minibatch to CPU memory.
Repeat for all minibatches.
Apply the accumulated gradient to update the MLP.

Only a single minibatch needs to be in GPU memory at any time. The paper explicitly contrasts this with softmax attention methods: "methods relying on softmax attention... require $q_i, k_i, v_i$ of all images to be in GPU memory which, even for large GPUs, leads quickly to out-of-memory errors when processing larger image collections." FlashAttention reduces the memory footprint of the attention computation itself, but the QKV tensors still need to be resident for the forward/backward pass through the rest of the network.

Why this matters for multi-GPU distributed inference: each GPU can process a minibatch independently and then synchronize gradients via all-to-all communication. The paper notes: "we shard images such that each GPU only processes a subset $s$ ." After computing local gradients, the GPUs perform an all-reduce to sum the gradients and update their local copies of the MLP parameters. The communication cost is proportional to the MLP size (a few MB) rather than the sequence length, making it efficient even at scale.

This is in contrast to VGGT's distributed inference, which requires "carefully engineered context-parallel implementations for softmax attention (e.g., ring attention)"—complex system-level optimizations to distribute the attention computation itself across GPUs. VGG-T³ works directly with standard distributed data parallel (DDP) because the only cross-GPU communication is the gradient synchronization for the MLP weights.

Quantitative scaling (Table 4 and Figure 3). For 1000 images:

VGGT: ~11 minutes (with context-parallel inference and FlashAttention)
FastVGGT: ~4.3 minutes
VGG-T³: 58 seconds (single GPU) → 11.6× speedup over VGGT

For 2000 images:

VGGT: 27 minutes
VGG-T³: 48.5 seconds → 33× speedup

With 2 GPUs, VGG-T³ achieves near-linear speedup (about 2× faster than single GPU), while VGGT's distributed inference sees sub-linear scaling due to communication overhead.

Query-mode inference for visual localization (Section 3.3). After the MLP weights $\theta$ are optimized on the mapping images (the images used to reconstruct the scene), they can be frozen. A novel query image can then be processed by:

Encoding it through the frozen tokenizer and per-image attention layers (same as any other image).
In the global attention layers, applying ONLY the query step: $o_i = T_\theta(q_i)$ , without updating $\theta$ .
Decoding the output tokens through the frozen prediction heads to obtain depth and camera pose for the query image.

The MLP, having been trained on the mapping images' keys and values, has internalized a compressed scene representation. When queried with a new image's query vector $q_i$ , it retrieves the relevant scene information—effectively answering "given what I know about this scene, what should this new image's geometry and pose be?"

This is conceptually elegant because it uses the SAME model for both mapping (optimizing $\theta$ ) and localization (querying frozen $\theta$ ). There is no separate localization pipeline, no feature matching, no PnP solver. The paper acknowledges this is a proof-of-concept—accuracy doesn't match specialized localization methods—but demonstrates technical feasibility.

Camera token handling during localization (Appendix A). VGGT's camera head requires the camera tokens of ALL mapping images as input, not just the query image. For localization, the frozen MLPs process the mapping images' camera tokens normally, and the query image's camera token participates in softmax attention within the camera head before being decoded to camera parameters. This is a small exception to the "all attention is replaced by TTT" rule—the camera head still uses softmax attention among camera tokens because the MLP was found to struggle with the heterogeneous token types (Section 4.1).

Summary of Design Choices and Their Justifications

Post-training linearization over training from scratch: the pre-trained model's QKV spaces are better organized; linearization approximates an already-good representation rather than discovering it from scratch, avoiding local optima (Table 6).
TTT over other linear attention mechanisms (T2R, LoLCats, Mamba): empirically superior in this domain (Table 6, variants ii–iv). TTT's explicit per-scene optimization provides more expressive power than fixed recurrent updates.
SwiGLU MLP over standard MLP: gating mechanism allows selective information flow, following LaCT's successful recipe for test-time training.
Dot product loss over MSE: focuses on directional alignment rather than exact magnitude matching; magnitudes are normalized by subsequent layers anyway.
$L_2$ normalization over LayerNorm in QK projections: removes learned parameter distortions that interfere with TTT optimization; enables fast convergence from pre-trained weights.
ShortConv2D on values only (not keys): breaks the linear $K \to V$ dependency without destroying the query-key correspondence needed for the query step.
$3 \times 3$ filter over larger filters: sufficient local context; larger neighborhoods add no benefit (Table 9).
2 optimizer steps at inference: sweet spot between convergence quality and computational cost; 1 step insufficient for long sequences, 3+ steps show diminishing returns (Appendix C, Figure 4).
Muon optimizer for inner loop: well-conditioned updates crucial when only 1–2 steps are available; Newton-Schulz iterations provide approximate orthogonal preconditioning.
Per-layer MLPs rather than shared: different transformer depths represent different abstraction levels; per-layer specialization is more parameter-efficient.
Freezing encoder, per-image attention, and prediction heads: preserves general vision knowledge learned during VGGT pretraining; reduces training cost to ~12% of full training.
Co-visibility-based training collection sampling (threshold 0.3): ensures collections contain geometrically connected views; prevents training on impossible reconstruction tasks.

4. Key Insights and Innovations

Innovation 1: Scene Representation as Test-Time Compression — Reframing Global Attention as a Fitting Problem Rather Than a Querying Problem

The paper's deepest conceptual move is not the specific TTT mechanism (which it borrows from Sun et al., 2025) but the reframing of what global attention does and how that function can be achieved differently. Prior work treated the Key-Value cache as a database to be queried: every output token asks "which input tokens are relevant to me?" via softmax attention, and the KV representation exists to be searched. FastVGGT and SparseVGGT accepted this framing and tried to make the searching cheaper (by compressing what gets searched). Chunked methods accepted it and restricted the search radius (local windows). Autoregressive methods accepted it and restricted the search direction (causal).

VGG-T³ asks a fundamentally different question: instead of "how do I search the KV space efficiently?", it asks "can I fit the entire KV mapping into a compact function and then evaluate that function instead of searching?" This is the difference between building a faster search engine and pre-computing the answers. The conceptual leap is from treating the scene representation as something to be queried interactively to something to be compressed proactively.

This reframing is significant because it changes the computational role of test-time compute. In softmax attention, the computation happens at query time (the $O(n^2)$ pairwise dot products and softmax). In VGG-T³, the computation happens at compression time (the optimization of the MLP), and querying becomes trivially cheap ( $O(1)$ per token). The paper explicitly analogizes this to DeepSDF (Park et al., 2019), where a pre-trained decoder is conditioned on a test-time-optimized latent code. But the analogy is deeper than the paper explicitly develops: in both cases, the hard work of understanding the data is done once by a pre-trained model (the decoder/encoder), and the test-time optimization only needs to encode instance-specific details into a compact representation. This is a design pattern that generalizes beyond 3D reconstruction to any domain where pre-trained models need to process variable-length inputs at scale.

The significance of this innovation is that it decouples the model's capacity to reason globally (which is preserved, because the MLP sees all tokens during optimization) from its computational complexity (which becomes linear, because querying the MLP is $O(1)$ ). Prior approaches forced a tradeoff: either accept quadratic complexity for global reasoning, or sacrifice global reasoning for linear complexity. VGG-T³ rejects this tradeoff by changing the when of global computation rather than the whether. The evidence that this reframing is productive—not merely philosophical—is in Figure 3: VGG-T³ achieves reconstruction quality approaching VGGT (the quadratic upper bound) while matching TTT3R's linear scaling, something no prior method accomplished.

Innovation 2: ShortConv2D as a Degeneracy-Breaking Mechanism — Converting a Mathematical Vulnerability into an Architectural Strength

The ShortConv2D contribution initially appears to be a minor architectural tweak—"add a 3×3 convolution to the values." But the intellectual contribution is the paper's diagnosis of why the naive TTT objective degenerates and its solution, which reveals a subtle relationship between transformer projections and optimization objectives that has implications beyond this specific architecture.

The degeneracy is not obvious. A reader might assume that learning $K \to V$ via MLP is simply a harder optimization problem than softmax attention, and any accuracy gap is due to insufficient MLP capacity. The paper's key diagnostic insight (Section 3.2) is that there exists a data-independent linear relationship between keys and values—namely $V = W_v W_k^{-1} K$ —that the MLP can learn without encoding any scene-specific information. This means the TTT objective, as originally formulated, is partially solved by the projection matrices themselves, leaving the MLP's capacity underutilized for actual scene representation.

This is fundamentally different from the usual "MLPs are less expressive than attention" concern. It's not about capacity; it's about incentive structure. The objective function has a trivial component that doesn't require scene understanding, and optimization will naturally exploit it. ShortConv2D solves this not by increasing capacity or changing the optimizer, but by changing the task: instead of predicting $v_i$ from $k_i$ , predict a spatially-mixed $\tilde{v}_i$ from $k_i$ . The spatial mixing injects information into the target that is not present in the input, forcing the MLP to infer contextual relationships rather than memorizing projection matrices.

This is a case study in a broader principle: when converting a retrieval operation (attention) into a learning operation (TTT), you must ensure the learning objective doesn't have shortcuts that bypass the information you actually want to encode. The paper demonstrates this principle empirically through the ablation in Table 9: applying the same spatial mixing to both keys and values ( $KV\text{-}3$ ) reduces performance because it partially restores the linear dependency (both sides are convolved), while applying it only to values ( $V\text{-}3$ ) maximizes the gap between what the key contains and what the value demands.

The finding that $5 \times 5$ filters don't help over $3 \times 3$ (Table 9) is also informative: it suggests the necessary context is strictly local—knowing a patch's immediate neighbors is sufficient to break the degeneracy, and larger receptive fields add redundant information. This constrains the design space for future work applying similar techniques to other domains.

Innovation 3: Optimizer Steps as a Sequence-Length Generalization Mechanism — Discovering a "Test-Time Scaling Law" for Compressed Representations

The paper's finding about optimizer steps (Section 3.2, Figure 2a) is more than a hyperparameter tuning result—it reveals a fundamental scaling behavior of test-time-trained representations that the authors partially characterize as a "test-time scaling law." The critical observation is that the number of optimizer steps needed to achieve good reconstruction is not fixed but depends on the information content of the sequence being compressed.

When compressing a short sequence (tens of images) into a fixed-size MLP, one gradient step suffices because the information-to-parameter ratio is favorable—the MLP has enough capacity relative to the data that even a coarse fit captures the essential structure. When compressing a long sequence (thousands of images), one step is insufficient because the average gradient over many more tokens becomes a blurry compromise. The paper's demonstration that simply increasing from 1 to 2 steps recovers near-constant scaling behavior (Figure 2b, Figure 4 in Appendix C) is evidence that the bottleneck is optimization convergence, not MLP capacity—at least up to the tested scales.

This has conceptual implications beyond the specific numbers. It suggests that test-time-trained representations exhibit a form of computational scalability analogous to how larger models benefit from more training steps: you can trade additional inference computation (more optimizer steps) for better compression quality on harder problems. The paper explicitly connects this to chain-of-thought reasoning in LLMs (citing DeepSeek-R1), framing it as "test-time scaling via additional computation." But the mechanism is different: in LLMs, more tokens allow more reasoning steps; in VGG-T³, more optimizer steps allow better compression of a fixed-size representation. Both are forms of allocating computation proportional to problem difficulty.

The finding is empirically anchored in Figure 2a, which shows the optimal step count shifting from 1 (at 20 images) to 2 (at 1k images), and Figure 4 in Appendix C, which shows 2 steps being robust across a wide range while 3–4 steps show diminishing returns. This pattern—diminishing returns after a small number of additional steps—is consistent with the MLP capacity eventually becoming the bottleneck, creating a two-phase scaling behavior: first optimizer-limited (more steps help), then capacity-limited (more steps don't help). Characterizing this phase transition would be a natural extension.

Innovation 4: Unified Mapping and Localization Through Frozen Implicit Representations — Eliminating the Pipeline Boundary

The demonstration of feed-forward visual localization (Section 4.3, Table 5) is a proof-of-concept, but its conceptual significance is that it dissolves the traditional boundary between reconstruction and localization within a single model architecture. In classical pipelines, reconstruction (SfM) and localization (feature matching + PnP) are separate stages with different algorithms, different failure modes, and different computational profiles. Even in learning-based approaches, they typically use different models: a feed-forward reconstruction network for mapping, and a scene coordinate regression network or retrieval-based pipeline for localization.

VGG-T³ achieves both with the same learned parameters. The MLP weights $\theta$ , optimized on the mapping images, constitute the "map." Querying them with a novel image without further optimization constitutes "localization." The model doesn't need to know which task it's performing—the same forward pass through the frozen MLPs produces geometry and pose regardless. This is conceptually elegant because it means the scene representation learned during reconstruction is directly reusable for downstream tasks without retraining, fine-tuning, or even architectural modification.

The paper is appropriately circumspect about the current accuracy (Section 4.3 notes that specialized methods like Reloc3R achieve $e_r = 1.02^\circ, e_t = 0.04\text{m}$ on 7scenes, far better than VGG-T³'s results). But the architectural capability is what matters here—it demonstrates that compressed scene representations can generalize to unseen viewpoints, which is not obvious a priori. One might have worried that the MLP overfits to the specific views seen during optimization and cannot interpolate or extrapolate to novel poses. The localization results (and the in-the-wild KITTI example in Appendix D, Figure 7, where a tourist photo taken 7 years later is successfully localized) provide preliminary evidence that the representation captures genuine 3D structure rather than view-specific memorization.

The paper also notes a limitation that points to future work: the camera head still requires softmax attention among camera tokens because the MLP struggles with heterogeneous token types (image patches vs. camera tokens). Removing this last vestige of quadratic attention is an open challenge, but the proof-of-concept is established—a single compressed representation can support both scene reconstruction and query-time localization. </output>

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on several standard benchmarks for 3D reconstruction tasks. For pointmap estimation: NRGBD, 7scenes, DTU, and ETH3D, each with their standard test splits following prior work (Wang and Agapito, 2025; Wang et al., 2025). For video depth estimation: Bonn, KITTI, and Sintel evaluation sets, again following prior evaluation protocols (Wang et al., 2025). For camera pose estimation: TUM-RGBD, ScanNet, and Sintel. For large-scale reconstruction experiments, the 7scenes dataset is used by aggregating all video frames and uniformly subsampling to form validation sets of varying sizes (100, 500, 1000 images). For visual localization: 7scenes and Wayspots, with mapping images subsampled at strides of 200 and 20 respectively.
Base model(s). All experiments start from the public VGGT checkpoint (Wang et al., 2025), a multi-view feed-forward transformer that predicts per-image depth, camera poses, and intrinsics from unordered image collections. VGG-T³ converts this model by replacing all global attention layers with TTT layers while freezing the image tokenizer, per-image self-attention layers, and all prediction heads. Only the QKV projection matrices, output projections, and newly introduced TTT MLPs in the global attention layers are fine-tuned. The fine-tuning uses a dataset comparable to VGGT's original training data (24 datasets listed in Table 7), running for 100k steps on 8 NVIDIA A100-80GB GPUs, which the paper states is approximately 12% of the cost of training VGGT from scratch.
Metrics. For pointmap estimation, the paper uses Chamfer Distance (CD) and Normal Consistency following Wang and Agapito (2025). Chamfer Distance measures the average nearest-neighbor distance between predicted and ground-truth point clouds (lower is better). For video depth estimation, the paper reports Absolute Relative Error (Abs. Rel.) and the percentage of predictions with $\delta < 1.25$ , following standard depth evaluation protocols where predictions are aligned using a single scale per sequence. For camera pose estimation, the paper reports Absolute Trajectory Error (ATE) for the aligned trajectory. For visual localization, rotation error $e_r$ (degrees) and translation error $e_t$ (meters) are reported, along with the percentage of query images localized within thresholds $e_r < T_r$ and $e_t < T_t$ , following standard visual localization metrics.
Baselines. The paper compares against both offline and online methods. Offline baselines include: VGGT (Wang et al., 2025), the original quadratic-complexity model serving as an accuracy upper bound; FastVGGT (Shen et al., 2025), which uses token merging to reduce the constant factor of attention; and SparseVGGT (Wang et al., 2025), which employs block-sparse attention. All offline baselines exhibit $O(n^2)$ complexity. The online baseline is TTT3R (Chen et al., 2025), a concurrent method that improves upon CUT3R and uses test-time training for autoregressive, ordered-sequence processing with linear complexity. For the visual localization experiments, the paper additionally references Reloc3R (Dong et al., 2025) as a state-of-the-art specialized method for context, though not as a direct baseline since it uses explicit mapping with accurate poses.
Generation budget / compute accounting. Compute is measured in wall-clock inference time (seconds or minutes), enabling direct comparison of practical throughput across methods with different algorithmic complexities. For large-scale experiments, runtime is reported on the same hardware (NVIDIA A100-80GB GPUs) with the same input image collections. The paper carefully notes when baselines require additional engineering to even process large collections: VGGT requires context-parallel inference using Ulysses (Jacobs et al., 2023) implemented in TransformerEngine with FlashAttention2 to handle 2000 images on 2 GPUs, and FastVGGT code "fails on NRGBD-S due to one instance having only two views" (Table 1 note). For VGG-T³, inference runs with 2 optimizer steps in the TTT objective unless otherwise noted. Distributed inference experiments (Table 4) report latency scaling with 1, 2, and 4 GPUs.
Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported. The paper evaluates on standard test splits for each benchmark. For the large-scale reconstruction experiments (Section 4.2), validation sets are formed by uniform subsampling from aggregated video frames for each scene in 7scenes. For the ablation studies (Section 4.4), experiments are performed "in a smaller scale setting with image resolution of $224 \times 224$ on ScanNet++ training with 2–24 views." No confidence intervals or error bars are reported for any experiments. The paper acknowledges that pointmap evaluation for large image sets is computationally expensive due to iterative closest point (ICP) alignment, so it uses 10 equally spaced keyframes to compute metrics while treating other frames as supporting views—this is an important methodological detail that could affect metric comparability across methods.

Main Quantitative Results

Pointmap Estimation (Table 1)

The headline result is that VGG-T³ substantially outperforms the only other $O(n)$ baseline, TTT3R, while remaining competitive with $O(n^2)$ methods. On the NRGBD dataset (dense split), VGG-T³ achieves Chamfer Distance of 0.052 versus TTT3R's 0.121 (a 2.3× reduction in error). The gap is even larger on the sparse split: 0.038 versus 0.106 (2.8× reduction). On DTU, VGG-T³ achieves 0.023 versus TTT3R's 0.046 (2.0× reduction) and actually surpasses all $O(n^2)$ baselines—VGGT, FastVGGT, and SparseVGGT all achieve 0.027–0.028. On ETH3D, VGG-T³ achieves 0.044 versus TTT3R's 0.103 (2.3× reduction), comparable to SparseVGGT (0.039) and FastVGGT (0.040), though behind VGGT (0.020). On 7scenes-D, VGG-T³'s Chamfer Distance of 0.024 is slightly worse than TTT3R's 0.019—the one exception to the pattern of VGG-T³ dominance. Normal Consistency results follow similar trends, with VGG-T³ consistently outperforming TTT3R by substantial margins (e.g., 0.321 vs. 0.172 on NRGBD-D, where higher is better).

An important caveat: FastVGGT code fails on NRGBD-S due to one instance having only two views (as noted in Table 1), so those entries are incomplete. This highlights a practical robustness issue with token-merging approaches—they can fail on edge cases that VGG-T³ handles naturally since it makes no assumptions about minimum sequence length.

Video Depth Estimation (Table 2)

On video depth estimation, VGG-T³ again outperforms TTT3R on two of three datasets by substantial margins. On Bonn, VGG-T³ achieves Abs. Rel. of 0.077 and $\delta < 1.25$ of 0.898 versus TTT3R's 0.125 and 0.782. On Sintel, VGG-T³ achieves 0.289 and 0.671 versus TTT3R's 0.513 and 0.482—an even larger gap. On KITTI, the two methods are comparable: VGG-T³ achieves 0.103 and 0.918 versus TTT3R's 0.107 and 0.903. Compared to $O(n^2)$ baselines, VGG-T³ performs on par with VGGT on KITTI (0.103 vs. 0.105 Abs. Rel.) and close on Bonn (0.077 vs. 0.053), though a gap remains on Sintel (0.289 vs. 0.204 for VGGT).

The KITTI result is notable because it shows that on some datasets, the linear-time approximation essentially matches quadratic attention's accuracy, suggesting that the MLP compression is not inherently lossy for all scene types—it depends on scene complexity and baseline width.

Camera Pose Estimation (Table 3)

Camera pose estimation is where VGG-T³ shows its primary weakness. On TUM-RGBD, VGG-T³ achieves ATE of 0.058m versus TTT3R's 0.032m (ordered) and 0.194m (unordered). On ScanNet, VGG-T³ achieves 0.124m versus TTT3R's 0.082m (ordered). On Sintel, VGG-T³ achieves 0.203m versus TTT3R's 0.090m (ordered). The paper explicitly notes this limitation (Section 4.1):

"We suspect this is related to VGGT's special treatment of camera pose, where a dedicated camera token is appended to the image tokens immediately before the attention layer, effectively creating two input 'modalities'. This heterogeneous structure may be challenging for the MLP within the TTT layer to memorize."

However, VGG-T³ has a crucial advantage over TTT3R on this task: it naturally supports both ordered and unordered input sequences. TTT3R, being autoregressive, degrades significantly on unordered inputs (ATE jumps from 0.032m to 0.194m on TUM-RGBD for the ordered vs. unordered setting). VGG-T³'s performance is essentially unchanged regardless of input ordering because its global optimization sees all images simultaneously. The paper marks best performance on ordered inputs in bold and best unordered in blue in Table 3, with VGG-T³ taking the blue (unordered) marks.

Large-Scale Reconstruction (Figure 3, Table 4)

This is the paper's most impactful set of results, directly demonstrating the scalability claims. Figure 3 shows runtime versus Chamfer distance for image collections of sizes 100, 500, and 1000 on the 7scenes dataset, creating a Pareto frontier visualization.

For 1000 images:

VGGT: Chamfer Distance ~0.022, runtime ~11 minutes
FastVGGT: CD ~0.025, runtime ~4.3 minutes
SparseVGGT: CD ~0.029, runtime ~2.5 minutes
VGG-T³: CD ~0.027, runtime 58 seconds (11.6× faster than VGGT)
TTT3R: CD ~0.037, runtime comparable to VGG-T³

The key observation from Figure 3 is the narrowing gap between VGG-T³ and the quadratic baselines as image count increases. At 100 images, VGG-T³'s CD is noticeably higher than VGGT's, but at 1000 images, the gap is small (~0.027 vs. ~0.022). This suggests that VGG-T³'s compressed representation actually scales better in quality relative to softmax attention as the scene becomes more complex—counterintuitively, the MLP compression becomes more competitive with more data, not less. The paper does not explicitly analyze this phenomenon, but it may be because softmax attention's fixed $1/\sqrt{d}$ scaling (Equation 2) becomes suboptimal for very long sequences (which the paper partially addresses for VGGT via entropy scaling, Appendix B), while the MLP's optimization-based approach adapts naturally.

The distributed inference results (Table 4) demonstrate near-linear scaling for VGG-T³: 1000 images take 58.0s on 1 GPU, 31.5s on 2 GPUs (1.84× speedup), and 17.8s on 4 GPUs (3.26× speedup). For 2000 images: 97.8s on 1 GPU, 50.1s on 2 GPUs (1.95× speedup), and 27.3s on 4 GPUs (3.58× speedup). The sub-linear scaling is expected due to communication overhead for gradient synchronization, but the efficiency remains high. In contrast, VGGT's distributed inference requires context-parallel attention implementations and shows worse scaling behavior.

The qualitative comparison (Figure 5, Appendix D) reinforces the quantitative results. TTT3R and VGG-T³ both process 1k image collections within 1 minute, but TTT3R's reconstructions show visible degradation—inconsistent geometry, missing surfaces—while VGG-T³ produces 3D-consistent reconstructions that are visually close to VGGT's quality (which takes 11+ minutes). The ScanNet sequences in Figure 5 (scenes 0726, 0734, 0735, 0738, 0757) show this pattern consistently: VGGT produces the sharpest details, VGG-T³ is slightly smoother but coherent, and TTT3R shows substantial artifacts.

Feed-Forward Visual Localization (Table 5)

VGG-T³ demonstrates that its compressed MLP representation can localize novel query images without additional optimization, outperforming TTT3R (which also supports querying its autoregressive state in $O(1)$ ). On 7scenes, VGG-T³ achieves median rotation error $e_r = 2.18^\circ$ and translation error $e_t = 0.074$ m, versus TTT3R's $e_r = 3.65^\circ$ and $e_t = 0.101$ m. The percentage of queries localized within $(5^\circ, 0.05\text{m})$ is 61.2% for VGG-T³ versus 48.3% for TTT3R. On Wayspots, the improvement is larger: VGG-T³ achieves $e_r = 5.51^\circ$ , $e_t = 0.523$ m, with 42.4% of queries within $(10^\circ, 1.0\text{m})$ , versus TTT3R's $e_r = 14.51^\circ$ , $e_t = 2.503$ m with only 21.2% within thresholds.

The paper contextualizes these numbers by noting that specialized localization methods like Reloc3R achieve far better accuracy ( $e_r = 1.02^\circ$ , $e_t = 0.04$ m on 7scenes), emphasizing that the contribution is demonstrating feasibility rather than state-of-the-art localization. The in-the-wild example in Appendix D (Figure 7) is particularly compelling: a tourist photo taken 7 years after a KITTI sequence is successfully localized within the reconstruction, despite changes in the street (the tourist photo captures upper building parts not visible from the car-mounted camera). This demonstrates robustness to viewpoint variation and temporal scene changes.

Ablation Studies and Robustness Checks

The ablation studies (Table 6) are conducted at a smaller scale (image resolution $224 \times 224$ , ScanNet++ training, 2–24 views) using pointmap estimation metrics (Chamfer Distance at $224 \times 224$ resolution and $518 \times 518$ resolution, plus Normal Consistency at $518 \times 518$ ). This reduced-scale protocol allows faster experimentation while preserving the relative ordering of methods.

Training from scratch with TTT (variant i) vs. post-training linearization: Training a TTT model from scratch (no softmax pretraining) achieves dramatically worse performance than linearizing a softmax-pretrained model. At $518 \times 518$ resolution, the scratch-trained TTT model achieves CD of 0.098 versus 0.047 for the linearized model (variant v)—more than 2× worse. The paper states this variant "gets stuck in a local optimum," supporting the claim that softmax pretraining produces better-organized QKV spaces that linearization can then approximate.

Linearization method comparison (variants ii–iv): VGG-T³'s TTT-based linearization is compared against T2R (Kasai et al., 2021) and LoLCats (Zhang et al., 2025), two methods from the language model linearization literature adapted to this setting. T2R achieves CD of 0.052 at $518 \times 518$ , LoLCats achieves 0.064, and VGG-T³ achieves 0.047—substantially outperforming both. The paper does not provide detailed analysis of why TTT outperforms these alternatives, but the implication is that test-time optimization provides more expressive power than the fixed recurrent updates used in T2R and LoLCats.

ShortConv2D contribution (variant v vs. variant without): This is the most critical ablation for validating the paper's architectural innovation. The model without ShortConv2D (implied as the baseline, though not explicitly shown as a separate row in Table 6—the comparison is between variant v with ShortConv2D achieving CD 0.047 and the discussion in Section 4.4 stating it "further closes the gap towards softmax attention") is not directly quantified in the table. However, the discussion makes clear that ShortConv2D is essential for closing the accuracy gap. The additional ShortConv2D configurations tested in Appendix C (Table 9) show that $3 \times 3$ convolution on values only achieves CD of 0.047 at $518 \times 518$ , while $5 \times 5$ convolution achieves 0.048 (no improvement), and applying convolution to both keys and values ( $KV\text{-}3$ ) degrades performance to 0.053—consistent with the paper's explanation that applying the same mixing to both sides partially restores the linear dependency.

Softmax attention upper bound: The softmax attention model (trained and evaluated with standard global attention) achieves CD of 0.039 at $518 \times 518$ , representing the accuracy ceiling that VGG-T³ approaches but does not match. The gap of 0.008 CD (roughly 20% relative) quantifies the cost of linearization. The paper's narrative emphasizes that this gap narrows with increasing image count (Figure 3 shows convergence at 1k images), though this specific claim about narrowing is based on the 7scenes large-scale results rather than the ablation table.

Optimizer steps for sequence-length generalization (Appendix C, Figure 4): This experiment validates the claim that increasing optimizer steps compensates for out-of-distribution sequence lengths. On NRGBD, with varying image collection sizes and 1–4 optimizer steps, the key finding is that 2 steps achieves the best performance across a wide range of collection sizes, while 1 step degrades noticeably beyond the training distribution (~24 images), and 3–4 steps show comparable or slightly worse performance. This supports the paper's default of 2 inference steps and demonstrates that the sequence-length generalization problem can be addressed by a simple, principled increase in test-time computation.

Attention entropy scaling for VGGT baseline (Appendix B, Table 8): While not strictly a VGG-T³ ablation, this enhancement to the VGGT baseline is methodologically important. Without entropy scaling, VGGT's performance degrades on large image collections because the softmax temperature ( $1/\sqrt{d}$ ) is optimized for training sequence lengths (~24 images, ~33k tokens) and becomes poorly calibrated for longer sequences. By scaling $\lambda' = \lambda \max(1.0, \log_{N_T} N)$ , where $N_T = 32,856$ is the maximum training tokens, the entropy of the attention matrix is kept approximately constant. Table 8 shows that this improves VGGT's CD on 500-image collections from 0.026 to 0.023 at $518 \times 518$ , making it a stronger baseline. This adjustment means VGG-T³'s competitive performance is measured against an optimized VGGT, not a degraded one.

Pointmap prediction source (Appendix A, inference details): The paper notes that VGGT has multiple decoders predicting redundant geometric quantities. Pointmaps can be obtained either from the global pointmap prediction head directly or by unprojecting depth maps using camera predictions. VGGT finds unprojecting to be more precise, but VGG-T³ uses the global prediction head because its camera pose predictions are imprecise (Section 4.1). This is an important design choice that affects metric comparability—VGG-T³'s pointmap quality would likely be worse if it relied on unprojected depth due to the camera pose errors documented in Table 3.

Critical Assessment

Claim 1: "Linear-time scaling while preserving competitive accuracy"

The experiments strongly support the linear-time scaling claim. Figure 3 and Table 4 provide direct evidence: VGG-T³ processes 1000 images in 58 seconds versus VGGT's 11 minutes (an 11.6× speedup), and 2000 images in 48.5 seconds versus 27 minutes (a 33× speedup). The linear scaling behavior is visible in the runtime curves of Figure 3, which show VGG-T³ and TTT3R growing linearly while VGGT, FastVGGT, and SparseVGGT curve upward quadratically.

The "competitive accuracy" claim requires more nuance. On pointmap estimation (Table 1), VGG-T³ achieves CD of 0.052 on NRGBD-D versus VGGT's 0.039—a 33% relative increase in error. On ETH3D, the gap is larger: 0.044 versus 0.020 (2.2× worse). On DTU, VGG-T³ actually surpasses VGGT (0.023 vs. 0.028), but this appears to be an outlier. On video depth (Table 2), the gap varies: minimal on KITTI (0.103 vs. 0.105 Abs. Rel.), moderate on Bonn (0.077 vs. 0.053), and substantial on Sintel (0.289 vs. 0.204). On camera pose (Table 3), VGG-T³ is consistently worse than VGGT.

The claim holds most strongly in the large-scale regime where the quadratic baselines become impractical (Figure 3, 1000 images). Here, the accuracy gap narrows considerably (CD ~0.027 vs. ~0.022), making VGG-T³ the clear Pareto-optimal choice—it achieves near-VGGT quality at a fraction of the time. However, the paper does not test beyond 1000 images for the quadratic baselines on this metric (2000-image results are runtime-only in the teaser), so we cannot confirm whether the narrowing trend continues.

Claim 2: "Outperforms TTT3R (the only other linear-time baseline) by 2–2.5× on pointmap error"

This claim is well-supported by Table 1. On NRGBD-D: CD 0.052 vs. 0.121 (2.3×). On NRGBD-S: 0.038 vs. 0.106 (2.8×). On DTU: 0.023 vs. 0.046 (2.0×). On ETH3D: 0.044 vs. 0.103 (2.3×). The one exception is 7scenes-D where TTT3R achieves 0.019 versus VGG-T³'s 0.024, but VGG-T³ wins on 7scenes-S (0.020 vs. 0.033). Video depth (Table 2) shows similar trends, with VGG-T³ outperforming on Bonn and Sintel by large margins and roughly matching on KITTI. Visual localization (Table 5) extends this advantage to a new task.

However, there is an important methodological asymmetry: TTT3R is designed for ordered sequences and processes images autoregressively, while VGG-T³ processes all images simultaneously. For the pointmap and depth experiments, the paper provides TTT3R with images in sequential order with keyframes last "such that the model has seen all images of the scene before making predictions" (Appendix A). This is a generous setup for TTT3R—it sees all context before predicting. The fact that VGG-T³ still substantially outperforms it suggests the advantage comes from bi-directional global reasoning rather than ordering. But on camera pose (Table 3), TTT3R achieves better results on ordered inputs (ATE 0.032m vs. VGG-T³'s 0.058m on TUM-RGBD), suggesting that for pose estimation specifically, TTT3R's sequential processing has advantages that VGG-T³'s global compression does not capture.

Claim 3: "Enables feed-forward visual localization"

The experiments in Table 5 demonstrate this capability, with VGG-T³ localizing 61.2% of queries on 7scenes within $(5^\circ, 0.05\text{m})$ and 42.4% on Wayspots within $(10^\circ, 1.0\text{m})$ . The in-the-wild example (Figure 7, Appendix D) visually demonstrates generalization across a 7-year temporal gap. However, the paper does not compare against any classical or learning-based localization baselines beyond TTT3R. The stated comparison to Reloc3R ( $e_r = 1.02^\circ$ , $e_t = 0.04\text{m}$ on 7scenes) shows that VGG-T³ ( $e_r = 2.18^\circ$ , $e_t = 0.074\text{m}$ ) is substantially less accurate than specialized methods. The claim should be understood as "demonstrates feasibility of unified mapping and localization" rather than "achieves competitive localization accuracy." The paper is transparent about this framing (Section 4.3: "our aim is to show that feed-forward visual localization without explicit mapping is indeed feasible").

Genuine weaknesses in the experimental design:

Single model family and initialization. All experiments use VGGT as the base architecture. The paper does not demonstrate that the linearization approach generalizes to other multi-view reconstruction architectures (Fast3R, π³, DUSt3R variants). The ablation showing training-from-scratch failure (Table 6, variant i) is conducted on a VGGT-like architecture; we cannot know whether this is a general property of TTT for 3D reconstruction or specific to VGGT's design.
Narrow difficulty range in large-scale experiments. The large-scale experiments (Figure 3) use 7scenes, which consists of small indoor rooms with dense video coverage. This is a relatively "easy" setting for multi-view reconstruction—high overlap, consistent lighting, limited scale. The paper does not test large-scale reconstruction on the challenging in-the-wild scenarios shown in the teaser (Rome landmarks from tourist photos). The teaser figure (Figure 0) shows reconstructions of the Colosseum, Castel Sant'Angelo, Pantheon, and Trevi Fountain, but no quantitative metrics are reported for these scenes. This is a significant gap between the demonstrated capability (the teaser) and the evaluated capability (7scenes).
Pointmap evaluation on keyframes only. For large image collections, ICP alignment of full point clouds is computationally expensive, so the paper evaluates on 10 equally spaced keyframes while treating other frames as supporting views. This means the reported CD and Normal Consistency metrics capture only the quality of those 10 views, not the full scene reconstruction. If the supporting views contain unique geometry not visible in the keyframes, their quality is not reflected in the metrics. The paper does not discuss how keyframe selection might bias the results.
No statistical significance reporting. None of the tables or figures include confidence intervals, standard deviations, or significance tests. Given that many metrics are computed on relatively small test sets (e.g., 7scenes has only 7 scenes), the reported differences may not be statistically reliable. The TTT3R comparison appears robust given the consistent 2–2.5× improvements across multiple datasets, but the quantitative gap between VGG-T³ and VGGT (Table 1, Table 2) could be within the noise for some metrics.
Camera pose as an acknowledged failure mode. Section 4.1 explicitly states that VGG-T³ "struggles on camera pose estimation" and attributes this to VGGT's heterogeneous token types (image patches vs. camera tokens). This is a significant limitation for practical use—camera poses are essential for most downstream applications of 3D reconstruction (novel view synthesis, AR, robotics). The paper does not propose a solution or demonstrate that pose quality is sufficient for any specific application.
Visual localization comparison gap. Table 5 compares only against TTT3R. A minimal additional baseline would be a retrieval-based approach (e.g., using VGGT features with NetVLAD) or a PnP-based pipeline using VGGT-predicted 2D–3D correspondences. Without such baselines, we cannot assess whether the unified mapping+localization capability offers practical advantages over simply running an off-the-shelf localizer on the VGG-T³ reconstruction.

Missing experiments that would strengthen the paper:

In-the-wild quantitative evaluation. Running the large-scale protocol on a dataset like MegaDepth, 1DSfM, or the Rome landmarks shown in the teaser would directly validate the practical claim. The teaser is qualitatively impressive but scientifically insufficient.
Ablation on MLP capacity. The paper uses a fixed MLP architecture ( $1024 \to 4096 \to 1024$ ) for all experiments. Varying the hidden dimension or depth would characterize the capacity-accuracy tradeoff and help determine whether the remaining gap to softmax attention is capacity-limited or representation-limited.
Generalization to other base architectures. Applying the same linearization recipe to Fast3R or π³ would test whether the approach is architecture-specific.
Fine-tuning data scale ablation. The paper uses 100k steps on a large dataset. How much fine-tuning data is actually needed? Could a lighter fine-tuning protocol (fewer datasets, fewer steps) still achieve reasonable performance?
Latency breakdown. The paper reports total runtime but does not decompose it into tokenization, per-image attention, TTT optimization, and decoding. Such a breakdown would clarify which components dominate at different scales and guide future optimization.

Overall, the experiments convincingly demonstrate the scalability contribution (linear time, competitive quality at large scales) and the relative advantage over TTT3R. The visual localization results establish technical feasibility. The remaining accuracy gap to softmax attention, the camera pose weakness, and the lack of in-the-wild quantitative evaluation are the primary limitations that temper the claims.

6. Limitations and Trade-offs

Fundamental Capability Ceiling: Hard Problems Are Outside the Method's Reach

The assumption or constraint. VGG-T³ inherits its base model's capabilities; the TTT linearization compresses existing knowledge but does not create new geometric understanding. The paper explicitly acknowledges this several times. Section 5 states: "there is still a gap w.r.t. softmax attention, especially in the wide-baseline setting." Appendix D, Figure 8b shows specific failure cases on Waymo sequences with "more complex scene layouts" where "the reconstruction quality is degraded," and the paper notes: "This suggests that future work should focus on reconciling the fixed expressivity of the MLP scene representation with the high accuracy of quadratic attention."

The consequence. The method cannot reconstruct scenes that the base VGGT model cannot reconstruct. This is not merely an accuracy gap—it represents a hard capability boundary. For scenes with wide baselines, low overlap, or complex geometry, the gap to softmax attention is large enough that VGG-T³'s output may be unusable for downstream applications. On the hardest benchmark categories, the paper shows VGG-T³ substantially underperforming VGGT: on ETH3D pointmap estimation (Table 1), VGG-T³ achieves CD of 0.044 versus VGGT's 0.020—more than 2× worse. On Sintel video depth (Table 2), VGG-T³ achieves Abs. Rel. of 0.289 versus VGGT's 0.204. On camera pose estimation (Table 3), VGG-T³ is consistently worse than VGGT across all datasets, with ATE on Sintel reaching 0.203m versus VGGT's 0.110m. A practitioner facing a challenging reconstruction task has no principled way to know in advance whether VGG-T³ will produce acceptable results or whether they should fall back to VGGT (paying the quadratic cost).

What evidence exists in the paper. The evidence is distributed across multiple experiments. Table 1 shows the accuracy gap on pointmap estimation widening on more challenging datasets (DTU: VGG-T³ actually beats VGGT; ETH3D: VGG-T³ is 2.2× worse). Table 2 shows the gap on depth estimation being dataset-dependent (minimal on KITTI, large on Sintel). Figure 8b in Appendix D provides qualitative failure cases on Waymo. The paper does not systematically characterize which scene properties predict the accuracy gap—is it baseline width, number of views, scene scale, texture sparsity, or something else? This missing characterization makes it hard for practitioners to assess risk.

Mitigation status. The paper acknowledges this limitation (Section 5: "future work should focus on reconciling the fixed expressivity of the MLP scene representation with the high accuracy of quadratic attention") but proposes no concrete mitigation. Suggestions like "adapting the amount of computation depending on scene complexity and designing more expressive linear attention mechanisms" (Appendix D) are aspirational. The limitation is fundamental to the approach—the MLP has fixed capacity—so closing the gap likely requires architectural innovations beyond the scope of this paper.

Camera Pose Estimation Degradation: A Critical Downstream Bottleneck

The assumption or constraint. VGG-T³ replaces all global attention layers with TTT layers, but VGGT uses a heterogeneous token structure for camera pose prediction: dedicated camera tokens are appended to image patch tokens immediately before the attention layer. Section 4.1 explains the consequence: "We suspect this is related to VGGT's special treatment of camera pose, where a dedicated camera token is appended to the image tokens immediately before the attention layer, effectively creating two input 'modalities'. This heterogeneous structure may be challenging for the MLP within the TTT layer to memorize."

The consequence. Camera pose accuracy degrades significantly under VGG-T³, and this has cascading effects. On TUM-RGBD (Table 3), VGG-T³ achieves ATE of 0.058m versus VGGT's 0.018m—more than 3× worse. On ScanNet: 0.124m versus 0.045m (2.8× worse). On Sintel: 0.203m versus 0.110m (1.8× worse). This is not just a metric concern—imprecise camera poses undermine downstream applications. For novel view synthesis, incorrect poses cause misalignment artifacts. For AR, pose errors cause virtual objects to drift. For robotics, pose errors degrade mapping and planning. Furthermore, the paper notes (Appendix A) that because camera poses are imprecise, VGG-T³ must use the direct pointmap prediction head rather than the more accurate unprojection from depth—meaning the pose errors indirectly degrade geometry quality as well, since the model cannot leverage the depth-to-pointmap pathway that VGGT finds superior.

On ordered sequences, TTT3R actually outperforms VGG-T³ on camera pose (Table 3: ATE 0.032m vs. 0.058m on TUM-RGBD). This means the linear-time method that VGG-T³ otherwise dominates surpasses it on this specific but critical metric. The paper's primary advantage on camera pose is robustness to input ordering (TTT3R degrades to 0.194m on unordered TUM-RGBD, while VGG-T³ stays consistent)—but this is a functionality advantage, not an accuracy advantage.

What evidence exists in the paper. Table 3 provides the direct comparison. The paper's attribution of the problem to heterogeneous token types is speculative rather than experimentally validated—there is no ablation testing this hypothesis (e.g., removing camera tokens and predicting pose differently, or using separate MLPs for image and camera tokens). The claim that "camera tokens of all mapping images" are retained as input to the camera head during localization (Appendix A) suggests the camera head retains some softmax attention, meaning the linearization is incomplete for this pathway.

Mitigation status. The paper does not attempt to solve this problem. It acknowledges the degradation (Section 4.1) and works around it (using direct pointmap prediction instead of unprojection, Appendix A) rather than addressing the root cause. This is a significant practical limitation because camera poses are essential output for most reconstruction pipelines. A practitioner who needs accurate poses has no recourse except to fall back to quadratic-attention VGGT or a separate pose estimation method.

Unaccounted Difficulty Estimation Cost: The $4\times$ Speedup Should Amortize an Expensive Prerequisite

The assumption or constraint. VGG-T³ operates on image collections where all views are provided upfront for offline reconstruction. Unlike the paper analyzed in the prior sections, VGG-T³ does not explicitly require a difficulty estimation step. However, there is an analogous hidden cost: the method requires knowing that all views belong to the same scene and have sufficient co-visibility for reconstruction to be possible. The training data is constructed using a greedy co-visibility sampling algorithm with a pre-computed co-visibility matrix (Appendix A: "We pre-compute the required co-visibility matrix via a depth consistency check"). At inference time, the paper assumes this co-visibility structure exists.

The consequence. In real-world deployment with in-the-wild image collections (exactly the scenario the teaser figure advertises—tourist photos of Rome landmarks), the input images are not pre-filtered for co-visibility. Outlier images (wrong landmark, interior shots, close-ups of details) will be fed to the model, potentially degrading the reconstruction in unpredictable ways. The paper provides no mechanism for detecting or handling such outliers. The co-visibility assumption is satisfied in the evaluated benchmarks (7scenes, NRGBD, ScanNet) because they consist of connected video sequences, but these are not representative of the "Building Rome in a Day" scenario. The gap between the evaluated setting and the advertised application is significant.

More subtly, the inference-time optimization (the TTT inner loop) processes all tokens with equal weight. Outlier tokens from irrelevant images will contribute gradients that push the MLP parameters away from the correct scene representation. Unlike softmax attention—where an irrelevant image's tokens would receive low attention weights because their keys don't match relevant queries—the TTT objective (Equation 3) sums over ALL tokens indiscriminately. There is no built-in mechanism to down-weight or ignore irrelevant information.

What evidence exists in the paper. The paper does not evaluate robustness to outlier images or irrelevant views. All benchmarks use curated, co-visible image sets. The teaser (Figure 0) implies the method works on in-the-wild tourist photos, but no quantitative evaluation on such data is provided. The co-visibility threshold of 0.3 used during training data construction (Appendix A) is mentioned but its impact is not ablated. We do not know what happens when this threshold is violated at inference time.

Mitigation status. Completely unaddressed. The paper does not discuss outlier handling, co-visibility estimation at inference time, or any filtering mechanism. A practitioner deploying this on real-world photo collections would need to build a separate preprocessing pipeline to cluster images by scene and filter for co-visibility—adding cost and complexity not reflected in the headline inference times.

Single Architecture, Single Initialization: Unknown Generalization to Other Feed-Forward Models

The assumption or constraint. VGG-T³ is exclusively evaluated as a linearization of VGGT. The paper's method is described in general terms—replacing global attention layers with TTT—but all experiments use the same base architecture, the same pre-trained checkpoint, and the same fine-tuning dataset. Section 4 states: "We start from the public VGGT checkpoint and convert it to a linearized model by replacing all global attention layers with TTT layers."

The consequence. We cannot know whether the approach transfers to other multi-view reconstruction architectures. VGGT has specific properties that may be important: it uses QK normalization (which the paper identifies as problematic and replaces with $L_2$ norm), it has alternating per-image and global attention blocks, and it uses a specific token structure (image patches + camera tokens + register tokens). Other architectures—Fast3R, π³, DUSt3R variants—have different designs. Fast3R, for instance, uses a different tokenization and attention pattern. Would the LayerNorm removal and ShortConv2D design transfer? Would training-from-scratch TTT perform differently on a different base architecture? The paper's key finding that post-training linearization dominates training from scratch (Table 6) might be specific to VGGT's training dynamics.

This matters practically because the field is moving rapidly, and VGGT may not remain the dominant architecture. If VGG-T³'s approach is VGGT-specific, its shelf life is limited. If it generalizes, it becomes a broadly applicable technique.

What evidence exists in the paper. None. No other base architecture is tested. The ablation in Table 6 showing that training-from-scratch TTT "gets stuck in a local optimum" is conducted on a VGGT-like architecture (the paper says "All models use the same base architecture" in Section 4.4). We do not know whether this failure mode is inherent to TTT for multi-view reconstruction or specific to VGGT's design choices.

Mitigation status. Not addressed. The paper does not claim generalization to other architectures and does not discuss what properties of VGGT make it amenable to TTT linearization. A practitioner using a different feed-forward reconstruction model (e.g., Fast3R) has no guidance on whether and how to apply this technique.

The Distributed Inference Advantage Relies on an Unusual Communication Pattern

The assumption or constraint. VGG-T³'s distributed inference strategy (Section 3.3, Table 4) exploits the decomposition of the TTT gradient as a sum over per-token losses (Equation 5). Each GPU processes a subset of images, computes local gradients, and then synchronizes via all-to-all communication before updating the shared MLP weights. The paper states: "we shard images such that each GPU only processes a subset $s$ " and "synchronize the MLP weights across GPUs by performing all-to-all communication which is efficient due to their small size."

The consequence. This approach requires a communication step in every global attention layer. With 12–24 transformer layers, the all-to-all communication happens 12–24 times per forward pass, once per layer's TTT optimization. While the paper argues this is efficient because the MLP weights are small, the latency of many small all-to-all operations may dominate in practice, especially across nodes with network interconnect rather than within a single node with NVLink. The paper reports near-linear scaling on up to 4 GPUs (Table 4: 3.26× speedup on 4 GPUs for 1000 images), but these are almost certainly single-node experiments (all GPUs sharing NVLink). Multi-node scaling—where network latency is orders of magnitude higher—is untested. The paper contrasts this with VGGT's need for "carefully engineered context-parallel implementations," but VGGT's attention is a single large operation per layer that can be optimized with fused kernels, while VGG-T³ requires many small synchronization points.

Additionally, the paper's comparison to TTT3R—"TTT3R is not compatible with multi-GPU inference due to its autoregressive processing" (Section 4.2)—is valid but highlights a different issue: VGG-T³'s distributed inference works because it is offline (all images available at once). For applications that actually need autoregressive processing, VGG-T³'s advantage disappears.

What evidence exists in the paper. Table 4 shows latency scaling on 1, 2, and 4 GPUs. The scaling is good but sub-linear: 3.58× on 4 GPUs for 2000 images. The paper does not report communication overhead, does not test multi-node configurations, and does not compare against VGGT's distributed inference scaling in the same setup. The claim that all-to-all communication is "efficient due to their small size" is qualitative, not quantitative.

Mitigation status. Not addressed. The paper presents distributed inference as a strength but does not analyze its limits. A practitioner deploying this across multiple nodes might find that communication latency erodes the speedup, particularly for smaller image collections where the computation-per-communication ratio is unfavorable.

No Characterization of Memory-Limited Regimes or Consumer Hardware

The assumption or constraint. The paper's inference strategy for single-GPU processing (Section 3.3) describes offloading minibatches to CPU memory and loading them one at a time: "we can then compute the update for the entire sequence by loading a minibatch at a time to device memory, compute the gradient, and off-load minibatch back to host memory." This enables processing of arbitrarily large collections on a single GPU, regardless of GPU memory capacity.

The consequence. The paper reports all runtimes on NVIDIA A100-80GB GPUs. We have no measurements on consumer hardware (e.g., RTX 3090 with 24GB, RTX 4090 with 24GB) or on older datacenter GPUs (V100 with 16–32GB). The CPU offloading strategy fundamentally trades memory for time—offloading minibatches over PCIe is orders of magnitude slower than keeping them in GPU memory. The paper's headline numbers (58 seconds for 1000 images) are measured on an A100, but a practitioner with a consumer GPU would experience dramatically different performance because:

The base VGGT model (frozen tokenizer, per-image attention, prediction heads) has substantial memory and compute requirements even without the global attention component. The paper does not report memory usage for any component.
CPU offloading on a system with slower PCIe (common in consumer setups) would increase the time per minibatch substantially.
Small GPU memory means smaller minibatches, which means more iterations of the load-compute-offload cycle, further amplifying the PCIe bottleneck.

The claim that VGG-T³ enables processing "on a single GPU" is technically true for an A100-80GB, but the paper does not establish the minimum GPU requirements or characterize the performance degradation on more constrained hardware. This matters because one of the paper's motivating scenarios is democratizing large-scale reconstruction—making it accessible without datacenter resources.

What evidence exists in the paper. None. All experiments use A100-80GB GPUs. Memory usage is never reported. The CPU offloading strategy is described in principle (Section 3.3) but its actual runtime impact is never measured or compared to the in-memory case.

Mitigation status. Not addressed. The paper's "single GPU" claim should be qualified as "single high-memory datacenter GPU with CPU offloading." A practitioner reading the headline numbers should understand that consumer hardware will produce very different runtimes, likely making the method impractical for very large collections without a high-end GPU.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing of the scalability problem in feed-forward 3D reconstruction that opens a new design axis. Rather than treating global attention as an operation to be approximated (token merging, sparse patterns) or restricted (local windows, causal masking), VGG-T³ reinterprets the Key-Value cache not as a database to be queried but as a mapping to be compressed. This shift from "searching the KV space" to "fitting the KV function" is the paper's deepest contribution, and it is not incremental—it changes the architectural question from "how do we make attention cheaper?" to "what is the minimal representation that captures the information attention would extract?"

The practical consequence is that offline, bi-directional global reasoning is now achievable in linear time. Prior to this work, the field faced a hard tradeoff: accept quadratic scaling for global consistency (VGGT, Fast3R) or accept accuracy degradation for linear scaling (CUT3R, TTT3R). VGG-T³ demonstrates that this tradeoff is not fundamental—it is an artifact of the retrieval-based formulation of attention. By compressing the scene into MLP weights through test-time optimization, VGG-T³ achieves the scaling behavior of online autoregressive methods (Figure 3: comparable runtime to TTT3R at 1k images) while preserving the reconstruction quality of offline global methods (Table 1: 2–2.5× lower CD than TTT3R, approaching VGGT). No prior method simultaneously occupied this point on the Pareto frontier.

The paper also provides a diagnostic that shifts research priorities. The ShortConv2D discovery—that a naive TTT objective degenerates because keys and values share a linear projection structure—reveals that the bottleneck in linearized attention is not simply MLP capacity but objective design. The fact that a 3×3 convolution on values breaks this degeneracy (Table 9, Appendix C) suggests that the gap between linear and quadratic attention may be bridged not by larger MLPs but by better-constructed self-supervised compression objectives. This redirects attention from capacity scaling (the natural instinct when MLPs underperform attention) to task design—what should the compression objective actually ask the MLP to learn?

The visual localization proof-of-concept (Section 4.3, Table 5) demonstrates that compressed scene representations can generalize to unseen viewpoints, which is not obvious a priori. One might have expected the MLP to overfit to the specific views seen during optimization, producing a degenerate representation that only works for those exact camera poses. The fact that query images from novel poses can be localized—even across a 7-year temporal gap with appearance changes (Appendix D, Figure 7)—provides preliminary evidence that the MLP captures genuine 3D structure. This opens the possibility of unified reconstruction and localization pipelines within a single model, eliminating the traditional boundary between mapping and querying.

Methodologically, the paper establishes that post-training linearization dominates training from scratch for multi-view reconstruction (Table 6). This is consistent with findings in the language model literature (Kasai et al., 2021; Mercat et al., 2024) but was not guaranteed to transfer—3D reconstruction has different data characteristics, different attention patterns, and different architectural constraints. The failure of training-from-scratch TTT (which "gets stuck in a local optimum") suggests that softmax attention pretraining organizes the QKV spaces in ways that make the compression problem well-posed, and that discovering this organization directly through TTT is substantially harder. This has implications for any future work attempting to build linear-complexity reconstruction models from scratch: pretrain with quadratic attention first, then linearize.

The practical speedup numbers (11.6× at 1k images, 33× at 2k images) establish a new baseline for what "real-time large-scale reconstruction" means. Prior feed-forward methods were considered fast relative to iterative SfM (minutes vs. hours), but VGG-T³ pushes the boundary to under one minute for thousand-image collections, making interactive-scale reconstruction feasible for the first time. This is not merely an engineering improvement—it enables use cases (real-time mapping from streaming photo uploads, on-demand reconstruction of event photo collections) that were previously impractical regardless of hardware budget.

Follow-Up Research This Work Enables

Characterizing the MLP capacity bottleneck as a function of scene complexity. The paper shows that increasing optimizer steps from 1 to 2 improves quality on long sequences but that 3–4 steps show diminishing returns (Appendix C, Figure 4). This suggests a two-phase behavior: first optimizer-limited (more steps help compress the information), then capacity-limited (the MLP saturates). A systematic study sweeping MLP hidden dimension (currently fixed at 4× input), depth (currently 1 hidden layer + SwiGLU), and scene complexity (measured by number of distinct surfaces, baseline width, or viewpoint variation) would characterize where each phase transition occurs. The key measurement would be reconstruction error as a function of MLP parameter count for scenes of varying complexity, analogous to scaling law studies in language modeling. If the capacity bottleneck is the primary limiter on wide-baseline scenes (where VGG-T³ shows the largest gap to VGGT), then this points toward dynamic capacity allocation—spending more MLP parameters on complex scenes and fewer on simple ones, using a difficulty estimator analogous to the one in the reference paper's compute-optimal framework.

Generalizing the linearization recipe to other feed-forward architectures. The paper's approach is validated exclusively on VGGT. A direct replication study applying the same recipe—remove LayerNorm from QK projections, substitute $L_2$ norm, add ShortConv2D on values, fine-tune global attention parameters—to Fast3R (Yang et al., 2025) or π³ (Wang et al., 2025) would test whether the approach is architecture-general or VGGT-specific. Fast3R uses a different tokenization strategy and attention pattern; if the recipe transfers without modification, it suggests the design principles (post-training linearization, degeneracy breaking via spatial mixing) are broadly applicable. If it fails, the failure analysis would reveal which architectural properties of VGGT are essential—a more informative result than a success. The key metric is the relative accuracy gap between the linearized model and its quadratic-attention upper bound, compared to the gap VGG-T³ achieves versus VGGT.

Adaptive optimizer steps as a test-time scaling mechanism. The paper demonstrates that longer sequences need more optimizer steps (Figure 2a), but the number of steps is currently a fixed hyperparameter (2). A dynamic policy that monitors the TTT loss during optimization and continues stepping until convergence slows below a threshold would automatically allocate computation proportional to scene difficulty, analogous to the difficulty-conditioned allocation in the reference paper. The experiment would compare fixed-2-step VGG-T³ against an adaptive-step variant on a benchmark with controlled difficulty variation (e.g., 7scenes with varying image counts). The hypothesis is that adaptive stepping would close the gap to VGGT on complex scenes (where 2 steps may be insufficient) while saving computation on simple scenes (where 1 step suffices). A negative result—adaptive stepping not helping—would indicate that the bottleneck is MLP capacity rather than optimization convergence, providing a clear diagnostic.

Training a difficulty predictor to enable compute-optimal allocation for reconstruction. The paper does not account for the fact that VGG-T³'s inference cost scales linearly with the number of images but uses the same 2 optimizer steps and identical MLP capacity regardless of scene difficulty. A system that estimates scene complexity before reconstruction—perhaps from the co-visibility graph structure, the number of images, or a lightweight forward pass on a subset of views—could then allocate the MLP capacity, optimizer steps, and even the decision of whether to use VGG-T³ or fall back to VGGT adaptively. The reference paper's compute-optimal framework (difficulty bins + per-bin strategy selection) is directly applicable here: bin scenes by estimated reconstruction difficulty, select the optimal (model choice, MLP size, optimizer steps) per bin via cross-validation, and measure whether adaptive allocation improves the accuracy-vs-runtime Pareto frontier. The key challenge is defining a cheap difficulty estimator—co-visibility density, number of input views, or average PRM-like confidence score from an initial forward pass—that predicts the linearization accuracy gap.

Closing the camera pose gap through token-type-aware compression. The paper identifies heterogeneous token types (image patches vs. camera tokens) as the likely cause of VGG-T³'s degraded camera pose estimation (Section 4.1), but proposes no solution. A targeted study that trains separate MLPs for image tokens and camera tokens within the same global attention layer—or that projects camera tokens into the same representation space via an additional learned adapter before TTT compression—would test whether the degradation is fixable within the linear-time framework. The experiment would compare pose accuracy (ATE on TUM-RGBD, ScanNet, Sintel) across three conditions: baseline VGG-T³ (single MLP for all tokens), separate MLPs per token type, and a control using quadratic attention for camera tokens only (hybrid linear/quadratic). If separate MLPs recover most of the accuracy gap, it confirms the heterogeneity hypothesis and provides a practical fix. If not, the degradation is more fundamental (perhaps camera pose requires pairwise token interactions that MLP compression cannot capture regardless of architecture), which would set a clear boundary on the approach.

In-the-wild benchmarking on tourism-scale photo collections. The paper's teaser (Figure 0) shows reconstructions of Roman landmarks from tourist photos, but no quantitative evaluation is performed on such data. A rigorous evaluation on standard in-the-wild SfM benchmarks—such as the 1DSfM dataset (Wilson and Snavely, 2014), the MegaDepth test scenes (Li and Snavely, 2018), or a curated set of Wikimedia Commons landmark collections—would validate the paper's primary advertised use case. The experiment would compare VGG-T³ against VGGT, COLMAP, and GLOMAP on standard SfM metrics: number of registered images, reprojection error, and reconstruction completeness. This would also stress-test the unaddressed co-visibility assumption: real tourist photo collections contain outlier images (wrong landmark, interior shots, close-ups) that VGG-T³'s indiscriminate TTT optimization would process with equal weight. A negative result—VGG-T³ performing well on curated benchmarks but degrading on truly uncurated collections—would precisely characterize the method's practical domain of applicability and motivate outlier-handling extensions.

Practical Applications and Downstream Use Cases

Rapid 3D mapping from event photo collections. Consider a wedding, conference, or festival where hundreds of attendees upload photos to a shared album within hours. These photo collections are unordered, taken from diverse viewpoints with different cameras, and lack any pose metadata. Classical SfM pipelines can reconstruct such collections but often require hours of processing and are brittle to matching failures. VGG-T³ can process a 500-image collection in approximately 30 seconds (extrapolating from the 58-second runtime for 1000 images in Table 4), producing a dense 3D reconstruction with per-image depth maps in under a minute on a single datacenter GPU. This makes same-day 3D reconstruction feasible for the first time—event organizers could provide attendees with an explorable 3D model before the event ends, rather than days or weeks later. The practical bottleneck is not reconstruction speed but image upload and preprocessing time, which is naturally parallelized.

Scalable digital twin creation for urban environments. City-scale digital twins require processing millions of images from heterogeneous sources—aerial surveys, street-level mapping cars, drone footage, and crowd-sourced photos. Current feed-forward methods cannot process such collections globally due to quadratic attention; chunked methods lose global consistency. VGG-T³'s linear scaling and distributed inference (Table 4: 3.58× speedup on 4 GPUs for 2000 images) enable a practical pipeline where images are sharded across GPU clusters, each processing a spatial sub-region with global consistency within that region, then aligned at region boundaries. The 33× speedup at 2000 images means that a collection that would take VGGT 27 minutes per 2k-image chunk can be processed in under 1 minute per chunk, reducing a week-long reconstruction job to hours. The key engineering challenge is region boundary alignment—VGG-T³'s frozen MLPs can be queried with images from adjacent regions to verify geometric consistency, using the visual localization capability demonstrated in Section 4.3.

On-device visual localization with cloud-computed maps. A mobile AR application could use VGG-T³ to reconstruct a venue (museum, shopping mall, historic site) from a pre-captured image collection on a cloud GPU, then distribute the frozen MLP weights (a few MB per scene) to user devices. When a user points their phone camera at the venue, the app runs only the query step of VGG-T³—a single forward pass through the frozen MLPs—to estimate depth and camera pose relative to the pre-computed map, without any feature matching, PnP solving, or cloud communication. Table 5 shows that this yields usable localization accuracy (61.2% of queries within 5° and 5cm on 7scenes), which is sufficient for AR overlay applications where coarse alignment is acceptable. The key advantage over traditional localization pipelines is no per-query cloud dependency: the entire map is local, inference is ~10–50ms on a phone GPU (since it is a single-image forward pass through a compact model), and no image data leaves the device. The current limitation is accuracy—specialized localizers like Reloc3R significantly outperform on pose precision—so this use case is viable for applications tolerant of moderate pose error, such as information overlays rather than precise measurement tools.