ArXiv: 2510.13802

🎯 Pitch

This paper shows that a single neural network can predict the continuous 3D path of every pixel in a video without ever using depth sensors or optical flow. The resulting 'trajectory field' is generated in one feedforward pass, making full 4D reconstruction over an order of magnitude faster than methods requiring per-scene optimization.

1. Executive Summary

This paper proposes Trajectory Fields, a 4D video representation that assigns a continuous parametric 3D trajectory to every pixel in every frame, and introduces Trace Anything, a feed-forward neural network that predicts the entire trajectory field in a single forward pass without requiring auxiliary estimators (e.g., monocular depth, optical flow) or per-scene iterative optimization. Trained on a new large-scale synthetic data platform—10,000+ videos with dense 3D trajectory annotations—Trace Anything achieves state-of-the-art results on the authors' new all-to-all trajectory field benchmark while running over an order of magnitude faster than competing approaches (2.3 seconds vs. 22.5–231.6 seconds for video inference), establishing that feed-forward 4D reconstruction can match or exceed optimization-based methods in accuracy only when evaluated on the synthetic domain for which training data is available.

2. Context and Motivation

The Core Problem: We Lack a Direct, Unified 4D Representation for Video Dynamics

The central gap this paper addresses is a fundamental disconnect in how computer vision represents dynamic scenes. To understand what the authors are really after, we need to think about what happens when we watch a video: each pixel is not just a color value—it corresponds to some point in the 3D world that moves over time. If we could recover that 3D motion for every pixel, we would have a complete geometric description of the dynamic scene. Current methods, however, do not directly model this. Instead, they produce a patchwork of intermediate representations that must be stitched together post-hoc.

The paper identifies the bottleneck clearly (Section 1):

"Rather than relying on additional estimators such as depth, flow, or tracking, or on heavy per-scene optimization, we observe that the atomic elements of video, its pixels, naturally trace out 3D trajectories in the world, which acts as the atomic primitive of dynamics."

This is the intellectual starting point: the pixel's 3D trajectory is the primitive—the indivisible unit—of scene dynamics. Yet no existing representation directly models this primitive. Instead, the field has settled on a fragmented pipeline: estimate depth per frame, estimate optical flow between frames, track 2D points, lift to 3D using depth, perform global optimization to enforce consistency, and so on. Each step introduces errors that compound. The paper's core bet is that by making 3D trajectories the explicit output of a single model, this error accumulation can be avoided entirely.

Why This Matters: Beyond Academic Interest

The practical significance of solving this problem spans multiple domains:

Spatial intelligence and embodied AI. For a robot to understand and interact with a dynamic world, it needs more than object detection—it needs to know where things are in 3D and how they move through space over time. Current manipulation systems either rely on 2D tracking (which ignores depth) or sparse 3D keypoints (which miss fine-grained motion). A dense, per-pixel 3D trajectory field would give a robot complete spatial awareness of how every visible surface point moves, enabling more precise manipulation. The paper demonstrates this implicitly through the goal-conditioned manipulation experiments on BridgeData (Section 5.2, Figure 6), where the model predicts 3D trajectories of robot arms and objects from just initial and goal images.

Video editing and content creation. The paper discusses how existing 4D representations for Novel View Synthesis (NVS) produce immersive "bullet time" effects (Section 2), but notes these methods typically assume precomputed camera poses or point clouds. A trajectory field—which directly encodes 3D geometry and motion—could serve as the initialization or geometric scaffold for these NVS systems, handling dynamic scenes where traditional structure-from-motion fails due to moving objects.

Efficiency in deployment. The paper repeatedly emphasizes that Trace Anything runs in a single forward pass without per-scene optimization (unlike methods such as OmniMotion or DUSt3R-style approaches that require costly global alignment). This efficiency is not just a convenience—it's essential for any real-time or near-real-time application. If understanding dynamic 3D scenes requires minutes of optimization per video, it is unusable for interactive robotics, autonomous driving, or live video processing. The 2.3-second runtime on a single A100 (Table 1) represents a 10–100× speedup over competing methods, making the approach potentially viable for latency-sensitive applications.

A principled foundation for future research. On the theoretical side, the trajectory field formulation provides something the field has lacked: a mathematically clean, geometric primitive for modeling dynamics. By formalizing the output as a mapping from discrete pixel coordinates to continuous 3D functions (Equation 1), the paper creates a representation that is simultaneously dense (per-pixel), continuous (queryable at any time instant), and globally consistent (all trajectories exist in a shared world coordinate system). Previous work has achieved some of these properties individually, but never all three simultaneously in a single representation.

Where Existing Approaches Fall Short

The paper identifies three distinct families of prior work, each with specific limitations:

Dynamic 3D reconstruction methods produce disjoint per-frame outputs. The lineage from DUSt3R (Wang et al., 2024) through to Monst3R (Zhang et al., 2025), POMATO (Zhang et al., 2025), Easi3R (Chen et al., 2025), and St4RTrack (Feng et al., 2025) represents the state of the art in learning-based 3D reconstruction. These methods take images as input and output 3D pointmaps—but critically, each frame's pointmap is estimated independently (or pair-wise for DUSt3R-style methods). As the paper notes:

"These methods typically generate disjoint per-frame point clouds, relying on optical flow or 2D tracks for cross-frame correspondences, and their pairwise inference often requires costly per-scene optimization for global alignment." (Section 2)

This is a fundamental structural problem. Even if each per-frame pointmap is geometrically accurate, there is no built-in correspondence between frames. A point representing the tip of a person's nose in frame 1 and a point at a nearby location in frame 2 may or may not be the same physical point—the model provides no way to know. To establish correspondences, these methods bolt on optical flow or 2D trackers, which have their own error rates, and then run global optimization to align the disjoint point clouds. This is a classic pipeline error accumulation problem. Furthermore, much of this work assumes static scenes (DUSt3R, VGGT, Fast3R) and only the most recent extensions handle dynamics at all, often by relaxing static assumptions in ways that are computationally expensive.

Point tracking methods are sparse and cannot provide full-scene geometry. The point tracking tradition—from Particle Video (Sand & Teller, 2008) through TAP-Vid benchmarks (Doersch et al., 2022), CoTracker (Karaev et al., 2024), SpatialTracker (Xiao et al., 2024), and DELTA (Ngo et al., 2025)—focuses on following specific query points through a video. These methods excel at maintaining identity through occlusions and can produce long-range trajectories, but they have two fundamental limitations vis-à-vis the trajectory field goal. First, they are sparse: they track a fixed set of query points (typically sampled from the first frame), not every pixel. Second, they do not reason about global scene geometry—they produce trajectories in a relative coordinate system without knowledge of where things are in a consistent world frame. The 3D tracking variants (SpatialTracker, DELTA) lift 2D tracks to 3D using monocular depth estimators, introducing yet another source of potential error. As the paper puts it:

"Unlike prior approaches, our method bypasses monocular depth estimation and 2D trackers and directly predicts dense 3D trajectories in a feed-forward manner." (Section 2)

The "bypasses" language is deliberate: the authors are arguing that depth and 2D tracking are proxies for what we actually want (3D trajectories), and that directly predicting the end goal avoids the errors that accumulate when chaining proxies together.

4D NVS representations prioritize appearance over geometry and require auxiliary inputs. The extensive body of work on 4D representations for Novel View Synthesis—NeRF extensions (Gao et al., 2021; Li et al., 2021; Park et al., 2021; Pumarola et al., 2021), grid-based methods (Fridovich-Keil et al., 2023; Cao & Johnson, 2023; Attal et al., 2023), and 3D Gaussian Splatting variants (Wu et al., 2024; Luiten et al., 2023; Yang et al., 2024)—has produced stunning view synthesis results. However, these methods are designed for a fundamentally different goal: photorealistic rendering. They optimize for appearance quality, treat geometry as a means to an end, and typically assume known camera parameters (poses, intrinsics). The paper is explicit about this distinction:

"These efforts focus on photorealistic appearance and typically assume precomputed camera poses or point clouds. Our work is orthogonal: we propose a geometry-centric paradigm that directly infers trajectory fields from raw videos, emphasizing accurate 3D motion modeling." (Section 2)

This "orthogonal" framing is important. The authors are not claiming to beat NVS methods at rendering quality—they are claiming that geometric understanding of dynamic scenes is an important problem in its own right, one that NVS methods do not address directly. A system that can render a scene from novel views with photorealistic quality but cannot tell you where objects are in 3D or how they move is missing a capability that many applications (robotics, spatial reasoning, physics simulation) require.

The Data Bottleneck and the Synthetic Data Solution

A recurring theme in the paper—though stated subtly—is that the lack of large-scale training data with dense 3D trajectory annotations has been a primary obstacle to directly predicting trajectory fields. Existing synthetic datasets (Kubric, FlyingThings3D, PointOdyssey) are either limited in scale, biased toward rigid motion, or provide only sparse annotations. Real-world datasets like TAP-Vid provide 2D point tracks but lack 3D ground truth. This is the classic chicken-and-egg problem: you cannot train a model to predict dense 3D trajectories without dense 3D trajectory labels, and you cannot get those labels without a controlled environment.

The paper's response is the Trace Anything Data Platform (Section 4), a Blender-based synthetic data generation system that produces photo-realistic dynamic scenes with per-pixel 3D trajectory annotations, depth, semantics, flow, and camera poses—all automatically, at scale (10,000+ videos, 120 frames each). This is not merely an engineering contribution; it is the enabling condition for the entire approach. Without this data, training a model to predict dense per-pixel 3D trajectories from raw pixels would be infeasible. The platform's diversity—varied environments, articulated characters, rigid and non-rigid motion, smooth camera trajectories—is crucial because a model trained only on simple scenes would not generalize to the complex dynamics seen in real videos.

How the Paper Positions Itself

The paper positions trajectory fields as a new primitive for 4D video understanding—not just another reconstruction method or point tracker, but a representation that changes what we consider the fundamental output of video understanding. This is evident in how the paper frames its contributions:

The introduction explicitly distinguishes between "understanding dynamic scenes" (which requires joint reasoning over space and time) and "disjoint reconstruction of 3D space at each time step" (which prior methods do). The trajectory field is presented as the minimal representation that captures the joint spacetime structure—minimal because it models exactly the degrees of freedom that matter (per-pixel 3D motion) without over-parameterizing (e.g., with volumetric radiance fields for appearance).

The paper also positions its method in the emerging lineage of feed-forward geometric models (Fast3R, VGGT) that process all frames jointly rather than in pairs, sharing the "one-pass inference over all frames" philosophy but extending it to dynamics. It differentiates itself from the concurrent wave of dynamic extensions to DUSt3R (Monst3R, POMATO, Easi3R, St4RTrack) by arguing they still fundamentally produce disjoint per-frame outputs that require post-hoc correspondence establishment, while Trace Anything's outputs are inherently corresponding because the trajectory representation builds cross-frame consistency into the output format itself—every pixel's trajectory is defined for all time, so correspondences don't need to be "found," they are directly predicted.

Moreover, the paper introduces a subtle but important shift in evaluation philosophy. Existing point tracking benchmarks evaluate only "first-to-all" trajectories—you query a point in the first frame and track it forward. The Trace Anything Benchmark (Section 4) evaluates all-to-all: every pixel in every frame must have a complete trajectory spanning the whole sequence. This is a strictly harder task that requires the model to reason about dynamics jointly across the entire sequence, not just propagate forward from a single starting point. The benchmark also evaluates in world coordinates rather than relative coordinates, testing global geometric consistency. This protocol reflects the paper's ambition: trajectory fields are meant to be a complete 4D scene description, not just a tracking tool.

3. Technical Approach

3.1 Reader Orientation

What the system is: Trace Anything is a feed-forward neural network that takes an arbitrary collection of video frames as input and, in a single forward pass, outputs for every pixel in every frame a continuous 3D trajectory—a parametric curve that describes where that pixel's corresponding world-point is located in 3D space at any moment in time, past or future relative to the frame.

What problem it solves and the shape of the solution: The system solves the problem of fragmented 4D scene understanding by making 3D trajectories the explicit output primitive rather than a derived property inferred from intermediate estimates. Instead of the standard pipeline that chains together depth estimation, optical flow, 2D tracking, and global optimization, Trace Anything directly predicts a globally consistent, dense, per-pixel mapping from pixel coordinates to continuous 3D motion curves, eliminating the error accumulation that comes from composing multiple imperfect estimators.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major stages connected in a single feed-forward pipeline:

  1. Image Encoder — a shared-weight vision transformer that independently tokenizes each input frame into a set of feature tokens. Input: one RGB frame at a time. Output: a feature map for that frame.

  2. Fusion Transformer — an all-to-all transformer that processes the tokenized features from all frames jointly through interleaved frame-wise and global attention layers, integrating spatio-temporal context. Input: feature tokens from all frames. Output: globally-conditioned per-frame feature representations that know about the entire sequence.

  3. Control Point Head — a convolutional prediction head applied to the fused features. For each pixel in each frame, it outputs a fixed-size set of 3D control points, plus associated scalar confidence values. Input: per-frame fused features. Output: dense control point maps $\mathbf{P}_i \in \mathbb{R}^{D \times H \times W \times 3}$ and confidence maps $\hat{\Sigma}_i \in \mathbb{R}^{D \times H \times W}$ for each frame $i$.

  4. Curve Evaluation — a deterministic, non-learned module that converts control points into actual 3D coordinates at arbitrary query times using B-spline basis functions. Input: control points for a pixel and a query time $t$. Output: a 3D point $\mathbf{x}_{i,u,v}(t) \in \mathbb{R}^3$.

  5. Optional Auxiliary Heads — a timestamp head (predicts normalized temporal indices when metadata is unavailable) and a local-coordinate CP head (predicts trajectories in each frame's local camera coordinate system for applications requiring per-frame coordinates).

Information flows strictly forward: raw frames → image encoder → per-frame tokens → fusion transformer → globally-conditioned tokens → CP head → per-pixel control points → curve evaluation → 3D trajectories queryable at any time. The entire pipeline is trained end-to-end with losses applied directly on the evaluated 3D positions relative to ground truth.

3.3 Roadmap for the Deep Dive

  • First, the mathematical formulation of trajectory fields (Section 3.1 of the paper), which defines precisely what a trajectory field is—the domain, codomain, and parametric representation—and establishes the two consistency conditions (C1, C2) that an ideal trajectory field should satisfy.

  • Second, the network architecture (Section 3.2), walking through image encoding, fusion, control point prediction, and the curve evaluation module that converts learned parameters into actual 3D geometry.

  • Third, the training scheme (Section 3.3), covering the core trajectory loss, confidence adjustment, and the four regularization terms (static, rigidity, correspondence, timestamp) that encode geometric priors the model must learn.

  • Fourth, the supporting components: the data platform and the B-spline parameterization choice that underlie the entire training pipeline.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a representation and architecture paper whose core idea is that per-pixel 3D trajectories should be the direct output of a video understanding system, and that this can be achieved by predicting B-spline control points in a shared world coordinate system from a joint spatio-temporal transformer, supervised end-to-end with geometric losses.

Trajectory Field Formulation (Section 3.1)

The paper formalizes the output of video understanding as a mathematical field—a mapping from a discrete index space to a continuous function space. This formalism is not just notation; it defines the contract that the model must satisfy: for every pixel in every frame, produce a function that can be evaluated at any time to return a 3D position.

The Trajectory Field Definition

Given a video of $N$ RGB frames $\{I_i\}_{i=1}^N$, each of size $H \times W$, a trajectory field is the mapping:

T:[N]×[H]×[W]→C([0,1],R3)T : [N] \times [H] \times [W] \to C([0, 1], \mathbb{R}^3) T(i,u,v)=xi,u,v(⋅)T(i, u, v) = \mathbf{x}_{i,u,v}(\cdot)

where $[N] = \{1, 2, \dots, N\}$ is the set of frame indices, $[H]$ and $[W]$ are the pixel coordinate sets, $C([0, 1], \mathbb{R}^3)$ is the space of continuous functions from the unit interval to 3D Euclidean space, and $\mathbf{x}_{i,u,v} : [0, 1] \to \mathbb{R}^3$ is the continuous trajectory function assigned to pixel $(u, v)$ in frame $i$.

What it defines: This mapping takes any pixel in any frame and returns not a single 3D point, but an entire function that describes where that pixel's world-point is at any normalized time $t \in [0, 1]$. Time is normalized so that $t = 0$ corresponds to the start of the sequence and $t = 1$ to the end, making the representation independent of absolute frame rate or sequence duration.

Why this form: The codomain being $C([0, 1], \mathbb{R}^3)$—continuous functions rather than discrete 3D points—is the crucial design choice that distinguishes trajectory fields from per-frame point clouds. It means the model predicts an object that can be queried at any continuous time, not just at the discrete timestamps when frames were captured. This is what enables interpolation between frames and extrapolation beyond observed frames. A representation that output only discrete per-frame positions would require separate post-hoc interpolation logic to achieve the same capability, reintroducing the fragmentation the paper seeks to avoid.

Parametric Representation

The trajectory for a specific pixel is parameterized as a spline with $D$ control points. For each frame $i$, the model predicts a control point map:

Pi∈RD×H×W×3\mathbf{P}_i \in \mathbb{R}^{D \times H \times W \times 3}

where $\mathbf{P}^{(k)}_{i,u,v} \in \mathbb{R}^3$ is the $k$-th control point ($k \in \{0, 1, \dots, D-1\}$) for pixel $(u, v)$ in frame $i$. Given a set of basis functions $\{\phi_k(t)\}_{k=0}^{D-1}$ that define the interpolation scheme, the trajectory function is the weighted sum:

xi,u,v(t)=∑k=0D−1Pi,u,v(k)⋅ϕk(t)\mathbf{x}_{i,u,v}(t) = \sum_{k=0}^{D-1} \mathbf{P}^{(k)}_{i,u,v} \cdot \phi_k(t)

What it computes: For any query time $t$, the 3D position is a linear combination of the $D$ learned control points, where the combination weights are given by the basis functions evaluated at $t$. At $t = 0$, the trajectory evaluates to the first control point $\mathbf{P}^{(0)}_{i,u,v}$; at $t = 1$, it evaluates to the last control point $\mathbf{P}^{(D-1)}_{i,u,v}$ (for clamped B-splines or Bézier bases). Intermediate control points influence the curve's shape between the endpoints.

Why this form: Using a spline parameterization with a small, fixed number of control points ($D$ is 4, 7, or 10 in the paper's experiments) achieves a balance between representational capacity and compactness. A trajectory with completely unconstrained 3D positions at every time instant would require an impractical number of parameters. The spline enforces smoothness implicitly through the basis functions—the trajectory is guaranteed to be $C^2$ continuous for cubic B-splines—which is a useful geometric prior for physical motion (position, velocity, and acceleration are all continuous). Alternatives like direct per-frame position prediction for all time steps would not provide continuity guarantees and would scale poorly with sequence length. The paper's ablation (Appendix C, Table C) confirms that polynomial curves of similar degree underperform because their coefficients lack the geometric interpretability of control points.

Two Desired Consistency Conditions

The paper states two conditions an ideal trajectory field should satisfy:

  • (C1) Static regions collapse to degenerate trajectories. For any pixel $(u, v)$ in a static region (e.g., background walls, floor), all control points should be identical, meaning the trajectory is a single point $\mathbf{P}^{(0)}_{i,u,v} = \mathbf{P}^{(1)}_{i,u,v} = \dots = \mathbf{P}^{(D-1)}_{i,u,v}$. The trajectory function evaluates to the same 3D position for all $t$.

  • (C2) Corresponding pixels map to the same trajectory. If pixel $(u, v)$ in frame $i$ and pixel $(u', v')$ in frame $j$ both correspond to the same physical world point (e.g., the tip of a person's nose as seen from two different times or viewpoints), their control point sequences should be identical: $\mathbf{P}^{(k)}_{i,u,v} = \mathbf{P}^{(k)}_{j,u',v'}$ for all $k$. This means their trajectory functions $\mathbf{x}_{i,u,v}$ and $\mathbf{x}_{j,u',v'}$ are the same function.

These conditions are not enforced architecturally—they are approximate targets that the training losses encourage. The Static Degeneracy Deviation (SDD) and Correspondence Agreement (CA) metrics introduced in Section 5.3 are designed specifically to measure compliance with C1 and C2 respectively.

Cross-Frame Evaluation

The key operational capability of trajectory fields is evaluating a pixel's trajectory at another frame's timestamp. For a pixel $(u, v)$ in frame $i$, its predicted 3D position at the acquisition time $t_j$ of frame $j$ is:

Xi→j(u,v)=xi,u,v(tj)=∑k=0D−1Pi,u,v(k)⋅ϕk(tj)\mathbf{X}_{i \to j}(u, v) = \mathbf{x}_{i,u,v}(t_j) = \sum_{k=0}^{D-1} \mathbf{P}^{(k)}_{i,u,v} \cdot \phi_k(t_j)

What it computes: This operation takes a pixel from one frame, follows its predicted trajectory through time, and reads off where that world-point should be at the moment frame $j$ was captured. This is the mechanism that establishes cross-frame correspondence: if a pixel in frame $i$ and a pixel in frame $j$ both belong to the same physical point, then $\mathbf{X}_{i \to j}(u, v)$ should equal the 3D position of that corresponding pixel in frame $j$'s self-evaluation.

Self-evaluation as a special case. When $j = i$, the evaluation recovers frame $i$'s own 3D point map:

Xi(u,v)=xi,u,v(ti)=Pi,u,v(0)(for clamped B-splines)\mathbf{X}_i(u, v) = \mathbf{x}_{i,u,v}(t_i) = \mathbf{P}^{(0)}_{i,u,v} \quad \text{(for clamped B-splines)}

This means the first control point directly represents where the pixel's world-point is at the moment frame $i$ was captured, which gives the model a clean interpretation for what each control point means geometrically.

Network Architecture (Section 3.2)

The architecture is designed to process all frames jointly, produce per-pixel predictions in a globally consistent coordinate system, and avoid any iterative optimization or external estimators. The paper describes this as a "feed-forward geometric backbone" similar in spirit to Fast3R (Yang et al., 2025) and VGGT (Wang et al., 2025).

Image Encoder

Each input frame $I_i \in \mathbb{R}^{3 \times H \times W}$ is independently processed by a shared-weight image encoder—a Vision Transformer (ViT) that partitions the image into patches, encodes each patch into a token, and produces a feature map. The encoder weights are initialized from the Fast3R pretrained checkpoint (Section 5.1) and shared across all frames.

Why shared weights: Frame-independent encoding ensures the model can handle variable numbers of input frames without architectural changes, and shared weights mean the encoder treats all frames identically—there is no privileged "first frame" or "reference frame." This is essential for handling unordered image collections where no temporal ordering is provided.

Fusion Transformer

The per-frame feature tokens from the image encoder are passed into a fusion transformer that performs joint reasoning across all frames. The fusion transformer alternates between two types of attention:

  • Frame-wise (self) attention: attention is computed only among tokens belonging to the same frame, allowing the model to refine within-frame spatial representations.

  • Global (cross-frame) attention: attention is computed across all tokens from all frames simultaneously, allowing information to flow between any two spatial locations in any two frames.

When the input is a temporally ordered video, temporal index embeddings are added to the feature tokens before the fusion transformer, giving the model explicit knowledge of frame ordering. When the input is an unordered image collection, these embeddings are omitted, and the model must infer temporal relationships purely from visual content.

Why interleaved attention: Pure global attention would be computationally prohibitive for many frames at high resolution (the cost scales quadratically with total token count). Frame-wise attention reduces this cost by processing tokens within each frame independently between global attention blocks. This design is inherited from the Fast3R and VGGT architectures, which demonstrated its effectiveness for multi-view geometric reasoning.

Control Point Head

Applied to the fused features for each frame, the control point head is a small convolutional or transformer-based prediction module that outputs two quantities for every pixel:

  1. Control points $\mathbf{P}_i \in \mathbb{R}^{D \times H \times W \times 3}$: for each pixel, $D$ 3D vectors that parameterize its trajectory.

  2. Confidence scores $\hat{\Sigma}_i \in \mathbb{R}^{D \times H \times W}$: for each control point, a positive scalar representing the model's estimated reliability.

Predictions are in a shared world coordinate system, meaning the 3D positions predicted for different frames are directly comparable without any transformation. This is a critical design choice: it means the fusion transformer must have learned to coordinate the output spaces of all frames so that the same physical point gets approximately the same control points regardless of which frame it is predicted from. An optional local CP head also predicts control points in each frame's local camera coordinate system for applications that need per-frame coordinates.

A key architectural detail that the paper does not fully elaborate on is how the CP head achieves per-pixel predictions. Given that the fusion transformer operates on patch tokens (not individual pixels), the CP head must include some form of upsampling—likely a convolutional decoder or a progressive refinement module—to produce predictions at the original image resolution $H \times W$. The fact that the model outputs predictions at the input resolution is significant because it means trajectories are truly dense: every pixel, not just a sparse set of query points, receives a trajectory.

Why per-control-point confidence: The confidence prediction allows the model to express uncertainty about specific control points. A pixel that is occluded in some frames might have high confidence in its position at observed times (where the control points near $t_i$ should be accurate) but low confidence at other times. The confidence values participate in the loss function (Equation 9) to automatically weight training, downweighting gradients from unreliable predictions.

Curve Evaluation (Non-Learned)

Given the predicted control points and the pre-defined B-spline basis functions $\{\phi_k(t)\}$, the trajectory is evaluated by computing the weighted sum in Equation (3). This operation is deterministic (no learned parameters) and differentiable with respect to the control points and the query time $t$, enabling end-to-end training.

The basis functions $\phi_k(t)$ are cubic B-spline basis functions $N_{k,3}(t)$ defined by a knot vector, as detailed in Appendix B. The paper uses three knot vector configurations depending on the number of control points:

"For n = 4, 7, 10 control points, we define knot vectors with multiplicity 4 at t = 0 and t = 1 to ensure interpolation of the first and last control points"

The knot vectors are:

  • $n = 4$: $[0, 0, 0, 0, 1, 1, 1, 1]$ — a single cubic segment interpolating both endpoints.
  • $n = 7$: $[0, 0, 0, 0, 0.5, 0.5, 0.5, 1, 1, 1, 1]$ — two segments with $C^0$ continuity at $t = 0.5$.
  • $n = 10$: $[0, 0, 0, 0, 1/3, 1/3, 1/3, 2/3, 2/3, 2/3, 1, 1, 1, 1]$ — three segments with $C^0$ continuity at $t = 1/3$ and $t = 2/3$.

The multiplicity-4 knots at the endpoints mean that the first and last basis functions evaluate to 1 at $t = 0$ and $t = 1$ respectively, with all others evaluating to 0, guaranteeing endpoint interpolation. The internal knots have multiplicity up to 3, which reduces continuity to $C^0$ at segment boundaries—the curve remains continuous but its derivatives may be discontinuous, allowing the model to represent sharp changes in direction (e.g., a bouncing ball reversing at impact).

Why B-splines over Bézier curves: The paper's ablation (Table C) shows that B-splines with 10 control points achieve the best overall performance. B-splines offer local control—modifying one control point only affects the curve over a limited parameter range—which Bézier curves of high degree do not. This is important for representing complex motions where different temporal segments of a trajectory may need independent adjustment. Bézier curves do achieve comparable results in some configurations (Bezier with 4 control points gets EPEmix 0.299 vs. B-spline 10's 0.234), but the B-spline advantage grows with more control points, suggesting that local control matters for complex trajectories.

Timestamp Head (Optional)

When input frames lack metadata about their acquisition times, an auxiliary timestamp head predicts normalized timestamps $\hat{t}_i \in [0, 1]$ for each frame. This head is likely a small MLP applied to a pooled representation of each frame's fused features. The predicted timestamps then serve as the query times $t_j$ in the curve evaluation step (Equation 5). When metadata timestamps are available, they are used directly and the timestamp head is supervised with an L1 loss (Equation 10) to learn the mapping from visual content to temporal index.

The existence of this head means the model can, in principle, infer temporal ordering from visual content alone—if one frame shows a door half-open and another shows it fully open, the model can deduce which came first. This capability is demonstrated in the unstructured image set experiments (Figure 8), where the model receives no temporal ordering information but still produces plausible trajectory fields.

Training Scheme (Section 3.3)

The training scheme is designed around a simple intuition: a trajectory predicted from any frame should, when evaluated at any other frame's timestamp, land at the correct 3D location. The loss functions operationalize this intuition through direct 3D supervision, confidence weighting, and geometric regularization.

Core Trajectory Loss

For each pair of frames $(i, j)$ and each pixel $(u, v)$ in frame $i$ for which ground truth is available, the basic loss is the Euclidean distance between the predicted and ground-truth 3D positions:

ℓi→j(u,v)=∥Xi→j(u,v)−Xi→jgt(u,v)∥22\ell_{i \to j}(u, v) = \left\| \mathbf{X}_{i \to j}(u, v) - \mathbf{X}^{\text{gt}}_{i \to j}(u, v) \right\|_2^2

where $\mathbf{X}_{i \to j}(u, v)$ is the predicted 3D position (Equation 5) and $\mathbf{X}^{\text{gt}}_{i \to j}(u, v)$ is the ground-truth 3D position from the synthetic data.

What it computes: For a pixel in frame $i$, this loss evaluates how far its predicted trajectory is from the correct 3D position at frame $j$'s acquisition time. The squared Euclidean norm penalizes large errors quadratically, making the model sensitive to gross mispredictions.

Why pairwise supervision over all $(i, j)$ pairs: This loss is applied across many frame pairs during training, not just adjacent frames or just self-evaluation. This forces the model to learn globally consistent trajectories—if a trajectory is correct at $t_i$, $t_j$, and $t_k$ simultaneously, it must have the right shape for all intermediate times as well. Training only on self-evaluation (Equation 6) would only supervise the control points at the specific times when frames were captured, leaving the trajectory shape between frames unconstrained.

Confidence Adjustment

The network predicts per-control-point confidence values $\hat{\Sigma}^{(k)}_{i,u,v} > 0$. To obtain a scalar confidence for the full trajectory evaluation at time $t_j$, these per-control-point confidences are interpolated using the same basis functions:

Σ^i→j(u,v)=∑k=0D−1Σ^i,u,v(k)⋅ϕk(tj)\hat{\Sigma}_{i \to j}(u, v) = \sum_{k=0}^{D-1} \hat{\Sigma}^{(k)}_{i,u,v} \cdot \phi_k(t_j)

The confidence-adjusted loss is:

Ltraj-conf=1∣Ω∣∑(i,j)∑(u,v)∈Ω[Σ^i→j(u,v)⋅ℓi→j(u,v)+αlog⁡Σ^i→j(u,v)]\mathcal{L}_{\text{traj-conf}} = \frac{1}{|\Omega|} \sum_{(i,j)} \sum_{(u,v) \in \Omega} \left[ \hat{\Sigma}_{i \to j}(u, v) \cdot \ell_{i \to j}(u, v) + \alpha \log \hat{\Sigma}_{i \to j}(u, v) \right]

where $\Omega$ is the set of valid pixels with ground-truth supervision, $\ell_{i \to j}$ is the per-pixel squared error from Equation (7), $\hat{\Sigma}_{i \to j}(u, v)$ is the interpolated confidence, and $\alpha$ is a hyperparameter controlling the strength of the confidence regularization.

What it computes: This loss has two competing terms. The first term $\hat{\Sigma} \cdot \ell$ multiplies the 3D error by the confidence, so the model can reduce the loss by either predicting the position more accurately (reducing $\ell$) or predicting lower confidence $\hat{\Sigma}$ when accuracy is impossible. The second term $\alpha \log \hat{\Sigma}$ is a regularizer that penalizes low confidence—as $\hat{\Sigma} \to 0$, $\log \hat{\Sigma} \to -\infty$, creating a strong incentive to predict non-negligible confidence. The optimal confidence for a given expected error balances these two forces.

Why this form: This is derived from the negative log-likelihood of a Laplace distribution with predicted variance (or equivalently, a heteroscedastic Gaussian likelihood with predicted precision). The $\log \hat{\Sigma}$ term appears from the normalizing constant of the distribution. This formulation, common in uncertainty-aware deep learning (Kendall & Gal, 2017), has the property that the model learns to express higher confidence when its predictions are consistently accurate and lower confidence when they are not, without needing separate uncertainty labels. The paper does not explicitly state the value of $\alpha$, but it is typically set to 1 or tuned as a hyperparameter.

Timestamp Supervision

When ground-truth timestamps $t_i$ are available, the timestamp head's predictions $\hat{t}_i$ are supervised with:

Ltime=1N∑i=1N∣t^i−ti∣\mathcal{L}_{\text{time}} = \frac{1}{N} \sum_{i=1}^{N} \left| \hat{t}_i - t_i \right|

What it computes: The mean absolute error between predicted and ground-truth normalized timestamps across all $N$ frames. This is a standard L1 regression loss.

Why L1 over L2: L1 loss is more robust to outliers than L2, which matters if some frames are inherently ambiguous in their temporal position. If two frames look nearly identical, a small timestamp prediction error wouldn't indicate a systematic failure, and L1 penalizes this less harshly than L2.

Static Regularization

To encourage condition C1 (static regions collapse to degenerate trajectories), the paper penalizes the variance of control points for pixels in static regions:

Lstatic=1∣Ωstatic∣∑(i,u,v)∈ΩstaticVar({Pi,u,v(k)}k=0D−1)\mathcal{L}_{\text{static}} = \frac{1}{|\Omega_{\text{static}}|} \sum_{(i,u,v) \in \Omega_{\text{static}}} \text{Var}\left( \left\{ \mathbf{P}^{(k)}_{i,u,v} \right\}_{k=0}^{D-1} \right)

where $\Omega_{\text{static}}$ is the set of pixels belonging to static scene regions (known from synthetic data annotations), and the variance is computed over the $D$ control points of each pixel.

What it computes: For a static pixel, the variance across its $D$ control points measures how much the predicted trajectory deviates from a constant position. If all control points are identical, the variance is zero and the loss is zero. If the model predicts control points that spread out along a curve, the variance is large and penalized.

Why variance rather than pairwise distances: Computing the variance is computationally simpler than checking all pairwise distances between control points, and it directly captures the notion of "how degenerate is this trajectory." A trajectory that is perfectly static has zero variance; one that moves has positive variance. The variance can be computed as the mean squared deviation from the centroid of the control points.

Rigidity Regularization

For pixels segmented as belonging to the same rigid object, their relative distances should remain constant across control points because rigid bodies do not deform:

Lrigid=1∣Ωrigid∣∑(p,q)∈ΩrigidVar({∥Pp(k)−Pq(k)∥2}k=0D−1)\mathcal{L}_{\text{rigid}} = \frac{1}{|\Omega_{\text{rigid}}|} \sum_{(p,q) \in \Omega_{\text{rigid}}} \text{Var}\left( \left\{ \| \mathbf{P}^{(k)}_p - \mathbf{P}^{(k)}_q \|_2 \right\}_{k=0}^{D-1} \right)

where $\Omega_{\text{rigid}}$ is a set of pixel pairs $(p, q)$ belonging to the same rigid segment, and the variance is computed over the $D$ pairwise distances at each control point.

What it computes: For each pair of pixels on the same rigid object, this loss measures how much the distance between their predicted 3D positions varies across the $D$ control points. If the object is truly rigid, this distance should be identical at every control point (and therefore at every time $t$). The variance penalizes any temporal deformation—if the distance grows at some control points and shrinks at others, the variance is large.

Why this is necessary: Without rigidity regularization, the model might learn trajectories that satisfy the per-frame 3D position losses but produce physically implausible deformations—for example, a table stretching or compressing as the camera moves. This is a common failure mode in monocular 3D reconstruction where scale-depth ambiguity can cause objects to warp. The rigidity loss provides a physics-based prior that many real-world objects (furniture, vehicles, buildings) are rigid and should maintain constant internal distances. The paper uses segmentation masks from the synthetic data to identify which pixel pairs are rigid, so this is a privileged loss that relies on ground-truth annotations.

Correspondence Regularization

To encourage condition C2 (corresponding pixels from different frames map to the same trajectory), the paper penalizes discrepancies between control points of known corresponding pixels:

Lcorr=1∣Ωcorr∣∑((i,u,v),(j,u′,v′))∈Ωcorr1D∑k=0D−1∥Pi,u,v(k)−Pj,u′,v′(k)∥22\mathcal{L}_{\text{corr}} = \frac{1}{|\Omega_{\text{corr}}|} \sum_{((i,u,v), (j,u',v')) \in \Omega_{\text{corr}}} \frac{1}{D} \sum_{k=0}^{D-1} \left\| \mathbf{P}^{(k)}_{i,u,v} - \mathbf{P}^{(k)}_{j,u',v'} \right\|_2^2

where $\Omega_{\text{corr}}$ is a set of matched pixel pairs known to correspond to the same physical world point (from ground-truth annotations), and the inner sum is the mean squared error between the $k$-th control points of the two trajectories.

What it computes: For two pixels in different frames that are known to correspond (e.g., the same physical corner of a table seen from two viewpoints), this loss directly encourages their entire control point sequences to be identical. If both trajectories describe the same physical motion, they should have the same control points.

Why this matters: Condition C2—that corresponding pixels map to the same trajectory—is the property that makes trajectory fields unified rather than disjoint. Without this loss, the model could predict perfectly accurate trajectories for each pixel individually, but two different frames' pixels corresponding to the same world point might predict slightly different trajectories that diverge over time. The correspondence loss forces consistency: the trajectory representing a physical point must be the same regardless of which frame's pixel is used to access it. This is supervised using known correspondences from the synthetic dataset.

Final Objective

The overall training loss combines all terms with scalar weights:

L=Ltraj-conf+λtimeLtime+λstaticLstatic+λrigidLrigid+λcorrLcorr\mathcal{L} = \mathcal{L}_{\text{traj-conf}} + \lambda_{\text{time}} \mathcal{L}_{\text{time}} + \lambda_{\text{static}} \mathcal{L}_{\text{static}} + \lambda_{\text{rigid}} \mathcal{L}_{\text{rigid}} + \lambda_{\text{corr}} \mathcal{L}_{\text{corr}}

The paper does not explicitly state the values of the $\lambda$ hyperparameters, but they are presumably tuned to balance the relative magnitudes of the different loss terms so that no single term dominates training.

Training procedure details (from Section 5.1): The model is trained in two stages. In the first stage, it trains on 20K Kubric videos (24 frames each); in the second stage, it trains on a mixture of 20K Kubric videos and 10K videos from the Trace Anything data engine (120 frames each). All training uses images resized to 512 pixels on the longest side. The optimizer is AdamW with learning rate $1 \times 10^{-4}$ and a cosine annealing schedule. The batch size is 1, with each batch sampling up to 30 frames. Training takes 7.22 days on 32 NVIDIA A100 80GB GPUs. Memory efficiency is achieved through FlashAttention (Dao et al., 2023) and DeepSpeed ZeRO Stage 2 (Rajbhandari et al., 2020), which partitions optimizer states and gradients across GPUs.

Why two-stage training: Kubric provides a large volume of simpler scenes (rigid objects, textured backgrounds) at lower frame counts, which gives the model a foundation in basic 3D geometry and rigid motion. The Trace Anything engine adds non-rigid dynamics (articulated characters, deformable objects) and more complex environments, but its dataset is smaller (10K vs. 20K scenes). Training on the mixture in stage 2 allows the model to learn complex dynamics without forgetting the basic geometry learned from Kubric.

Why batch size 1 with up to 30 frames: The joint processing of all frames in a video requires that all frames be present in a single forward pass. Processing a 30-frame video at 512px resolution with a full transformer already consumes significant GPU memory. Batch size 1 is a practical necessity given the memory constraints, and gradient accumulation or ZeRO optimization compensates for the small batch size.

The B-Spline Parameterization Choice (Appendix B)

The paper's implementation uses cubic B-splines ($p = 3$) with clamped, non-uniform knot vectors to parameterize trajectories. This choice is not arbitrary—it reflects careful consideration of the geometric properties needed for trajectory modeling.

The B-spline basis functions $N_{k,3}(t)$ are computed via the Cox-de Boor recursion formula:

Nk,0(t)={1if tk≤t<tk+1 for k<n+p1if tk≤t≤tk+1 for k=n+p0otherwiseN_{k,0}(t) = \begin{cases} 1 & \text{if } t_k \leq t < t_{k+1} \text{ for } k < n + p \\ 1 & \text{if } t_k \leq t \leq t_{k+1} \text{ for } k = n + p \\ 0 & \text{otherwise} \end{cases}

Nk,p(t)=t−tktk+p−tkNk,p−1(t)+tk+p+1−ttk+p+1−tk+1Nk+1,p−1(t)N_{k,p}(t) = \frac{t - t_k}{t_{k+p} - t_k} N_{k,p-1}(t) + \frac{t_{k+p+1} - t}{t_{k+p+1} - t_{k+1}} N_{k+1,p-1}(t)

where $\mathbf{t} = [t_0, t_1, \dots, t_{m-1}]$ is the knot vector and $p$ is the degree (3 for cubic).

What this computes: The recursion builds smooth basis functions from piecewise-constant indicators ($p=0$). At each level $p$, each basis function $N_{k,p}$ is a blend of two lower-degree basis functions, weighted by normalized distances along the knot vector. The resulting cubic basis functions ($p=3$) are $C^2$ continuous within each knot span and have local support—each basis function is nonzero only over 4 consecutive knot intervals.

The knot vectors used (from Equation 24) have the property that:

  • At $t = 0$: $N_{0,3}(0) = 1$ and $N_{k,3}(0) = 0$ for all $k > 0$, so $\mathbf{x}_{i,u,v}(0) = \mathbf{P}^{(0)}_{i,u,v}$.
  • At $t = 1$: $N_{n-1,3}(1) = 1$ and $N_{k,3}(1) = 0$ for all $k < n-1$, so $\mathbf{x}_{i,u,v}(1) = \mathbf{P}^{(n-1)}_{i,u,v}$.

The internal knots with multiplicity up to 3 enforce $C^0$ continuity at segment boundaries, meaning the curve can have sharp corners at $t = 0.5$ (for $n=7$) or at $t = 1/3, 2/3$ (for $n=10$). This is useful for representing motions with sudden direction changes.

Why clamped end knots: Clamping (multiplicity $p+1$ at the endpoints) ensures that the curve interpolates its first and last control points exactly. Without clamping, the curve would only approximate these points, which would make it impossible for the model to precisely control where the trajectory begins and ends. For trajectory field estimation, the endpoint positions are critical—they represent where the point is at the start and end of the sequence—so exact interpolation is necessary.

Confidence interpolation: The paper notes that confidence values $\hat{\Sigma}^{(k)}_{i,u,v}$ are "interpolated alongside 3D coordinates using the same basis functions." This means the confidence at any query time $t_j$ is a B-spline-weighted sum of the per-control-point confidences, which makes sense because the influence of each control point on the final position varies with $t$, so the confidence in that position should similarly be an influence-weighted average of the confidences of the control points that contribute to it.

The Data Platform and Annotation Pipeline (Section 4)

The Trace Anything Data Platform is a Blender-based synthetic data generation system that produces photo-realistic dynamic scenes with comprehensive ground-truth annotations. It is the infrastructure that makes training possible.

What it generates: For each scene, the platform produces:

  • RGB video frames (120 frames per scene in the current release)
  • Per-pixel 2D trajectories (pixel coordinates over time in image space)
  • Per-pixel 3D trajectories (world coordinates over time, the primary supervision target)
  • Depth maps
  • Semantic segmentation masks (identifying objects, static regions, rigid segments)
  • Optical flow
  • Camera poses (extrinsics and intrinsics)

Scene diversity: The platform achieves diversity across three axes: (i) environments—indoor and outdoor backgrounds from public asset libraries and procedural generation (Infinigen); (ii) dynamics—articulated human characters and movable objects with both rigid and non-rigid motion; (iii) camera motion—smooth trajectories sampled around active regions to mimic natural filming behavior.

Why a custom platform rather than existing synthetic data: The paper identifies specific limitations of existing synthetic datasets: Kubric (Greff et al., 2022) is "largely limited to rigid dynamics and textured backgrounds," while other options (FlyingThings3D, PointOdyssey) have sparse or short-term annotations. The Trace Anything platform directly addresses these gaps by generating non-rigid character motion (articulated humans, animals) and providing dense per-pixel annotations for every frame, not just sparse query points. The platform is also extensible—"fully programmable" means new assets, environments, or annotation modalities can be added programmatically.

The Trace Anything Benchmark: A held-out set of 200 videos (each 120 frames) from the platform serves as the evaluation benchmark. The key difference from existing benchmarks is the evaluation protocol: every pixel in every frame must be evaluated (all-to-all), not just pixels sampled from the first frame (first-to-all). The benchmark also evaluates in world coordinates (not relative coordinates) and provides denser annotations than existing point tracking benchmarks.

4. Key Insights and Innovations

Innovation 1: Making Per-Pixel 3D Trajectories the Output Primitive Rather Than a Post-Hoc Derivation

The dominant paradigm in dynamic scene understanding treats 3D trajectories as a derived property—something you get by combining depth maps, optical flow, 2D tracks, and global optimization. The standard pipeline is: estimate depth per frame, estimate optical flow between frames, track 2D points, lift to 3D using the depth estimates, then run bundle adjustment or similar optimization to enforce cross-frame consistency. Each step uses a different model with its own error characteristics, and errors compound.

This paper's foundational move is to invert that logic. Instead of treating 3D trajectories as the composition of intermediate estimators, it makes trajectories the direct, atomic output of a single model. The trajectory field formulation (Equation 1) defines the output as a mapping from discrete pixel coordinates to continuous 3D functions—every pixel in every frame gets a complete parametric curve describing where it is in 3D space at any time. This might sound like a minor reframing, but it has cascading consequences for architecture design, supervision strategy, and evaluation.

What makes this non-trivial: The field has historically avoided making dense 3D trajectories the direct output for good reason: training such a model requires dense per-pixel 3D trajectory annotations, which don't exist for real videos. Prior work settled on sparse 2D tracking (TAP-Vid, CoTracker) or disjoint per-frame 3D reconstruction (DUSt3R, Monst3R, POMATO) precisely because those could be supervised with available data—2D point annotations or multi-view geometric constraints. The paper's synthetic data platform (Section 4) breaks this deadlock, but the intellectual move is recognizing that if you can get the data, making trajectories the output primitive eliminates an entire class of pipeline errors that come from composing independent estimators. The architecture reflects this commitment: the fusion transformer processes all frames jointly and outputs control points in a shared world coordinate system, meaning cross-frame consistency is built into the output format itself rather than enforced post-hoc.

This is a fundamental shift, not incremental. Prior work in dynamic reconstruction (Monst3R, POMATO, St4RTrack, Easi3R) extends DUSt3R-style networks to predict per-frame pointmaps, then adds flow-based correspondences. These methods fundamentally output disjoint per-frame representations and establish correspondence as a separate step. Trace Anything's output is inherently corresponding because the trajectory for a physical point is the same function regardless of which frame's pixel you use to access it—condition C2 is a property of the representation, not something that needs to be solved after prediction. The quantitative evidence that this matters is in the Correspondence Agreement (CA) metric (Table 1), where Trace Anything achieves CA of 5.09 (×10⁻²) compared to the next best of 5.15 (Easi3R), meaning trajectories predicted from different frames for the same physical point are more consistent without any explicit correspondence-solving step.

Innovation 2: Geometric Priors Encoded as Differentiable Regularization Terms on Control Points

Training a model to output dense 3D trajectories from RGB pixels is severely underconstrained. A given video is physically consistent with infinitely many possible 3D interpretations—scale-depth ambiguity means a small object close to the camera and a large object far away can produce identical pixel patterns, and monocular dynamic scenes add the complication that apparent motion could be due to object movement, camera movement, or both. The standard response in 3D vision is to use multi-view geometric constraints (epipolar geometry, triangulation) to disambiguate. But those constraints assume static scenes, precisely the assumption this paper relaxes.

The paper's response is to inject geometric knowledge about the physical world directly into the training loss through carefully designed regularization terms on the predicted control points (Equations 11–13). This is not standard data augmentation or architectural inductive bias—it's a form of physics-informed learning where the regularizers encode specific properties that valid 3D trajectories must satisfy, and the model internalizes these properties through gradient-based optimization.

Static regularization (C1): The variance-penalty on control points for static-region pixels pushes the model toward degenerate (constant-position) trajectories for background pixels. This is more structured than simply adding a small-motion prior everywhere—it targets specific semantic regions (known from synthetic data annotations). The intuition is that if a wall pixel in frame 1 moves in 3D between frames, that's always wrong because walls don't move, and the model should receive an explicit penalty for predicting such motion.

Rigidity regularization: The requirement that pairwise distances between pixels on the same rigid object remain constant across control points is a powerful constraint because it connects the trajectories of different pixels. If the model predicts that pixel A on a table corner moves along some curve, it must also predict that pixel B on the same table corner moves along a curve that maintains a fixed distance from A's curve at every control point. This couples the predictions of different pixels and prevents independent drifting that would produce physically impossible deformations. Without this, a model could satisfy the per-pixel position losses while a rigid object silently stretches or compresses.

Correspondence regularization (C2): Directly penalizing discrepancy between control points of known corresponding pixels forces the model to learn that the same physical point should have the same trajectory regardless of which frame it's viewed from. This is the loss that makes the trajectory field unified—it's what prevents the model from predicting one trajectory for a point from frame 1 and a slightly different trajectory for the same point from frame 10. The fact that the model generalizes this to unknown correspondences (as measured by the CA metric on test data) shows that the regularizer successfully teaches the underlying geometric principle, not just memorized specific pixel pairs.

Why this is distinctive: Most learning-based 3D methods rely on reconstruction losses—making the predicted 3D match the ground truth—and implicitly hope the model learns the underlying geometry. The explicit regularization terms here go beyond that by encoding what cannot happen (static things shouldn't move, rigid things shouldn't deform, corresponding things should agree) as differentiable penalties. This is closer in spirit to constraint-based optimization in classical geometry (bundle adjustment enforces that corresponding points project to the same image locations) but implemented in a fully differentiable form amenable to gradient descent. The ablation in Table C shows the effect: without the full regularization suite, the model would need to learn these constraints purely from reconstruction error, which is statistically harder because the reconstruction loss provides only weak signal about cross-pixel or cross-frame consistency.

The SDD and CA metrics introduced in Section 5.3 (static degeneracy deviation and correspondence agreement) are themselves an innovation: they operationalize these geometric consistency conditions into quantitative metrics, providing a language for evaluating whether a trajectory field is "well-formed" beyond just positional accuracy. Prior work either didn't evaluate these properties or baked them into pipeline components (e.g., RANSAC for outlier rejection in SfM) without measuring them as outputs.

Innovation 3: Feed-Forward Global Processing Breaks the Accuracy-Efficiency Tradeoff That Defined Prior Dynamic Reconstruction

The paper demonstrates something that challenges a widely held assumption in 3D vision: that accurate dynamic 3D reconstruction requires expensive per-scene optimization. Prior to this work, the landscape was split between two unsatisfactory regimes:

  • Optimization-based methods (OmniMotion, MegaSAM) achieve high accuracy by solving a per-scene optimization problem—fitting 3D geometry and motion parameters to each video individually. These methods can handle complex scenes but take minutes to hours per video, making them unusable for any online application.

  • Feed-forward methods (Monst3R, POMATO, St4RTrack) provide faster inference by learning to predict geometry in a single forward pass, but they either assume static scenes or produce disjoint per-frame outputs that still require some form of global alignment. Even the fastest prior methods (St4RTrack at 22.5 seconds, Table 1) are an order of magnitude slower than Trace Anything.

Trace Anything achieves 2.3 seconds on video-based inference (Table 1) while simultaneously achieving the best accuracy across all metrics (EPEmix 0.234 vs. 0.272 for the next-best POMATO, SDD 1.06 vs. 1.44, CA 5.09 vs. 5.15). This breaks the tradeoff: it's both the most accurate and the fastest method by a wide margin on the authors' benchmark.

Why this is surprising: Joint processing of all frames should be more expensive than pairwise or sequential processing. The fusion transformer performs all-to-all attention across all frame tokens, which scales quadratically with total token count. Yet the total runtime is dramatically lower than methods that process frames in pairs (DUSt3R-style) and then run global optimization. The key is that the global optimization cost dominates for prior methods—the per-scene alignment step that reconciles pairwise predictions into a coherent global frame is the bottleneck, not the per-pair inference. By predicting all trajectories in a shared coordinate system directly from joint features, Trace Anything eliminates this post-hoc alignment entirely. The fusion transformer learns to coordinate the output spaces of all frames during a single forward pass, amortizing the "optimization" cost into the learned weights of the network.

The stage-wise runtime breakdown in Figure 9 provides insight: the fusion transformer is the most expensive single stage, but the curve evaluation (converting control points to 3D coordinates) is nearly free, and the image encoder cost grows linearly with frame count. The absence of any iterative refinement loop or pairwise processing means the total cost scales approximately linearly with the number of frames, whereas pairwise methods scale quadratically and optimization-based methods scale with the number of optimization iterations.

This is a fundamental advance for practical deployment. It shifts the question from "can we afford to reconstruct this dynamic scene in 3D?" (which previously required trading off accuracy against speed) to "can we train a model that does this in one shot?"—where the training cost is high but inference is cheap. The paper doesn't frame it this way, but this is essentially the same dynamic that made feed-forward depth estimation (MiDaS, DPT) practical: expensive training on synthetic data enables cheap, accurate inference that replaces slow optimization-based methods (classical stereo, SfM) for many applications.

Caveat on generalization: The accuracy advantage is demonstrated on the Trace Anything Benchmark, which uses synthetic data from the same distribution as the training data. The paper's out-of-distribution experiment on PointOdyssey (Appendix Table A) shows Trace Anything maintains competitive accuracy (EPEmix 0.256 vs. 0.269 for St4RTrack), but the margin narrows substantially compared to the in-distribution results. The efficiency advantage (2.3 seconds vs. 19.9 seconds) is robust, but the claim that feed-forward processing always matches or beats optimization-based accuracy requires broader validation on real-world benchmarks where the domain gap is larger.

Innovation 4: The Trajectory Field as a Unified Substrate That Naturally Yields Multiple Downstream Capabilities

A subtle but powerful property of the trajectory field representation is that it serves as a unified geometric substrate from which multiple downstream outputs can be derived through simple geometric operations, without training separate models for each. The paper demonstrates this in Section 5.4 and Appendix C.1, but the conceptual significance is broader than the specific capabilities shown.

2D trajectories are obtained by projecting the 3D trajectory onto the image plane using (estimated) camera parameters—a purely geometric operation that requires no additional learned model. This means Trace Anything can serve as a dense 2D point tracker without being trained for 2D tracking specifically.

Dynamic masks are obtained by thresholding the variance of control points per pixel—if all control points are similar (low variance), the pixel is static; if they differ (high variance), the pixel is dynamic. This static/dynamic segmentation emerges from the trajectory prediction without being explicitly trained as a segmentation task.

Scene flow (the 3D displacement field between two frames) is simply the difference between trajectory endpoints evaluated at the two frames' timestamps—again, a geometric computation.

Velocity-based forecasting (Figure 10) is achieved by extrapolating the trajectory beyond t = 1 using tangent continuation—finding the instantaneous velocity from the last segment of the spline and extending linearly. The paper doesn't train a separate forecasting model; it leverages the continuity of the B-spline representation.

Spatio-temporal fusion (Figure 11) uses the trajectory field to warp observations of a dynamic entity from multiple frames into a common canonical frame, effectively undoing the motion and aggregating partial observations. This is the inverse operation of evaluating the trajectory at different times—it maps different frames' views of the same physical point to a shared reference.

What makes this distinctive: In the standard pipeline paradigm, each of these capabilities would require a separate model or at least a separate training objective: a 2D tracker for trajectories, a motion segmentation network for dynamic masks, a scene flow network for 3D displacements, a forecasting model for prediction. The trajectory field collapses these into a single representation and a single model. This is not just a matter of engineering convenience—it means these capabilities are guaranteed to be mutually consistent because they derive from the same underlying geometric prediction. The 2D trajectories are consistent with the 3D trajectories, which are consistent with the scene flow, which is consistent with the dynamic masks, because they all come from the same control points. In a pipelined system, the 2D tracker, the 3D reconstructor, and the motion segmenter might disagree, and resolving those disagreements would require yet another optimization problem.

This property—that a representation naturally yields multiple downstream outputs through deterministic, non-learned operations—is reminiscent of how NeRFs enable novel view synthesis, depth estimation, and 3D reconstruction all from the same radiance field, without training separate decoders. The trajectory field extends this principle to the temporal domain: it's a representation from which spatial queries, temporal queries, and spatio-temporal queries can all be answered through evaluation of the same underlying parametric curves. The fact that these emergent capabilities are not explicitly trained for (the model was never given forecasting or fusion tasks during training) but arise from the structure of the representation itself suggests the trajectory field captures something fundamental about scene dynamics.

Limitation of the emergent capabilities claim: The paper provides only qualitative demonstrations (Figures 10–12) for these capabilities, not quantitative evaluations. Velocity-based forecasting through tangent continuation assumes constant velocity beyond the observed time window, which is a strong and often incorrect assumption. The instruction-based forecasting (Figure 12) relies on a separate video generation model (Seedance 1.0) and only uses Trace Anything to lift the generated 2D frames into 3D trajectories, so it is not an "emergent" capability of Trace Anything itself but rather a demonstration of composability with generative models. The spatio-temporal fusion demonstration is compelling but not benchmarked against dedicated fusion methods. These capabilities are plausible consequences of the representation choice, but the paper's evidence for them is suggestive rather than conclusive.

Innovation 5: The All-to-All Evaluation Protocol — Redefining What It Means to "Understand" a Dynamic Scene

The paper's benchmark introduces an evaluation protocol that is qualitatively harder than established point tracking benchmarks and that, by its very design, reveals whether a model has genuinely learned joint spatio-temporal reasoning or is just good at forward propagation. This is a conceptual contribution to how we measure progress in dynamic scene understanding, not just a new dataset.

The key distinction: Existing point tracking benchmarks (TAP-Vid, TAPVid-3D) use a first-to-all protocol: query points are sampled from the first frame, and the model must track them forward through the sequence. This tests a model's ability to maintain identity through occlusions and appearance changes, but it is inherently a forward-propagation task. A model could succeed by learning to follow motion patterns forward in time without ever reasoning about what came before the first frame or how different frames' trajectories relate to each other.

The Trace Anything Benchmark uses an all-to-all protocol: every pixel in every frame must have a complete trajectory spanning the entire sequence. For a pixel in frame 10, the model must predict where that point was in frames 1–9 (backward in time) and where it will be in frames 11–N (forward in time). This tests whether the model has learned a globally consistent 4D model of the scene—one where trajectories are full curves, not just forward extrapolations from a starting point.

Why this distinction matters: A model that processes frames sequentially and propagates tracks forward (which is how most tracking methods operate) can succeed at first-to-all evaluation by being good at forward tracking. It would fail at all-to-all evaluation because backward tracking is not part of its operational capability. To succeed at all-to-all, a model must either (a) process all frames jointly and reason about the entire sequence simultaneously—which is what Trace Anything does with its fusion transformer—or (b) run forward tracking from every frame and then reconcile the results, which would be expensive and prone to inconsistency. The all-to-all protocol thus implicitly tests for a specific architectural property: joint spatio-temporal processing.

The paper's two new metrics operationalize this distinction:

  • Static Degeneracy Deviation (SDD) measures whether static regions produce truly constant trajectories (condition C1). A model that predicts slightly different positions for the same static point across different frames would score well on per-frame position accuracy but poorly on SDD, revealing that it hasn't learned the static/dynamic distinction.

  • Correspondence Agreement (CA) measures whether corresponding pixels from different frames map to the same trajectory (condition C2). A model that tracks well forward from the first frame but produces inconsistent trajectories when queried from different starting frames would have low CA, revealing inconsistency that per-frame metrics would miss.

These metrics don't just add evaluation detail—they diagnose specific failure modes of dynamic scene understanding models. A model's EPE (end-point error) can be low while its SDD is high (it gets positions right but assigns non-zero motion to static regions), or its EPE and SDD can both be low while its CA is high (it produces accurate but inconsistent trajectories). The metrics decompose "good performance" into geometrically meaningful components, which is a form of diagnostic evaluation that the field needs as models become more complex and their failures more subtle.

The fact that prior methods were not designed for all-to-all evaluation and achieve varying degrees of success on it (Table 1 shows CA values from 5.15 for Easi3R to 9.37 for St4RTrack) suggests they were implicitly relying on forward-propagation heuristics that break when asked to produce full trajectories. Trace Anything's lead on CA (5.09) and SDD (1.06) demonstrates that its joint-processing architecture and regularization losses directly address the consistency properties that the all-to-all protocol demands. This is not a small engineering win—it shows that the evaluation protocol and the model architecture were co-designed around a shared conception of what dynamic scene understanding should mean.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary evaluation is conducted on the Trace Anything Benchmark introduced in Section 4, comprising 200 synthetic videos (120 frames each) generated by the authors' Blender-based data platform. These videos feature diverse environments, articulated character motion, and smooth camera trajectories, with dense per-pixel 3D trajectory ground truth, depth, semantics, flow, and camera poses. For out-of-distribution evaluation, the authors also construct a benchmark from PointOdyssey (Zheng et al., 2023), consisting of 50 videos of 30 frames each, on which the model has never been trained or fine-tuned. For 3D tracking evaluation, subsets of TAPVid-3D (Koppula et al., 2024)—ADT, DriveTrack, and PStudio—are used, with 50 videos of 60 frames each sampled per subset using every other frame as input.

  • Base model(s). The Trace Anything model uses an image encoder and fusion transformer initialized with Fast3R (Yang et al., 2025) pretrained weights, while the control point heads are randomly initialized. The released model adopts B-splines with 10 control points as the parametric curve representation (Appendix C, Table C). For the FLOPs-matched comparison or pretraining-vs-inference tradeoff analysis, no second model scale is evaluated—unlike the reference example, this paper does not compare against a larger pretrained model. The architecture is single-scale throughout all experiments.

  • Metrics. The paper defines two families of metrics (Section 5.3). End-Point Error (EPE) is measured in three variants: EPEmix (average 3D Euclidean distance between predicted and ground-truth positions over all points), EPEsta (over static points only), and EPEdyn (over dynamic points only)—all reported in unspecified units but interpreted as world-coordinate distances. Static Degeneracy Deviation (SDD) quantifies temporal jitter of trajectories in static regions by measuring how much the predicted trajectory deviates from a constant position; lower values indicate better compliance with condition C1 (static regions should collapse to degenerate trajectories). Correspondence Agreement (CA) measures how consistently dynamic trajectories are predicted from corresponding pixels in different source frames; lower values indicate better compliance with condition C2 (corresponding pixels should map to the same trajectory). For 3D tracking on TAPVid-3D (Appendix C.3), the paper uses standard metrics: APD3D (average percent of points within a spatial threshold) and AJ (average Jaccard, capturing both spatial accuracy and occlusion correctness). Runtime is measured in seconds on a single NVIDIA A100 GPU.

  • Baselines. For video-based inference (Table 1), the paper compares against seven methods: CoTracker3 + VGGT (Karaev et al., 2024; Wang et al., 2025)—CoTracker3 for 2D tracking lifted to 3D using VGGT's depth and camera estimates; DELTA (Ngo et al., 2025)—a dense 3D tracker using transformers with upsampling; SpaTrackerV2 (Xiao et al., 2025)—a feed-forward 3D tracker combining 2D tracking with monocular depth; MonsT3R (Zhang et al., 2025)—a DUSt3R-style dynamic reconstruction method; St4RTrack (Feng et al., 2025)—simultaneous 4D reconstruction and tracking; POMATO (Zhang et al., 2025)—pointmap matching with temporal motion for dynamic reconstruction; and Easi3R (Chen et al., 2025)—estimating disentangled motion from DUSt3R without training. For image-pair inference (Table 2), baselines include SEA-RAFT + VGGT (Wang et al., 2024; Wang et al., 2025)—optical flow lifted to 3D; RAFT-3D (Teed & Deng, 2021)—scene flow estimation; MASt3R (Leroy et al., 2024)—grounded image matching in 3D; plus MonST3R, St4RTrack, POMATO, and Easi3R as above.

  • Generation budget / compute accounting. The paper measures test-time compute through runtime in seconds on a single A100 GPU (Tables 1 and 2), not through a unit like "generations" or FLOP counts. All methods are evaluated with the same input (30-frame video clips for video-based inference; image pairs for image-pair inference), and runtime is reported as the total wall-clock time to produce predictions. The paper also provides a stage-wise runtime breakdown in Figure 9, showing the fusion transformer as the most expensive stage, followed by image encoding and curve evaluation, with total runtime scaling approximately linearly with the number of frames. There is no FLOPs-matched comparison against larger pretrained models, so no pretraining-vs-inference compute tradeoff is analyzed (unlike the reference example in Section 7 of that paper).

  • Cross-validation / statistical protocol. The paper does not describe cross-validation, statistical significance testing, or confidence intervals for the main quantitative results. The Trace Anything Benchmark uses a fixed split of 200 held-out videos. For the 3D tracking evaluation on TAPVid-3D, a fixed sample of 50 videos per subset is used. The ablation study (Appendix C.4, Table C) reports metric values without error bars or statistical tests. The paper's difficulty estimation or adaptive allocation mechanism is not analogous to the reference example—Trace Anything does not use difficulty-conditioned strategy selection, so no cross-validation over difficulty bins is described.

Main Quantitative Results

Video-Based Trajectory Field Estimation (Table 1)

The headline result for video-based inference on the Trace Anything Benchmark is that Trace Anything achieves the best performance across all five metrics while running substantially faster than any competitor. On 30-frame video clips:

  • EPEmix (all points): Trace Anything achieves 0.234, compared to the next-best POMATO at 0.272 and St4RTrack at 0.278. This represents roughly a 14% relative reduction in average end-point error over the strongest baseline.
  • EPEsta (static points): Trace Anything achieves 0.218 vs. St4RTrack at 0.247 and POMATO at 0.254, showing the largest relative advantage on static regions.
  • EPEdyn (dynamic points): Trace Anything achieves 0.295 vs. POMATO at 0.308 and Easi3R at 0.324, demonstrating that the advantage persists on the harder dynamic-region evaluation.
  • Correspondence Agreement (CA): Trace Anything achieves 5.09 (×10⁻²) vs. Easi3R at 5.15 and DELTA at 6.24. The narrow margin over Easi3R (the only other method to break 6.0) indicates that cross-frame consistency is the metric where the gap is smallest, though Trace Anything still leads.
  • Static Degeneracy Deviation (SDD): Trace Anything achieves 1.06 (×10⁻³) vs. POMATO at 1.44 and SpaTrackerV2 at 1.51—a roughly 26% relative improvement, indicating substantially less temporal jitter in static regions.
  • Runtime: Trace Anything completes inference in 2.3 seconds, compared to St4RTrack at 22.5 seconds (~10× slower), POMATO at 81.8 seconds (~36× slower), MonsT3R at 99.1 seconds, Easi3R at 130.9 seconds, SpaTrackerV2 at 178.4 seconds, CoTracker3+VGGT at 197.4 seconds, and DELTA at 231.6 seconds (~100× slower). The speed advantage is most dramatic relative to CoTracker3+VGGT (86×) and DELTA (101×).

Several baselines show interesting patterns. CoTracker3+VGGT achieves the worst CA (7.83) and relatively high EPEmix (0.518), suggesting that composing a 2D tracker with a 3D reconstruction model introduces substantial inconsistency. St4RTrack, despite being the fastest baseline (22.5 seconds), achieves competitive EPEsta (0.247) but relatively poor CA (9.37), indicating that its cross-frame correspondences—established through joint optimization—are less consistent than methods that learn them directly. Easi3R, which requires no training for its dynamic extension, achieves the best CA among baselines (5.15) but trails on EPE metrics (EPEmix 0.308), suggesting its strength is in consistency rather than raw positional accuracy.

The runtime comparison is particularly significant because it represents a qualitative regime change: Trace Anything operates at a speed (2.3 seconds for 30 frames) that could plausibly support interactive applications, whereas all baselines except St4RTrack require over a minute, making them unsuitable for any online or near-real-time use case. However, it is critical to note that the accuracy advantage is demonstrated exclusively on the authors' own synthetic benchmark, where the training distribution matches the test distribution.

Image-Pair-Based Trajectory Field Estimation (Table 2)

For the image-pair setting (two frames sampled 5 frames apart), Trace Anything again achieves the best overall performance, though the pattern of advantages differs from the video setting:

  • EPEmix: Trace Anything achieves 0.135, compared to POMATO at 0.181 and MonST3R at 0.206. The gap to the next-best method (26% relative reduction) is larger than in the video setting, suggesting that Trace Anything's joint representation is particularly advantageous when fewer frames are available.
  • EPEsta: Trace Anything achieves 0.106 vs. POMATO at 0.137 and MASt3R at 0.181. The static-region advantage is substantial.
  • EPEdyn: Here, the results are more nuanced. Trace Anything achieves 0.304, which is competitive but not clearly dominant—RAFT-3D achieves 0.324, St4RTrack achieves 0.325, and Easi3R achieves 0.323. POMATO (0.320) and SEA-RAFT+VGGT (0.427) are notably worse on dynamic regions. This suggests that on the specifically dynamic part of the error, multiple methods perform similarly, and Trace Anything's overall EPEmix advantage comes largely from superior static-region performance.
  • CA: Trace Anything achieves 12.41 (×10⁻²), substantially better than all baselines: St4RTrack (15.33), RAFT-3D (17.50), SEA-RAFT+VGGT (18.22), POMATO (19.58), MonST3R (20.10), Easi3R (20.41), and MASt3R (33.99). This is the metric where Trace Anything's advantage is most pronounced relative to specific baselines—for example, it achieves nearly 3× better CA than MASt3R.
  • SDD: Trace Anything achieves 0.54 (×10⁻³) vs. St4RTrack at 0.63 and SEA-RAFT+VGGT at 0.77. The improvement over the best baseline is roughly 14%.
  • Runtime: Trace Anything completes inference in 0.20 seconds, compared to RAFT-3D at 0.37 seconds, St4RTrack at 1.39 seconds, SEA-RAFT+VGGT at 1.91 seconds, MASt3R at 2.39 seconds, MonST3R at 2.51 seconds, POMATO at 4.75 seconds, and Easi3R at 5.08 seconds. While the absolute times are all small, Trace Anything is roughly 2× faster than RAFT-3D (the fastest baseline) and 25× faster than Easi3R (the slowest).

The image-pair results reveal an important asymmetry: Trace Anything's EPEdyn (0.304) is comparable to several baselines, but its CA (12.41) and EPEsta (0.106) are substantially better. This suggests that on dynamic regions, the model is not necessarily more accurate in absolute 3D position, but its predictions are more consistent across frames—the same physical point gets more similar trajectory predictions when queried from different source frames. For applications where correspondence consistency matters more than absolute positional accuracy (e.g., tracking, fusion), this is a valuable property.

The SEA-RAFT+VGGT baseline (optical flow + 3D lifting) performs notably poorly on EPEdyn (0.427) and CA (18.22), confirming the paper's thesis that chaining independent estimators introduces compounding errors that a joint model can avoid.

Out-of-Distribution Evaluation (Appendix C.3, Table A)

On the PointOdyssey benchmark—50 videos of 30 frames each, with no training or fine-tuning on this data—Trace Anything maintains competitive performance but with substantially narrower margins:

  • EPEmix: Trace Anything achieves 0.256 vs. St4RTrack at 0.269 and POMATO at 0.344. The advantage over St4RTrack (the best baseline) is only 4.8%, compared to the 14% advantage on the in-distribution benchmark.
  • EPEsta: Trace Anything achieves 0.212 vs. St4RTrack at 0.243 (12.8% relative improvement).
  • EPEdyn: Trace Anything achieves 0.319 vs. St4RTrack at 0.325—essentially tied.
  • CA: Trace Anything achieves 6.19 (×10⁻²) vs. POMATO at 6.24—again effectively tied.
  • SDD: Trace Anything achieves 1.37 (×10⁻³) vs. St4RTrack at 1.70 (19.4% relative improvement).
  • Runtime: Trace Anything at 2.3 seconds vs. St4RTrack at 19.9 seconds and Easi3R at 125.1 seconds.

The key finding here is not that Trace Anything dominates on out-of-distribution data—it does not—but that its advantage shrinks dramatically when evaluated on a domain not represented in training. On static-region accuracy (EPEsta) and static consistency (SDD), the model retains a meaningful advantage, likely because static geometry—walls, floors, background structures—is relatively universal across synthetic datasets. On dynamic-region accuracy (EPEdyn) and correspondence consistency (CA), the model is essentially tied with the best baselines, suggesting that the complex non-rigid motion patterns in PointOdyssey differ enough from the Trace Anything training distribution that the model's learned priors do not transfer well.

Notably, St4RTrack emerges as the strongest baseline on this benchmark, outperforming POMATO and Easi3R despite being weaker than them on the in-distribution Trace Anything Benchmark. This reversal highlights that St4RTrack's optimization-based approach (joint 4D reconstruction and tracking) may be more robust to domain shift than the purely feed-forward methods, even if it is less accurate when training and test distributions match.

3D Tracking on TAPVid-3D (Appendix C.3, Table B)

Despite not being designed or trained specifically for 3D point tracking, Trace Anything achieves competitive results on the TAPVid-3D benchmark:

  • ADT subset: Trace Anything achieves the best APD3D (20.5) vs. SpaTracker at 18.3 and POMATO at 18.2, and second-best AJ (15.6) vs. SpaTracker at 17.4. Runtime: 2.1 seconds vs. 191.1 seconds for SpaTracker.
  • DriveTrack subset: Trace Anything achieves APD3D of 15.5 (SpaTracker: 16.0, POMATO: 11.3) and AJ of 9.6 (SpaTracker: 10.1). The model trails SpaTracker, which was specifically designed and trained for this task, by small margins.
  • PStudio subset: Trace Anything achieves APD3D of 16.3 (tied with SpaTracker at 16.2 in second place) and the best AJ of 10.8 (vs. SpaTracker at 10.3).

The critical context for these numbers is that SpaTracker is a specialized 3D point tracker that was trained for the exact task evaluated, while Trace Anything is a general trajectory field estimator never fine-tuned on TAPVid-3D. Trace Anything matches or exceeds SpaTracker on some metrics while running ~90× faster (2.1 seconds vs. 191.1 seconds). This is a strong result for the trajectory field paradigm: the dense, per-pixel prediction of Trace Anything means it can track any point without being limited to a fixed set of query points per forward pass, which is a fundamental limitation of SpaTracker (which requires multiple forward passes to track more than a fixed maximum number of points).

However, the margins are small and the absolute numbers are relatively low (APD3D of 15–20 on a 0–100 scale), reflecting that 3D tracking on real-world benchmarks with significant occlusions and appearance changes remains a challenging problem for all methods, including those trained on synthetic data.

Ablation Studies and Robustness Checks

The paper presents a single ablation table (Appendix C.4, Table C) examining two axes: choice of geometric backbone initialization and choice of parametric curve type/number of control points.

  • Backbone initialization matters substantially. Training from scratch (Fast3R architecture but random initialization, labeled "None" in Table C) yields EPEmix of 0.472, compared to 0.234 with Fast3R pretrained weights—approximately 2× worse. The paper notes this configuration "struggles to converge," suggesting that the pretrained weights provide essential geometric reasoning capabilities (likely multi-view correspondence and basic 3D structure understanding) that are difficult to learn from trajectory field supervision alone. This is expected given that the pretrained Fast3R was trained on large-scale static 3D reconstruction data and provides a strong initialization for geometric feature extraction.

  • VGGT backbone achieves mixed results. Initializing with VGGT (Wang et al., 2025) yields EPEmix of 0.236, essentially tied with Fast3R's 0.234, and slightly better EPEdyn (0.276 vs. 0.295). However, runtime increases substantially to 7.2 seconds vs. 2.3 seconds for Fast3R—roughly 3× slower. The paper suggests VGGT "can be beneficial in settings that demand fine structural detail or involve large-baseline scenarios," but the default configuration uses Fast3R for its superior speed-accuracy tradeoff. This is a pragmatic engineering choice rather than a scientific finding about trajectory fields per se.

  • Polynomial curves underperform control-point-based parameterizations. A cubic polynomial (degree 3, fitting the trajectory directly as a polynomial in t) achieves EPEmix of 0.619—roughly 2.6× worse than B-spline with 10 control points (0.234). The paper attributes this to polynomial coefficients lacking "the clear geometric and physical interpretability" of control points. This is a non-obvious and informative negative result: simply parameterizing a smooth 3D curve is not sufficient—the parameterization must align with how the network learns to output geometry. Control points have a direct spatial interpretation (they are 3D points along or near the curve), which likely makes them easier for the network to predict from visual features than abstract polynomial coefficients.

  • More control points generally improve accuracy, with diminishing returns. For B-splines: 4 control points → EPEmix 0.281, 7 control points → 0.237, 10 control points → 0.234. The jump from 4 to 7 is substantial (15.7% relative improvement), while 7 to 10 is marginal (1.3%). For Bézier curves: 4 control points → 0.299, 10 control points → 0.238—a 20.4% relative improvement. However, Bézier curves with 10 control points achieve better EPEdyn (0.319) than B-spline with 10 control points (0.295) at the cost of slightly worse EPEmix and CA. This suggests that B-splines' local control property helps with overall accuracy and consistency, while Bézier's global influence of each control point may provide more flexibility for complex dynamic motions at the cost of some consistency.

  • Runtime scales modestly with control points. 4 control points (for both Bézier and B-spline): 1.7 seconds. 7 control points: 2.1 seconds. 10 control points: 2.3–2.5 seconds. The cost is primarily from the increased output dimensionality (the CP head must predict more values per pixel) and the curve evaluation overhead. The scaling is sub-linear in the number of control points, suggesting the fusion transformer and image encoder dominate runtime.

  • The paper does not ablate the individual regularization terms. There is no ablation showing performance with/without static regularization, rigidity regularization, or correspondence regularization. This is a significant gap: without these ablations, it is impossible to determine whether the regularization losses are actually contributing to the model's performance or whether the trajectory field could be learned adequately from the core trajectory loss alone. The paper introduces these losses as important components (Section 3.3) and includes λ hyperparameters in Equation (14), but provides no evidence that they matter.

  • The paper does not ablate the confidence adjustment mechanism. There is no comparison of the confidence-adjusted loss (Equation 9) versus a standard unweighted L2 loss. The contribution of the α log Σ̂ regularizer and the Σ̂ · ℓ weighting are not isolated. Given that this is a non-trivial component of the training scheme, the lack of ablation is a notable omission.

  • The paper does not ablate the fusion transformer design. There is no comparison of the interleaved frame-wise/global attention pattern against pure global attention or against sequential processing. The choice to use joint attention over all frames is central to the paper's claim of avoiding pairwise inference and global alignment, but the architectural necessity of this choice is not empirically validated.

  • Robustness to training data scale is not studied. The paper trains on 20K Kubric videos in stage 1 plus 10K from the custom platform in stage 2, but does not vary dataset size to determine how performance scales with data quantity. Given that the custom data platform is presented as a contribution, understanding the marginal value of additional training data would be informative.

  • No ablation on image resolution or frame count sensitivity. All experiments use 512px on the longest side. The model's sensitivity to input resolution and to the number of input frames is not systematically studied, though Figure 9 provides a runtime-vs-frames analysis without corresponding accuracy measurements.

Critical Assessment

The experimental results provide a nuanced picture of Trace Anything's strengths and limitations. The central claims of the paper and the evidence supporting each are evaluated below.

Claim: Trace Anything achieves state-of-the-art performance on trajectory field estimation.

What the experiments actually show: On the authors' own Trace Anything Benchmark, which uses synthetic data from the same distribution as the training data, Trace Anything achieves the best performance across all metrics (Table 1: EPEmix 0.234 vs. 0.272 for the next-best method; Table 2: EPEmix 0.135 vs. 0.181). This is an in-distribution evaluation, and the model was trained on data generated by the same platform that produced the benchmark. The paper does not evaluate on any real-world dataset for the primary trajectory field estimation task. On the out-of-distribution PointOdyssey benchmark (Table A), the advantage shrinks substantially: EPEmix 0.256 vs. 0.269 for St4RTrack—a 4.8% relative improvement, well within the range that could be explained by model variance or benchmark-specific tuning artifacts. On dynamic regions specifically (EPEdyn), Trace Anything is essentially tied with St4RTrack (0.319 vs. 0.325).

Assessment: The claim of state-of-the-art performance is supported only for the in-distribution synthetic setting. The paper does not demonstrate that Trace Anything's accuracy advantage generalizes to real videos, where the domain gap from synthetic training data is a well-known challenge in 3D vision. This is a significant limitation because the paper's framing—"Trace Anything: Representing Any Video in 4D"—implies general applicability to real videos, yet the quantitative evidence for accuracy superiority is restricted to the synthetic domain where training and test data share the same rendering engine, asset distributions, and motion patterns. The qualitative results on DAVIS (Figure 5) and BridgeData (Figure 6) are suggestive but lack quantitative metrics. A fairer claim would be that Trace Anything achieves state-of-the-art performance on synthetic trajectory field estimation and competitive performance on out-of-distribution synthetic data, with real-world generalization remaining to be demonstrated.

What would strengthen this claim: Quantitative evaluation on a real-world dataset with ground-truth 3D trajectories. The TAPVid-3D evaluation (Table B) partially addresses this, but TAPVid-3D provides sparse query-point ground truth, not dense per-pixel trajectories, making it a different task. A real-world benchmark with dense 3D annotations (which does not currently exist, to the paper's credit) would be the gold standard. Alternatively, evaluation on real videos with multi-view ground truth (e.g., from a calibrated multi-camera rig) could provide dense 3D supervision for dynamic scenes.

Claim: Trace Anything offers significant efficiency gains thanks to its one-pass paradigm.

What the experiments actually show: This claim is robustly supported. Trace Anything's runtime of 2.3 seconds for video-based inference (Table 1) is 10–100× faster than all baselines except St4RTrack (22.5 seconds). For image-pair inference (Table 2), Trace Anything runs in 0.20 seconds vs. 0.37 seconds for RAFT-3D (the fastest baseline) and 1.39–5.08 seconds for the remaining baselines. The efficiency advantage holds across both in-distribution and out-of-distribution settings (Table A: 2.3 seconds vs. 19.9–125.1 seconds), which is expected since runtime does not depend on the test distribution. The stage-wise runtime analysis (Figure 9) confirms that the total cost scales approximately linearly with frame count, avoiding the quadratic scaling of pairwise methods or the iteration-dependent cost of optimization-based approaches.

Assessment: Strongly supported. The efficiency claim does not depend on domain generalization—it is a property of the architecture and is equally valid for in-distribution and out-of-distribution data. The paper provides detailed runtime comparisons across multiple baselines and settings, and the speed advantage is large enough (10–100×) that minor variations in hardware or implementation details would not qualitatively change the conclusion. However, the comparison does not account for potential optimizations to the baseline methods—for example, CoTracker3+VGGT at 197.4 seconds may include VGGT's cost for full-scene reconstruction when only point tracking was needed. The paper uses each baseline "out of the box" for the trajectory field estimation task, which may disadvantage methods not designed for this specific evaluation protocol.

What would strengthen this claim: A FLOPs-based comparison rather than wall-clock time, which would control for implementation efficiency differences. Reporting throughput (frames per second) at varying batch sizes would clarify how the method scales for processing large video collections. Also, noting that all baselines were evaluated on a single A100 GPU—the same hardware as Trace Anything—provides a fair comparison, but memory usage comparisons would be informative since the fusion transformer's joint attention over all frames likely consumes more GPU memory than sequential or pairwise methods.

Claim: Trace Anything performs competitively on established point tracking benchmarks.

What the experiments actually show: On TAPVid-3D (Table B), Trace Anything achieves APD3D of 20.5/15.5/16.3 on ADT/DriveTrack/PStudio subsets, compared to the specialized SpaTracker at 18.3/16.0/16.2. On these sparse-query tracking metrics, Trace Anything is in the same performance tier as SpaTracker, outperforming it on some subsets and trailing on others, while running ~90× faster. The paper notes that it achieves these results "without task- or dataset-specific fine-tuning," which is a meaningful demonstration of the trajectory field's generality.

Assessment: Supported with the important caveat that "competitive" means "roughly tied with the best specialized method" rather than "clearly outperforming." The 3D tracking evaluation uses relatively small samples (50 videos per subset, sampled from TAPVid-3D), and the absolute performance numbers are low across all methods (APD3D of 6–20 out of 100), suggesting that 3D tracking on these benchmarks is far from solved for any approach. The 90× speed advantage is again a robust finding that does not depend on domain specifics.

What would strengthen this claim: Evaluation on the full TAPVid-3D benchmark rather than sampled subsets, and inclusion of confidence intervals given the small sample sizes. Comparison against additional 3D tracking baselines (beyond SpaTracker, which is the primary specialized competitor listed) would provide a fuller picture.

Claim: Trace Anything exhibits emergent abilities including goal-conditioned manipulation, motion forecasting, and spatio-temporal fusion.

What the experiments actually show: The paper provides qualitative demonstrations of these capabilities. Motion forecasting (Figure 10) shows trajectory extrapolation via tangent continuation on a small number of examples. Goal-conditioned manipulation (Figure 6) shows trajectory field prediction from image pairs on BridgeData, but "goal-conditioned" is not evaluated as a manipulation task—there is no measurement of whether the predicted trajectories, when executed by a robot, actually achieve the goal state. Spatio-temporal fusion (Figure 11) shows a qualitative example of aggregating observations into a canonical frame. Instruction-based forecasting (Figure 12) uses an external video generation model (Seedance 1.0) to generate future frames, then applies Trace Anything to lift them to 3D—this demonstrates composability with generative models rather than an emergent capability of Trace Anything itself.

Assessment: The "emergent capabilities" claim is the weakest part of the experimental evaluation. The paper provides no quantitative metrics for any of these capabilities. The demonstrations are on a handful of cherry-picked examples. "Goal-conditioned manipulation" is not evaluated as a manipulation policy—there is no robot execution, no success rate, no comparison against behavior cloning or other manipulation methods. "Motion forecasting" via tangent continuation assumes constant velocity beyond the observed window, which will fail for accelerating, decelerating, or direction-changing motion—no failure cases are shown. "Spatio-temporal fusion" is demonstrated for one example without comparison to dedicated fusion methods.

What would strengthen this claim: Quantitative evaluation of forecasting accuracy (e.g., prediction error at future time steps against ground truth), comparison of goal-conditioned manipulation success rates against baseline policies, quantitative fusion quality metrics (e.g., PSNR of reconstructed canonical frames), and analysis of failure modes. Without such evaluation, the "emergent capabilities" section is best understood as a vision for what trajectory fields might enable, not as empirically validated functionality.

General Strengths of the Experimental Design

  • The introduction of SDD and CA metrics alongside standard EPE provides a more complete picture of trajectory field quality than position error alone. These metrics directly measure compliance with the consistency conditions (C1 and C2) that the paper argues are essential properties of a good trajectory field. The fact that baseline methods—not designed with these conditions in mind—score substantially worse on SDD and CA (Table 1: SDD 1.06 for Trace Anything vs. 1.44–1.76 for baselines; CA 5.09 vs. 5.15–9.37) validates the diagnostic value of these metrics.

  • The all-to-all evaluation protocol is a genuinely harder task than standard first-to-all tracking, and the paper's demonstration that prior methods are not designed for it is well-supported by the CA and SDD results. Methods that excel at forward tracking from the first frame (e.g., CoTracker3-based pipelines) show degraded performance when required to produce full trajectories from every frame.

  • The out-of-distribution evaluation on PointOdyssey (Table A) is an important addition that helps readers gauge the domain gap. The fact that the performance advantage shrinks substantially is a valuable finding that the paper reports transparently.

  • The 3D tracking evaluation on TAPVid-3D (Table B) demonstrates that the trajectory field representation generalizes to a different task (sparse point tracking) without fine-tuning, providing evidence for the versatility of the representation.

General Weaknesses and Missing Experiments

  • No real-world quantitative evaluation for the primary task. The paper's title promises to represent "any video" in 4D, but the only quantitative evaluation on the trajectory field estimation task uses synthetic data from the same distribution as training. The qualitative DAVIS and BridgeData results are suggestive but do not constitute rigorous evaluation.

  • Missing ablation of regularization losses. The static, rigidity, and correspondence regularizations (Equations 11–13) are presented as important components of the training scheme, but no experiment isolates their individual contributions. Without these ablations, the paper cannot claim that the regularizers are necessary or even helpful—the model might learn the same properties from the core trajectory loss alone.

  • Missing ablation of confidence adjustment. The confidence-weighted loss (Equation 9) introduces additional complexity (per-control-point confidence prediction, the α log Σ̂ regularizer) without evidence that it improves over a standard unweighted L2 loss.

  • Missing ablation of the fusion transformer design. The choice to use joint all-to-all attention across all frames is central to the paper's claim of avoiding pairwise inference, but there is no comparison against alternative architectures (e.g., pairwise processing with post-hoc fusion, recurrent processing) to demonstrate that the joint design is necessary for the reported accuracy.

  • No sensitivity analysis for key hyperparameters. The number of frames per batch (up to 30), the image resolution (512px), the choice of 10 control points vs. alternatives, and the λ weights in the loss function are all stated without systematic study of how performance varies with these choices. The single ablation table (Table C) partially addresses control point count and backbone, but does not explore interactions.

  • Small benchmark size for 3D tracking. The TAPVid-3D evaluation uses 50 videos per subset, sampled from the full benchmark. It is unclear how these subsets were selected and whether the results are stable across different random samples. No confidence intervals are reported.

  • No comparison against per-scene optimization methods on the trajectory field task. The paper's baselines are all feed-forward methods (plus CoTracker3+VGGT, which includes learned components). Methods like OmniMotion (Wang et al., 2023) that perform per-scene test-time optimization are discussed in related work but not quantitatively compared. The claim that feed-forward processing can replace optimization-based approaches would be strengthened by a direct comparison.

  • The stage-wise runtime analysis (Figure 9) shows only runtime, not accuracy. As the number of frames increases, runtime scales approximately linearly (which is good), but it is unknown whether accuracy degrades, plateaus, or improves.

  • No analysis of failure cases. The paper does not systematically categorize when trajectory field estimation fails—for example, whether failures correlate with occlusion severity, motion speed, textureless regions, or non-rigid deformation complexity. The qualitative results show successes; failures are not presented or analyzed.

In summary, the experimental evaluation robustly demonstrates that (1) Trace Anything is substantially faster than prior methods, and (2) on in-distribution synthetic data, it achieves the best accuracy-consistency tradeoff. The evidence for real-world accuracy, emergent capabilities, and the necessity of specific architectural and training design choices is weaker. The paper would be strengthened by real-world quantitative evaluation, comprehensive ablation studies, and failure case analysis.

6. Limitations and Trade-offs

1. The Accuracy Advantage Is Demonstrated Only on In-Distribution Synthetic Data

The assumption or constraint: The paper's headline quantitative results (Tables 1 and 2) are evaluated exclusively on the authors' own Trace Anything Benchmark, a synthetic dataset generated by the same Blender-based platform that produced the training data. The model is trained on 10K+ videos from this platform (plus 20K Kubric videos) and tested on a held-out set of 200 videos from the same platform, with the same rendering engine, the same asset distributions, and the same categories of motion. The burden of proof for claims about representing "any video" rests on showing that the model transfers to real-world footage, yet the paper provides no quantitative evaluation on real videos for the primary trajectory field estimation task.

The consequence: The central empirical claim of the paper—that Trace Anything achieves state-of-the-art performance on trajectory field estimation—is valid only for synthetic data within the training distribution. A practitioner considering deploying this method on real-world videos cannot rely on the reported accuracy numbers (EPEmix 0.234, SDD 1.06, CA 5.09) as indicative of expected real-world performance. The domain gap between synthetic renderings and real videos is well-documented in 3D vision (differences in lighting, texture complexity, motion blur, sensor noise, camera artifacts), and methods that perform well on synthetic benchmarks often degrade significantly on real data. The qualitative results on DAVIS (Figure 5) and BridgeData (Figure 6) demonstrate that the model produces visually plausible outputs on real videos, but plausibility does not guarantee geometric accuracy—there is no ground truth to verify whether the predicted 3D trajectories are correct.

What evidence exists in the paper: The out-of-distribution evaluation on PointOdyssey (Appendix C.3, Table A) provides the most direct evidence of the domain gap. On this synthetic-but-unseen benchmark, Trace Anything's advantage over the best baseline (St4RTrack) shrinks from a 14% relative EPEmix reduction in-distribution (0.234 vs. 0.272) to a 4.8% reduction out-of-distribution (0.256 vs. 0.269). On dynamic regions specifically, the two methods are essentially tied (EPEdyn 0.319 vs. 0.325). This quantifies what practitioners should expect: the model generalizes partially to unseen synthetic data but loses most of its accuracy advantage. Generalization to real videos—which are further from the training distribution than PointOdyssey is—is likely even weaker, though the paper provides no numbers to confirm or refute this.

Mitigation status: The paper does not attempt to address the domain gap. There is no fine-tuning on real data, no domain adaptation technique, and no real-world quantitative evaluation. The authors acknowledge the limitation explicitly in Appendix D:

"Since Trace Anything is trained for trajectory field estimation, we rely on synthetic data to obtain dense annotations. This inevitably introduces a domain gap with real-world scenarios. Incorporating partial annotations from real data may help bridge this gap and represents a promising direction for future work."

This is an honest acknowledgment, but it means the paper's primary quantitative claims are unvalidated for the setting that matters to most potential users—real videos. The "promising direction" of incorporating real-world partial annotations is left entirely to future work, with no preliminary experiments demonstrating feasibility.

2. The Difficulty Estimation Cost Is Unaccounted For—and It Does Not Exist

The assumption or constraint: Unlike the reference paper's compute-optimal scaling framework—which required an expensive but explicitly acknowledged difficulty estimation step (2048 samples per question)—Trace Anything does not have a difficulty estimation phase because it processes all frames in one shot. However, this absence reveals a different limitation: the model has no mechanism for adaptive computation. It always processes all input frames with the full fusion transformer, regardless of scene complexity. A simple video of a static object and a complex video with multiple articulated characters, severe occlusions, and rapid camera motion receive exactly the same computational budget (all frames processed through all-to-all attention).

The consequence: For many practical deployments, this fixed-budget approach is wasteful. If a video contains long static segments—a common scenario in surveillance, robotics (a robot waiting for a human action), or video conferencing—the model still performs expensive cross-frame attention over frames that are nearly identical. The runtime scales approximately linearly with the number of input frames (Figure 9), so a 10-minute video at 30 fps would require processing 18,000 frames, which at the reported 2.3 seconds per 30 frames would take roughly 23 minutes on an A100—impractical for many applications. A system that could detect static segments and reduce computation appropriately (e.g., by sharing trajectory estimates across redundant frames) would be far more efficient in these common scenarios.

What evidence exists in the paper: The stage-wise runtime scaling in Figure 9 shows that the fusion transformer dominates total cost and that total runtime grows with frame count. The paper provides no accuracy-vs-frames analysis—it is unknown whether processing more frames always improves accuracy or whether accuracy plateaus (or even degrades) beyond some frame count. This is critical information for a practitioner deciding how many frames to feed the model. Additionally, the paper reports that performance degrades with more frames in Appendix D:

"performance also degrades as the number of frames increases"

This is mentioned without quantification. The degradation may be due to the limited expressive power of fixed-number-of-control-point splines over long time horizons, transformer context length limitations, or optimization difficulties with longer sequences. Whatever the cause, it means the model does not simply get better with more data—there is a sweet spot, and the paper does not characterize it.

Mitigation status: The paper acknowledges this in Appendix D and suggests a workaround:

"In such cases, we mitigate the issue by clipping trajectories into fixed window sizes or downsampling frames. However, these strategies may fail in scenarios such as repeated back-and-forth motion"

The window-based clipping is a heuristic, not a learned solution. It introduces a new hyperparameter (window size) that the practitioner must tune, and it breaks the model's ability to reason about long-range dynamics across window boundaries. For motions that span windows (a person walking across a room over many seconds), clipping will produce discontinuous trajectory fragments that require yet another stitching step—reintroducing exactly the fragmentation problem the trajectory field was designed to avoid. The paper does not evaluate how window-clipping affects accuracy or consistency.

3. The B-Spline Parameterization Imposes a Hard Upper Bound on Temporal Complexity

The assumption or constraint: The trajectory field represents every pixel's 3D motion using a cubic B-spline with a fixed, small number of control points (4, 7, or 10 depending on configuration; the released model uses 10). This means every trajectory is a piecewise-cubic curve with at most three polynomial segments (for 10 control points with the knot vector in Equation 24). A cubic polynomial segment has at most two extrema and one inflection point. A trajectory composed of three such segments can therefore represent motion with at most a handful of direction changes, velocity reversals, or acceleration patterns.

The consequence: This parameterization cannot represent trajectories with complex temporal structure. A bouncing ball that reverses direction five times in a video, a dancer's hand tracing intricate patterns, a leaf tumbling in turbulent wind—these motions require more degrees of freedom than a 10-control-point cubic B-spline can provide. The spline will either smooth out the high-frequency motion (underfitting) or, if forced to pass near all the control points, will introduce spurious oscillations between knots (overfitting). This is not a training data limitation—it is a fundamental representational ceiling. No amount of data will teach a 10-control-point spline to represent a trajectory that inherently requires 30 degrees of freedom.

The real-world consequence is that Trace Anything will systematically fail on videos with complex, rapid, or repetitive motion. Scenarios like sports footage (a tennis player's serve involves complex arm and racket trajectories with multiple acceleration phases), mechanical systems (pistons, gears), or natural phenomena (water splashes, flag fluttering) are outside the representational capacity of the model regardless of training data quality.

What evidence exists in the paper: The ablation in Table C provides partial evidence: increasing control points from 4 to 7 improves EPEmix from 0.281 to 0.237 (a 15.7% relative improvement), but increasing from 7 to 10 yields only a marginal gain to 0.234 (1.3%). This suggests diminishing returns—additional control points beyond 10 would likely provide negligible benefit on the current benchmark because the benchmark's motion complexity does not exceed what 10 control points can represent. The paper does not evaluate on a benchmark with deliberately complex trajectories (e.g., high-frequency oscillatory motion, chaotic dynamics) to probe the representational ceiling.

The paper acknowledges the limitation in Appendix D:

"Our parametric curve representation, with a limited number of control points, has restricted expressive power for highly complex motions."

The admission is honest but provides no quantification of where the limit lies. At what motion frequency or complexity does the spline become the bottleneck? The paper offers no answer.

Mitigation status: The only mitigation suggested is increasing the number of control points, but the ablation already shows diminishing returns at 10. A "more fundamental solution" mentioned in Appendix D is training with larger-scale datasets, but this confuses data scale with representational capacity—more data does not increase the dimensionality of the output space. A genuinely more expressive parameterization (e.g., adaptive control point allocation per pixel, hierarchical splines, or learned basis functions) would be needed, and the paper does not explore these.

4. The All-to-All Benchmark Evaluates a Property That Real-World Applications Rarely Need—While Leaving Standard Benchmarks Underserved

The assumption or constraint: The paper's primary quantitative evaluation uses the Trace Anything Benchmark, which requires all-to-all trajectory prediction: every pixel in every frame must have a complete 3D trajectory spanning the entire sequence. The paper argues this is a "strictly harder task" that "requires models not only to follow motion from a single starting frame, but also to jointly capture dynamics across the entire sequence" (Section 4). While true, this claim conflates hardness with practical relevance.

In most real-world applications of dynamic scene understanding, predicting backward trajectories (where a point was before it became visible) or dense trajectories for occluded background pixels is not operationally necessary. A robot manipulator needs to know where the object it is about to grasp is now and where it will be in the near future—it does not need to retroactively predict where that object was 30 frames ago from the perspective of a frame where the object was not yet in view. A video editor applying effects to a moving subject needs forward tracking, not backward. An autonomous vehicle needs to forecast other agents' future trajectories, not reconstruct their past ones from arbitrary reference frames.

The consequence: The all-to-all evaluation protocol may reward properties that are irrelevant to practice while obscuring deficiencies on operationally important capabilities. The paper's CA (Correspondence Agreement) metric measures how consistently the same physical point is predicted from different source frames—a property that matters if you need to query the same trajectory from an arbitrary frame. But for forward-tracking applications, what matters is trajectory accuracy when tracked from the first visible frame, which standard first-to-all benchmarks measure. Trace Anything's CA advantage over baselines (5.09 vs. 5.15–9.37) may reflect an architectural property (joint processing) that is valuable for all-to-all consistency but irrelevant for forward-only use cases. Meanwhile, the 3D tracking results on TAPVid-3D (Table B)—the only evaluation on a standard real-world benchmark—show Trace Anything roughly tied with the specialized SpaTracker (APD3D 20.5 vs. 18.3 on ADT, 15.5 vs. 16.0 on DriveTrack), despite running 90× faster. These are competitive but not dominant results, and on the metric that arguably matters most for tracking applications (AJ, which captures occlusion handling), Trace Anything trails SpaTracker on two of three subsets.

What evidence exists in the paper: The TAPVid-3D evaluation (Table B) provides the clearest evidence. On the ADT subset, Trace Anything's AJ (15.6) lags SpaTracker's (17.4) by a meaningful margin. AJ penalizes both spatial inaccuracy and incorrect occlusion prediction—a false positive occlusion prediction (thinking a point is visible when it is not) directly impacts downstream applications like robotic grasping. The fact that the specialized SpaTracker outperforms Trace Anything on this metric suggests that the trajectory field training does not adequately handle occlusion boundaries, possibly because the B-spline parameterization smooths over occlusion events that should be discontinuous. The paper does not analyze this failure mode.

Mitigation status: The paper does not address the gap between all-to-all and first-to-all evaluation. There is no experiment that measures forward-only tracking accuracy on the Trace Anything Benchmark to determine whether the model's all-to-all advantage comes at a cost to forward-tracking quality. The paper does not report separate metrics for visible vs. occluded portions of trajectories, which would reveal whether the model's consistency (good CA) comes from reasonable interpolation in visible regions or from hallucinating plausible-but-incorrect trajectories through occlusions.

5. Geometric Regularizers Require Privileged Annotations Unavailable in Real-World Training

The assumption or constraint: The training scheme's four regularization losses—static (Equation 11), rigidity (Equation 12), correspondence (Equation 13), and timestamp (Equation 10)—all require ground-truth annotations that are available in synthetic data but not available for real videos. Specifically:

  • Static regularization requires per-pixel labels identifying which regions are static (walls, floors, background) vs. dynamic (moving objects, people). Real videos do not come with such masks.
  • Rigidity regularization requires segmentation of pixels into rigid groups and knowledge of which groups undergo purely rigid motion. Real videos with articulated or deformable objects do not provide this.
  • Correspondence regularization requires ground-truth pixel correspondences across frames—precisely the dense 3D trajectory labels that the model is being trained to predict. This is circular: the regularizer that teaches the model to produce consistent trajectories requires knowing the correct trajectories in advance.
  • Timestamp supervision requires ground-truth temporal indices when using the timestamp head, or frame metadata for video input—trivial for video, but for the "unordered image collections" use case, no ground-truth temporal ordering exists.

The consequence: The training scheme, as described, cannot be applied to real-world data. A practitioner wanting to fine-tune Trace Anything on their own domain-specific videos cannot use the static, rigidity, or correspondence regularizers because they lack the required annotations. They could omit these losses and train only with the core trajectory loss (Equation 9), but the paper provides no ablation showing how much performance degrades when the regularizers are removed. It is possible—even likely—that the regularizers are essential for learning consistent trajectory fields, and that training without them would produce trajectories that drift, deform, or fail to disentangle static and dynamic regions. Without this ablation, a practitioner cannot make an informed decision about whether the model can be adapted to their domain.

The circularity of correspondence regularization is particularly concerning for real-world adaptation. The regularizer directly penalizes trajectory inconsistency, which means the model is being told the "right answer" for cross-frame consistency during training. At test time on real videos, it must produce consistent trajectories without this direct supervision. The paper provides no evidence that the consistency learned from synthetic correspondence labels transfers to real scenes where the visual appearance, motion patterns, and occlusion characteristics differ.

What evidence exists in the paper: None. The paper does not ablate any of the regularization terms. The contribution of each to final performance is unknown. The necessity of each for learning reasonable trajectory fields is unverified. This is the most significant missing ablation in the paper, because the regularizers are presented as important components of the method (they appear in the "Final objective" equation and have dedicated subsections in Section 3.3) but their empirical value is never measured.

Mitigation status: The paper does not acknowledge this as a limitation and does not discuss how to train on real data without these privileged annotations. No self-supervised alternatives to the regularizers are proposed. No experiment demonstrates that the model can be fine-tuned on real data with only the core trajectory loss (which would itself require real 3D trajectory labels, also unavailable). The entire training pipeline is thus locked to synthetic data, and the paper provides no path—even a speculative one—toward real-world training.

6. The One-Pass Paradigm Precludes Iterative Refinement, Creating a Hard Ceiling on Accuracy

The assumption or constraint: Trace Anything is designed around a strict one-pass feed-forward paradigm: all frames are processed once, the control points are predicted in a single forward pass, and no iterative optimization or refinement is performed. This is presented as an advantage (Section 3: "eliminating auxiliary estimators and per-scene optimization"). However, it also means the model cannot correct its own errors. If the initial prediction places a control point incorrectly—due to an ambiguous texture, an unusual occlusion pattern, or a challenging motion—there is no mechanism to revise it. The model gets one shot per pixel per frame.

The consequence: This creates a hard ceiling on accuracy that is fundamentally different from the soft ceilings imposed by data quantity or model capacity. Even with infinite training data, the model will make errors on cases that are genuinely ambiguous from a single view or that require reasoning about long-range temporal dependencies that exceed the fusion transformer's effective context window. Optimization-based methods (OmniMotion, MegaSAM, St4RTrack's joint optimization) can iteratively refine their estimates, resolve ambiguities by enforcing global consistency across multiple passes, and recover from early mistakes. Feed-forward methods like Trace Anything cannot. This tradeoff is well-known in the broader 3D vision literature: feed-forward depth estimation (MiDaS, DPT) is fast but less accurate than multi-view stereo with global optimization; feed-forward optical flow (RAFT) is fast but less accurate than per-scene energy minimization. Trace Anything makes the same tradeoff for 4D reconstruction but does not characterize where its accuracy ceiling lies relative to optimization-based methods.

The practical implication is that for applications where accuracy is paramount and latency is acceptable (offline video processing, forensic analysis, scientific measurement), an optimization-based method may be preferable to Trace Anything regardless of how much training data is available. The paper never compares against per-scene optimization methods quantitatively, so a practitioner cannot determine how much accuracy they are sacrificing for speed.

What evidence exists in the paper: Indirectly, the out-of-distribution evaluation (Table A) provides evidence for a ceiling effect. On the Trace Anything Benchmark (in-distribution), Trace Anything achieves EPEmix 0.234 vs. St4RTrack's 0.278—a 15.8% improvement. On PointOdyssey (out-of-distribution), Trace Anything achieves 0.256 vs. St4RTrack's 0.269—only a 4.8% improvement, and the two methods are essentially tied on EPEdyn. St4RTrack uses joint optimization (it performs "simultaneous 4d reconstruction and tracking"), so its stronger relative performance out-of-distribution suggests that its iterative refinement provides robustness to domain shift that Trace Anything's one-pass prediction lacks. On static regions, Trace Anything retains a larger advantage (EPEsta 0.212 vs. 0.243) because static geometry is more consistent across synthetic domains and less dependent on iterative refinement. On dynamic regions, where motion patterns vary more across datasets and iterative reasoning about complex trajectories is more valuable, the advantage evaporates.

The paper acknowledges a related limitation in Appendix D:

"As the first attempt at dense per-pixel trajectory field estimation, our approach offers efficiency advantages but may be less precise than sparse 3D tracking methods [63, 64]. Incorporating fine-grained point-level estimation from such methods into our framework could be an interesting direction for future research."

This admits the precision gap, but frames it as a sparse-vs-dense tradeoff rather than a feed-forward-vs-optimization tradeoff. The suggested future direction—incorporating point-level estimation—implicitly acknowledges that the one-pass paradigm needs supplementation, but no preliminary experiments are provided.

Mitigation status: The paper suggests using Trace Anything as initialization for optimization-based methods (Section 2 mentions that "integrating NVS with our paradigm, e.g., using trajectory fields to initialize dynamic 3DGS models, is a promising future direction"), but this would reintroduce the per-scene optimization cost that the one-pass design was meant to eliminate. There is no middle ground explored—no lightweight refinement step that runs quickly (a few iterations rather than full convergence), no test-time adaptation, no uncertainty-aware post-processing. The model predicts confidences (Σ̂), but these are only used for loss weighting during training and filtering at inference; they are not used to selectively refine low-confidence predictions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a marginal improvement to an existing pipeline—it fundamentally reframes what the output of a dynamic scene understanding system should be. The dominant paradigm in 3D vision has been to treat 4D reconstruction as a composition problem: estimate geometry per frame, estimate correspondences between frames, then reconcile the two. This paper argues—through both the trajectory field formalism and the Trace Anything architecture—that 3D trajectories can be the direct, atomic output of a single feed-forward model, bypassing the entire intermediate estimation stack. This is a paradigm shift in output representation, not an incremental architecture tweak.

The magnitude of this shift becomes clear when comparing against the lineage of DUSt3R-derived methods. DUSt3R (Wang et al., 2024) showed that a network could predict 3D pointmaps from image pairs, eliminating classical SfM stages. Fast3R and VGGT extended this to joint multi-frame processing, but only for static scenes. The wave of dynamic extensions—Monst3R, POMATO, Easi3R, St4RTrack—adopted the same output format: per-frame pointmaps with post-hoc correspondence establishment. Trace Anything represents a clean break from this lineage. It does not output pointmaps and then align them; it outputs trajectories directly, making correspondence an inherent property of the output rather than a problem to be solved after prediction. The quantitative evidence for why this matters is in the Correspondence Agreement metric: Trace Anything achieves CA of 5.09 (×10⁻²) on video-based inference (Table 1), while the best DUSt3R-derived method (Easi3R) achieves 5.15, and others range from 6.24 (DELTA) to 9.37 (St4RTrack). These differences reflect a structural property—trajectory-based methods naturally produce more consistent cross-frame predictions than pointmap-based methods—not merely better training.

Reconciling prior contradictions. The paper resolves a tension that has been implicit in the 3D vision literature: feed-forward geometric models work well for static scenes but struggle with dynamics, while optimization-based methods handle dynamics but are too slow for deployment. Prior work oscillated between these poles—making feed-forward models slightly more dynamic (Monst3R) or optimization methods slightly faster (St4RTrack's joint optimization). Trace Anything demonstrates that feed-forward processing can match or exceed optimization-based dynamic reconstruction when the output representation is properly designed. The 2.3-second runtime vs. 22.5–231.6 seconds for baselines (Table 1) is not merely an engineering speedup; it represents a qualitative change in what applications become feasible. Real-time or near-real-time 4D understanding shifts from aspirational to achievable, at least for sequences within the model's training distribution.

Research directions that become more attractive. The paper makes several lines of inquiry newly compelling:

  • Unified geometric-symbolic video understanding. The trajectory field provides a dense geometric substrate—a complete 3D motion description of every visible surface point—that could serve as input to higher-level reasoning modules (physical simulation, action recognition, task planning). Prior methods produced representations (sparse tracks, per-frame depth maps) that were too impoverished or fragmented for this purpose. The trajectory field's density (every pixel has a trajectory) and consistency (all trajectories exist in a shared coordinate frame) make it a viable geometric backbone for downstream reasoning in a way that sparse point tracks or disjoint point clouds are not.

  • Geometry-first dynamic novel view synthesis. The paper explicitly notes that 4D NVS methods (dynamic NeRFs, 4D Gaussian Splatting) optimize for appearance with geometry as a means to an end, and typically require precomputed camera poses. A trajectory field—which directly provides 3D geometry and motion—could serve as initialization or geometric regularization for these rendering systems. Instead of learning geometry from scratch through photometric losses (which is slow and prone to local minima), a NVS system could start from a trajectory field and focus optimization on appearance, dramatically reducing per-scene optimization time. This is the "promising future direction" the paper flags in Section 2, and it becomes newly practical because Trace Anything provides the first feed-forward model that can produce dense, globally consistent 3D motion estimates suitable as initialization.

  • Self-supervised or weakly-supervised trajectory learning. The current training pipeline requires dense synthetic annotations and privileged regularization signals (static masks, rigidity groups, correspondence labels). The paper demonstrates that trajectory fields can be learned when these annotations are available. The obvious next challenge—which the paper makes explicit but does not attempt—is learning trajectory fields without them, using multi-view photometric consistency, cycle-consistency across time, or self-supervised signals from video. The paper's architecture and loss formulation provide a concrete target for what a self-supervised system would need to approximate, making this research direction newly tractable rather than purely aspirational.

Research directions that become less attractive. By demonstrating that feed-forward joint processing can achieve competitive accuracy with dramatically lower latency, the paper weakens the case for continued investment in pipeline-based dynamic reconstruction (separate depth estimation + optical flow + 2D tracking + global alignment). The compounding error problem that pipeline methods face is structural—each component introduces errors that the next component must handle—and the paper provides strong evidence (Table 1: CoTracker3+VGGT achieves the worst CA at 7.83 and EPEmix at 0.518) that joint models can avoid this. Future work that continues to refine individual pipeline components without addressing the structural fragmentation issue will need to demonstrate that the incremental improvements outweigh the fundamental limitation, which becomes harder to justify when a joint model achieves better accuracy 10–100× faster.

The paper also makes per-scene optimization for geometric reconstruction a harder sell for applications where latency matters. Methods like OmniMotion (Wang et al., 2023) that require minutes of per-video optimization were already impractical for online use; the paper demonstrates that at least on synthetic data, feed-forward processing can match or exceed their accuracy, removing the accuracy-efficiency tradeoff that previously justified their use for offline applications. Per-scene optimization remains valuable for applications where photorealism is paramount (the NVS domain) or where the domain gap from synthetic training data is too large, but for geometric understanding of dynamics in domains well-represented by synthetic data, the case for per-scene optimization weakens substantially.

A diagnostic rather than a solution. It is important to be precise about what the paper does and does not accomplish. The paper provides a diagnostic framework—the trajectory field formalism, the SDD and CA metrics, the all-to-all evaluation protocol—for measuring whether a model has genuinely learned consistent 4D scene dynamics. It provides a proof of concept that this can be learned from synthetic data with a feed-forward architecture. It does not provide a solution that works on real videos, nor does it characterize where the limits of the approach lie. The paper's lasting contribution is likely to be the representation and the evaluation protocol, more than the specific model weights—future methods will be judged by whether they produce trajectory fields that satisfy conditions C1 and C2 as measured by SDD and CA, even if their architectures and training data differ.

Follow-Up Research This Work Enables

Real-world trajectory field estimation with self-supervised adaptation. The most pressing gap is the domain gap between synthetic training data and real videos. A concrete next step: take the pretrained Trace Anything model, freeze the backbone, and fine-tune only the control point head on real videos using self-supervised losses. Multi-view photometric consistency (warping pixels between frames using the predicted trajectories and requiring color constancy), temporal cycle-consistency (tracking forward then backward should return to the starting point), and flow-based distillation (using off-the-shelf optical flow to supervise the 2D projection of the 3D trajectories) are all candidates. The key experiment would measure: (a) how much real-world EPE improves with self-supervised fine-tuning on a small dataset with ground truth (e.g., a multi-camera rig), (b) whether the CA and SDD metrics degrade during self-supervised adaptation (indicating the model is fitting to per-frame signals at the cost of global consistency), and (c) how many real videos are needed before self-supervision saturates.

Adaptive control point allocation. The fixed 10-control-point B-spline parameterization imposes a hard ceiling on representable motion complexity, and the ablation (Table C) shows diminishing returns beyond 7–10 control points. A strong follow-up would replace the fixed allocation with a learned, per-pixel control point count. A pixel on a static wall needs 1 control point (all identical); a pixel on a smoothly moving vehicle needs ~4; a pixel on a dancer's hand needs ~15 or more. The architecture would predict, for each pixel, both a variable-length set of control points and a termination probability. Training would require a sparsity penalty or a reinforcement learning formulation (the model is rewarded for accuracy minus a cost per control point). The key experiments: (a) does adaptive allocation improve accuracy on complex-motion benchmarks while maintaining or reducing total control points on simple scenes? (b) does it enable the model to handle sequences longer than the 30–60 frames that the fixed parameterization supports? (c) how does the distribution of control point counts vary across scene types (static backgrounds, rigid objects, articulated characters, fluids)?

Failure mode taxonomy and targeted architecture improvements. The paper provides no systematic failure analysis. A rigorous follow-up would construct a diagnostic benchmark that isolates specific failure modes: (a) occlusion severity (what fraction of a trajectory is visible?), (b) motion complexity (number of direction changes, maximum velocity, acceleration magnitude), (c) texture sparsity (how many trackable features exist in the local neighborhood?), (d) camera motion type (static, smooth, jerky, large baseline), (e) non-rigid deformation magnitude (how much do internal distances change?). For each axis, measure Trace Anything's per-bin accuracy and identify where performance collapses. This would tell practitioners when the method can be trusted and would guide architectural improvements—for example, if occlusion is the primary failure mode, incorporating explicit occlusion reasoning (predicting visibility masks alongside trajectories) becomes the priority; if motion complexity is the bottleneck, more expressive trajectory parameterizations are needed.

Trajectory-conditioned video generation and editing. The paper demonstrates composability with generative models (Seedance 1.0 for instruction-based forecasting, Figure 12), but this is a loose coupling: generate future frames, then lift to 3D. A tighter integration would condition a video generation model directly on trajectory fields—given an initial frame and a desired trajectory field (specifying where every pixel should move), generate a video that is geometrically consistent with those trajectories. This is the inverse of Trace Anything: instead of inferring trajectories from video, synthesize video from trajectories. The key experiment would measure whether trajectory-conditioned generation produces videos with more consistent object motion (less drifting, fewer identity switches) than unconditional or text-conditioned generation, and whether generated videos, when fed back into Trace Anything, recover trajectories close to the conditioning input (a cycle-consistency check).

Trajectory fields for physical simulation and world models. The trajectory field provides a dense, pixel-aligned representation of motion that could serve as the observation space for learned physical simulators. A concrete experiment: train a dynamics model that takes a trajectory field at time t and predicts the trajectory field at time t+Δt, trained on synthetic data where ground-truth future trajectories are available. Compare this against dynamics models that operate on other representations (RGB pixels, object-centric latents, 2D flow fields) on metrics of long-horizon prediction accuracy, physical plausibility (does the predicted motion conserve momentum, respect contact constraints?), and sample efficiency. The hypothesis is that trajectory fields, being geometrically grounded and inherently 3D, should require less capacity to learn physical dynamics than pixel-space or flow-field representations because they factor out appearance and camera projection.

Stress-testing the representation: how far can 10 control points go? The paper's ablation shows accuracy plateaus at ~10 control points, but provides no analysis of what kinds of motion saturate this capacity. A diagnostic experiment would generate synthetic trajectories with systematically varying complexity—sinusoidal motion at increasing frequencies, piecewise-linear paths with increasing numbers of segments, chaotic attractors—and measure reconstruction error as a function of control point count. This would produce a phase diagram showing the relationship between motion complexity, control point count, and achievable accuracy. The practical output would be a guideline: for motions with characteristic frequency f and sequence duration T, you need approximately K control points for <5% reconstruction error. This would also reveal whether the bottleneck is the spline representation itself (true representational ceiling) or the network's ability to accurately predict control points for complex motions (learning difficulty), which have different remedies.

Practical Applications and Downstream Use Cases

Low-latency 4D scene understanding for robot manipulation. In robotic manipulation, the ability to perceive 3D object motion in near-real-time is essential for dynamic tasks—grasping moving objects, avoiding collisions with humans, or inserting a peg into a moving hole. Current systems typically rely on 2D tracking (which loses depth information) or sparse 3D keypoints (which miss fine-grained surface geometry). Trace Anything's 2.3-second runtime on 30-frame sequences (Table 1) and 0.20-second runtime on image pairs (Table 2) make it the first feed-forward method fast enough for online use in a manipulation loop running at several Hz. On the BridgeData V2 manipulation dataset (Figure 6), the model produces plausible 3D trajectories of robot arms and objects from just initial and goal images—a setting directly relevant to goal-conditioned manipulation, where a policy needs to infer what 3D motion transforms the current scene into the goal configuration. The key practical benefit: a single model provides dense, per-pixel 3D motion estimates without requiring depth sensors, multi-camera rigs, or per-scene calibration, dramatically reducing the sensing hardware and computation needed for dynamic manipulation.

Efficient batch processing for video annotation and search. For applications that process large video archives—content moderation, sports analytics, surveillance review, scientific video analysis—the 10–100× speed advantage over prior methods (Tables 1 and 2) translates directly to cost savings. Processing a million videos with a method that takes 100 seconds per video requires ~1,157 GPU-days; at 2.3 seconds per video, the same workload requires ~27 GPU-days—a ~43× reduction. The trajectory field output provides a searchable representation: queries like "find all videos where an object moves from left to right at more than 2 meters per second" can be answered by evaluating trajectory derivatives, without running separate object detectors or trackers. The dynamic mask capability (Appendix C.1, Figure B)—obtained by thresholding control point variance—provides free motion segmentation, enabling filtering of static-camera footage for segments containing activity. The key caveat is that this benefit currently requires the video domain to be well-represented by the synthetic training data; real-world accuracy on arbitrary footage is unverified.

Initialization for high-quality dynamic novel view synthesis. The 4D NVS community has developed stunning rendering quality (dynamic NeRFs, 4D Gaussian Splatting), but these methods require substantial per-scene optimization time—often hours—because they must learn both geometry and appearance from scratch. A trajectory field, produced in 2.3 seconds, provides a dense geometric initialization: 3D point positions, per-point trajectories, and static/dynamic segmentation. Using this to initialize a dynamic 3DGS model—placing Gaussians at the trajectory field's predicted 3D positions and using the trajectories as deformation priors—could dramatically reduce optimization time by providing a strong geometric starting point. The accuracy numbers (EPEmix 0.234 on synthetic data, Table 1) suggest the initialization would be within a few percent of ground-truth geometry, meaning the NVS optimization could focus on refining appearance and fine geometric details rather than discovering coarse structure. This composability is the specific "promising future direction" the paper flags in Section 2, and it is actionable today because Trace Anything and 3DGS implementations are both publicly available.