DSI-Bench: A Benchmark for Dynamic Spatial Intelligence

🎯 Pitch

Large vision-language models can't tell if you're moving or the world is moving around you—even top models mistake object motion for observer motion nearly half the time. DSI-Bench reveals this fundamental confusion through 1,700 questions across nine decoupled motion patterns, showing that scaling up models improves raw accuracy but leaves them just as brittle when the same scene is simply flipped or reversed.

1. Executive Summary

This paper introduces Dynamic Spatial Intelligence as a new evaluation paradigm and proposes DSI-Bench, a benchmark of nearly 1,000 dynamic videos with over 1,700 manually annotated VQA questions that systematically decouple nine motion patterns of observers and objects. The benchmark evaluates 14 vision-language models and 3D expertise models through spatially and temporally symmetric question designs—horizontal flipping and temporal reversal producing four variants per video—to isolate biases and hallucinations in dynamic spatial reasoning, revealing that models conflate observer and object motion (substituting one entity's movement for the other's), exhibit strong forward-motion selection bias (choosing "forward" far more often than the ground-truth distribution), and confuse rotation with translation when inferring observer movement. Free-form reasoning yields only marginal benefits—QwenVL2.5-72B gains just 0.17% overall accuracy while Gemini-2.5-Pro and InternVL3.5-241B perform worse than direct answering—and larger models improve accuracy but degrade in robustness under group-wise evaluation (requiring at least 3 of 4 augmented variants correct), establishing that model scale alone addresses perceptual precision without resolving the fundamental spatial biases that govern dynamic scene understanding.

2. Context and Motivation

The Core Problem: We Evaluate Spatial Intelligence in Static Worlds

The fundamental gap this paper identifies is deceptively simple: we systematically evaluate vision-language models' spatial reasoning, but almost entirely in settings where nothing moves. The real world is dynamic — observers walk, drive, and turn their heads while objects around them simultaneously translate, rotate, and change configuration. Yet the benchmarks we use to measure whether models "understand space" treat dynamic scenarios as an afterthought at best, and at worst, as something that static-scene reasoning should trivially handle.

This gap matters because it creates a false sense of progress. A model that correctly answers "the chair is to the left of the table" in a single image says nothing about whether it can track relative positions when the viewer pans right while the table slides forward. The paper argues that these are fundamentally different cognitive demands: dynamic spatial reasoning requires decoupling the observer's motion from object motion, maintaining consistent spatial representations across temporal changes, and inferring relative relationships from evolving, often ambiguous visual evidence. Humans do this instinctively; current VLMs, the paper demonstrates, do not.

Why Dynamic Spatial Intelligence Matters

The paper motivates this problem along two dimensions — practical deployment and cognitive plausibility — which together establish that this is not merely a niche academic concern.

Real-world deployment demands dynamic understanding. The paper opens with evocative examples that ground the motivation in human experience (Section 1): early humans chasing prey had to track both their own running path and the animal's evasive movements, adjusting continuously in a coupled dynamic system. Modern analogues permeate daily life — drivers merging onto highways must simultaneously track their own vehicle's motion and predict the trajectories of surrounding cars; augmented reality systems must maintain consistent spatial anchors as users move through environments; robotic assistants must navigate shared spaces with humans who are also in motion. These application domains all require models that can reason about two simultaneous, independent motion streams and understand how their interaction changes spatial relationships over time.

If VLMs cannot handle these scenarios — and the paper's results strongly suggest they cannot — then the current generation of multimodal systems is fundamentally unfit for deployment in any setting where either the agent or the environment is mobile. This is not an edge case; it is most settings.

A cognitive capability that should exist but doesn't. The paper implicitly argues that dynamic spatial reasoning is a basic cognitive building block that "intuitive... spatial intelligence" (Section 1) should encompass. The fact that models with strong 2D detection, temporal action recognition, and even static spatial reasoning capabilities nonetheless fail systematically on dynamic tasks suggests that these capabilities do not compose naturally. You cannot just bolt together a good object detector, a camera pose estimator, and a temporal understanding module and expect coherent dynamic spatial reasoning to emerge. This represents a fundamental architectural gap rather than a refinement problem.

Where Prior Approaches Fall Short

The paper identifies specific, concrete limitations across three categories of prior work, each of which motivates a different aspect of DSI-Bench's design.

Static spatial reasoning benchmarks ignore motion entirely. The paper cites VSI-Bench (Yang et al., 2025a), MMSI-Bench (Yang et al., 2025b), and VLM4D (Zhou et al., 2025) as recent efforts to evaluate spatial intelligence in VLMs. VSI-Bench measures performance across 8 categories of spatial reasoning — relative position, distance estimation, orientation, etc. — but operates on static scenes. MMSI-Bench introduces the useful constraint that some problems require multiple viewpoints to solve, but those viewpoints are independent static images, not temporally connected frames from a moving observer. VLM4D treats video as a four-dimensional modality, which sounds closer to dynamic evaluation, but the paper groups it with these static benchmarks, implying that its video understanding does not specifically target the decoupled observer-object motion challenge that DSI-Bench makes central.

The limitation is not that these benchmarks are poorly designed — they advance static spatial evaluation significantly — but that they create a evaluation blind spot. A model scoring high on VSI-Bench might appear to have strong spatial reasoning, yet crumble when the scene starts moving. The paper's dynamic-vs-static performance gap analysis (Figure 5) makes this concrete: the same model on the same task type (e.g., perceiving object motion) drops by up to 11.55% (Seed-1.6-vision) when the observer is moving compared to when the observer is static. This is a direct empirical demonstration that static benchmarks overestimate dynamic capability.

Existing benchmarks overlook multimodal hallucination and spatial biases. The paper explicitly credits 3DSR-Bench (Ma et al., 2024) and Ori-Bench (Wang et al., 2024c) with addressing a critical gap: models don't just fail at spatial reasoning — they exhibit systematic biases and hallucinations. 3DSR-Bench introduced the Evalflip strategy — horizontally flipping images and checking whether model predictions change accordingly — to detect orientation hallucinations. Ori-Bench demonstrated that VLMs are misled by semantic priors: when an object's typical orientation conflicts with its actual orientation in an image, models default to the semantic expectation, hallucinating a rotated object.

These are important methodological contributions that DSI-Bench directly builds on. However, they share the same fundamental limitation: they operate in static scenes. The paper argues that moving to dynamic scenarios introduces qualitatively new types of bias. The forward-motion bias documented in DSI-Bench (Figure 6) is not simply a static orientation hallucination — it involves temporal prediction of movement. The coupled motion reasoning errors (Figure 8) have no static analog: they emerge specifically because the model must attribute visual change to one of two simultaneously moving entities and gets the attribution wrong. These failure modes are invisible in static benchmarks.

3D expertise models assume static geometry. The paper evaluates two state-of-the-art 3D expertise models — VGGT (Wang et al., 2025a) and SpatialTrackerV2 (Xiao et al., 2025) — and finds that while they are significantly more robust than VLMs under the group-wise evaluation (minimal degradation when videos are flipped/reversed), they exhibit instability when classical geometric constraints are applied to dynamic scenes (Section 5.2). The paper specifically notes that "viewpoint changes and foreground occlusions during motion can cause keypoint loss, leading to instability in the bundle adjustment stage" and that "classical geometric constraints are primarily designed for static scenes" (Section 2.3). Structure-from-motion (Schönberger & Frahm, 2016) and multi-view stereo (Furukawa et al., 2015; Schönberger et al., 2016) assume that observed scene points are stationary; when both camera and objects move, these constraints break down.

This is a subtle but important point: even purpose-built 3D vision systems have difficulty with dynamic scenes, not because they're poorly designed, but because the mathematical foundations they rest on — epipolar geometry, triangulation, bundle adjustment — were built for static worlds. The paper positions DSI-Bench as exposing this limitation, which has practical implications for robotics and AR systems that rely on these geometric pipelines.

How This Paper Positions Itself

The paper frames its contribution not as a new model or method, but as a benchmark that defines and systematically evaluates a neglected capability. This is consistent with the benchmark-heavy tradition in VLMs — BLINK, MMBench, Video-MME, etc. — but with a specific philosophical stance: dynamic spatial reasoning is not just "spatial reasoning plus time." It is a distinct cognitive challenge that requires decoupled understanding of self-motion and object motion.

This stance is operationalized through several deliberate design choices that differentiate DSI-Bench from prior work:

The nine motion patterns as a taxonomy. By categorizing motion into three types (translation, rotation, combination) for two entities (observer, object) and further decoupling them into nine patterns (motion types cross motion categories), the paper provides a systematic decomposition rather than an ad-hoc collection of videos. This enables targeted error analysis — the paper can say not just "models struggle with dynamic scenes" but specifically "models confuse rotation with translation when inferring observer motion" (Section 5.1, Figure 7). Each finding maps to a specific motion pattern in the taxonomy.

Spatio-temporal symmetry as a bias-detection tool. The horizontal flipping and temporal reversal augmentation is not just data augmentation — it is an experimental methodology for measuring robustness. By comparing sample-wise accuracy (each variant counted independently) to group-wise accuracy (all four variants must be mostly correct), the paper can distinguish between models that get the right answer through genuine understanding versus those that succeed through brittle heuristics. A model that answers correctly on the original video but fails on the horizontally flipped version is not "understanding" the spatial relationship — it's exploiting a bias. This methodology is explicitly credited to 3DSR-Bench's Evalflip strategy and Ori-Bench's orientation analysis, but extended from static images to dynamic video.

The task taxonomy (Observer-Scene, Object-Scene, Observer-Object). By organizing questions around the three fundamental 3D entities and their pairwise relationships, DSI-Bench ensures comprehensive coverage and isolates which specific spatial reasoning sub-capability is failing. When the paper reports that VLMs "conflate observer and object motion" (Section 5.1, Figure 8), it can trace this specifically to the Observer-Object task category, where models must track the relative relationship between two independently moving entities.

Including both VLMs and expertise models. Most VLM benchmarks evaluate only VLMs. DSI-Bench's inclusion of VGGT and SpatialTrackerV2 serves a specific analytical purpose: it establishes an upper bound on what geometric approaches can achieve, revealing that the problem is hard even for specialized systems. This makes the poor VLM performance more meaningful — it's not just that VLMs are undertrained, but that dynamic spatial reasoning is genuinely challenging and current geometric constraints provide incomplete solutions.

Static cases as a special case of dynamic. The paper explicitly frames static scenarios as one point in the broader space of possible motion patterns (no observer motion, no object motion). This is a subtle but important conceptual move: it means DSI-Bench is not just a "dynamic benchmark" but a unified evaluation framework where static performance provides a baseline and dynamic performance reveals the gap that static-only evaluations hide. The performance drop from static to dynamic conditions (Figure 5) becomes the primary metric of interest — not absolute accuracy.

In summary, the paper positions DSI-Bench as filling a specific, well-motivated gap: the absence of systematic evaluation for how models handle simultaneous, independent observer and object motion in 3D scenes. It argues this gap exists because (1) spatial reasoning benchmarks are overwhelmingly static, (2) hallucination/bias analyses have focused on static image perturbations, and (3) classical 3D vision pipelines assume rigid scenes that break under dual-motion conditions. The benchmark's design — the motion taxonomy, the spatio-temporal symmetry methodology, the three-entity task decomposition, and the dual evaluation of VLMs and expertise models — is engineered to directly test the decoupled motion understanding capability that the paper identifies as missing.

3. Technical Approach

3.1 Reader Orientation

DSI-Bench is a carefully constructed evaluation dataset and testing protocol, not a new model or training method. It solves the problem of measuring whether vision-language models and 3D vision systems can independently reason about two simultaneous, decoupled motion streams — the observer's movement and the observed object's movement — by providing a benchmark where videos, questions, and answer options are systematically designed, augmented with spatio-temporal flips, and evaluated under two complementary scoring strategies (sample-wise and group-wise) to expose biases, hallucinations, and genuine understanding versus brittle heuristics.

3.2 Big-Picture Architecture (Diagram in Words)

The DSI-Bench system has five major components that operate in a pipeline from raw data to final evaluation scores:

Video Collection and Standardization — gathers nearly 1,000 dynamic videos from diverse sources (camera motion datasets, object motion datasets, synthetic control datasets, and web sources), standardizes them to 480p resolution at 3-second duration, and filters out clips with irregular observer motion using SpatialTrackerV2.
Motion Annotation Pipeline — human experts manually label each video with the motion patterns of both the observer and the observed object (translation, rotation, or combination), creating the ground-truth structure that enables systematic question generation.
VQA Generation Engine — uses template-based construction seeded by motion annotations to produce over 1,700 question-answer pairs organized into three task categories (Object-Scene, Observer-Scene, Observer-Object), with human review and refinement to ensure clarity.
Spatio-Temporal Augmentation Module — horizontally flips and temporally reverses each video to produce four variants per original (standard, horizontal flip, reverse, reverse+horizontal flip), with corresponding symmetric answer-option updates applied via rule-based substitution and manual correction.
Dual-Protocol Evaluation Harness — tests models under two answering modes (RAWQA direct answering, FFR free-form reasoning with chain-of-thought), computes two accuracy metrics (sample-wise treats all four augmented variants as independent; group-wise requires at least 3 of 4 correct per video group), and separately evaluates VLMs and 3D expertise models through appropriate interfaces.

Information flows as follows: raw videos enter → PySceneDetect splits them into scenes, SpatialTrackerV2 filters unstable clips, and human experts select start/end points → motion annotations are recorded → templates produce draft VQA pairs → human experts review and refine → spatio-temporal flips generate four variants per video → rule-based substitution updates answer options → manual correction fixes reference-frame ambiguities → VLMs receive video clips with system prompts in RAWQA or FFR mode → 3D expertise models receive videos through their native inference pipelines with calibrated keypoints and orientation masks → a rule-based procedure maps extracted trajectories and poses to answer choices → sample-wise and group-wise accuracy scores are computed.

3.3 Roadmap for the Deep Dive

First, the video collection and standardization pipeline — where the data comes from, how it is cleaned, and why each source was chosen — because the diversity and quality of the underlying video data determine what the benchmark can measure.
Second, the motion annotation and question generation process, including the template-based approach, the three-task taxonomy, and the human review loop, since this is where raw motion labels become structured evaluation items.
Third, the spatio-temporal flip augmentation methodology — the horizontal flipping, the temporal reversal, the rule-based answer-option updates, and the manual correction for reference-frame shifts — because this is the core experimental innovation that enables bias detection and robustness measurement.
Fourth, the evaluation protocols for VLMs (RAWQA vs. FFR, system prompts, sampling parameters) and 3D expertise models (keypoint calibration, mask refinement, trajectory-to-answer mapping), since the evaluation design directly shapes what conclusions can be drawn.
Fifth, the two-tier scoring system (sample-wise vs. group-wise accuracy) — what each metric measures, why they diverge, and what the gap between them reveals about model robustness versus superficial correctness.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a benchmark construction and evaluation methodology paper whose core idea is that dynamic spatial intelligence requires decoupled understanding of self-motion and object motion, and that systematic evaluation with symmetric perturbations can distinguish genuine spatial reasoning from brittle pattern matching and semantic bias exploitation.

Video Collection and Standardization

DSI-Bench draws its video data from four categories of sources, each selected to contribute specific motion characteristics that together span the space of dynamic scenarios the benchmark targets.

Source datasets and their roles. The paper collects videos from:

CameraBench (Lin et al., 2025): a dataset specifically designed for understanding camera motion, providing videos where the observer motion is well-characterized and diverse. This source anchors the observer-motion dimension of the benchmark.
Kinetics-700 (Smaira et al., 2020): a large-scale human action recognition dataset containing diverse object motions (people performing actions, objects being manipulated). This source supplies videos where the observed object is in motion, often with a stationary or minimally moving observer.
SynFMC (Shuai et al., 2025): a synthetic video generation dataset with controllable camera and object motions. Because motions are procedurally generated with known ground truth, this source provides clean, unambiguous dynamic scenarios where the observer and object motion patterns are exactly specified — eliminating the annotation ambiguity that can arise with real-world videos.
LLaVA-178K (Zhang et al., 2024) and additional online sources: these supplementary sources increase the diversity of motion patterns beyond what the three primary datasets cover, ensuring the benchmark captures a wide range of real-world dynamic relationships.

The deliberate combination of real-world videos (CameraBench, Kinetics-700), synthetic videos with ground-truth motion control (SynFMC), and diverse web sources creates a dataset where some videos have precisely known motion parameters while others reflect natural, unconstrained dynamics. This mixture enables the benchmark to test both idealized scenarios (where correct answers are unambiguous) and realistic scenarios (where motion must be inferred from visual evidence alone).

Video preprocessing pipeline. The processing follows a five-stage sequence designed to produce clean, standardized clips suitable for systematic evaluation:

Scene detection: PySceneDetect (Castellano) segments raw videos into individual scenes, splitting at shot boundaries so that each clip represents a temporally coherent segment without abrupt cuts. This is essential because dynamic spatial reasoning requires continuous visual evidence — a cut disrupts the motion trajectory and makes the temporal relationship between frames ambiguous.
Motion quality filtering: SpatialTrackerV2 (Xiao et al., 2025) processes each segmented clip to estimate camera and object trajectories, and clips exhibiting irregular or jittery observer motion are discarded. The paper does not specify the exact threshold for "irregular or jittery," but the purpose is clear: clips where the observer motion cannot be reliably characterized would produce ambiguous ground-truth annotations and introduce noise into the evaluation.
Human selection and trimming: human experts review the surviving clips and determine the starting and ending timestamps for each video. This step ensures that each clip contains a coherent, interpretable motion segment — for example, a continuous forward walk by the observer while an object rotates, rather than a partially captured or ambiguous sequence. The human judgment here is critical because automated scene detection and motion filtering can produce technically clean clips that nonetheless contain confusing or uninterpretable dynamics.
Resolution standardization: all videos are resized to 480p resolution. The paper does not specify the exact pixel dimensions (likely 854×480 for 16:9 aspect ratio or 640×480 for 4:3), but the standardization ensures that all models receive input at a consistent scale, eliminating resolution as a confounding variable in performance comparisons.
Duration normalization: overly short clips are slowed down to a duration of exactly 3 seconds. This is a practical choice: very short clips might not provide enough temporal evidence for models to perceive motion patterns, while 3 seconds at the standardized 5 fps sampling rate (used during evaluation — see below) yields 15 frames, which is sufficient for motion perception without being computationally burdensome. The paper does not specify the playback speed adjustment algorithm (e.g., frame interpolation versus simple frame duplication), but the intent is to guarantee a minimum temporal window for dynamic understanding.

Motion type coverage. The paper categorizes all motion into three types for each entity (observer and object), producing nine combinations when observer and object motions are decoupled:

Translation: movement along a linear path (e.g., moving forward, upward, leftward, backward). The paper specifically lists "forward" and "upward" as translation examples.
Rotation: in-place rotation around an axis (e.g., rotating clockwise or counterclockwise).
Combination: simultaneous translation and rotation (e.g., forward-left turns, forward-right turns).

These three motion types apply independently to both the observer and the object, creating a 3×3 grid of decoupled motion patterns. The middle and right panels of Figure 2 display the resulting motion distributions, showing that after spatio-temporal flip augmentation (described below), the benchmark achieves a roughly balanced distribution across motion types — a deliberate design choice to prevent motion-type biases from distorting evaluation results.

Scale. The final curated dataset contains 943 videos that yield over 1,700 question-answer pairs. Since not every video supports all question types (some videos may lack clear relative distance changes, others may not have orientation-annotatable objects), the number of QA pairs per video varies. The paper does not report the exact distribution of QA pairs per video, but the total of 1,700+ pairs across 943 videos averages to approximately 1.8 questions per video.

Motion Annotation and Question Generation

With the standardized video dataset in hand, the paper constructs VQA pairs through a process that moves from manual motion annotation to template-based question generation to human refinement.

Motion annotation protocol. Human experts manually annotate each video with the motion patterns of both the observer (camera) and the observed object(s) within the 3D scene. The annotation captures:

For the observer: whether it is translating (and in which direction), rotating (and in which sense — clockwise vs. counterclockwise), combining translation and rotation, or stationary.
For each annotated object: the same motion-type classification, plus orientation information (what direction the object faces at the start of the video).

These annotations serve as the ground truth for the benchmark. Because they are produced by human experts rather than automated algorithms, they represent the intended "correct" understanding of each video's dynamics. The paper does not report inter-annotator agreement metrics, but the subsequent human review and refinement step (see below) functions as a quality control mechanism.

Three-entity task taxonomy. The paper organizes all VQA tasks around three fundamental entities — the observer, the observed object, and the scene — and their pairwise relationships. This produces three task categories:

Object-Scene tasks: questions about spatial relationships between objects and the surrounding scene (e.g., "Is the object moving toward or away from the background wall?"). These tasks are evaluated separately for cases where the observer is moving versus stationary, enabling the paper to measure how observer motion interferes with object-scene relationship perception.
Observer-Scene tasks: questions about changes in the observer's 3D pose relative to the environment (e.g., "Is the camera moving forward or rotating in place?"). These are similarly evaluated under both dynamic and static object conditions.
Observer-Object tasks: questions about the relative relationship between observer and object (e.g., "Is the distance between the camera and the object increasing or decreasing?"; "Is the object rotating clockwise or counterclockwise relative to the camera?"). These tasks directly test the decoupled motion understanding that the paper identifies as the core challenge of dynamic spatial intelligence.

Figure 2 (left panel) displays the distribution of questions across these three categories. The paper does not provide exact percentages in the text, but the visual proportions in the figure show a roughly balanced split with Observer-Object tasks occupying the largest share — consistent with the paper's emphasis on decoupled motion understanding as the primary evaluation target.

Template-based question generation. Using the motion annotations as input, the paper applies a template-based approach to construct VQA pairs. This means that for each annotation pattern (e.g., "observer translates forward, object rotates clockwise"), there exists a pre-defined question template and a set of answer options that are populated with the specific entities from the video.

For Cam-Scene and Obj-Scene VQA pairs, the templates directly reference the annotated motion types. For example, a video with observer forward translation might generate: "What type of motion is the camera undergoing? A. Translating forward B. Translating backward C. Rotating clockwise D. Rotating counterclockwise" — with the correct answer determined by the annotation.

For a subset of videos, human experts additionally annotate relative distance changes between observer and object, enabling the construction of relative-distance VQA pairs (e.g., "Is the distance between the camera and the [object] increasing or decreasing?").

For all observed objects, orientation information is annotated (e.g., "the car faces left at the start of the video"), enabling relative-orientation VQA pairs (e.g., "Is the [object] rotating clockwise or counterclockwise relative to its initial orientation?").

Reference frame convention. A critical design choice addresses the ambiguity that arises in dynamic scenes: when both observer and object move, what is the reference point for questions about position, distance, or orientation? The paper follows the convention established by prior work (VGGT, SpatialTrackerV2) and fixes the 3D reference point to the initial pose of either the observer or the observed object in each video.

This means that questions like "Is the camera moving forward?" are evaluated against the camera's starting position and orientation, not against some moving reference. Similarly, "Is the object rotating clockwise?" is evaluated against the object's initial orientation. This convention is essential for unambiguous evaluation — without a fixed reference frame, the same video could support contradictory "correct" answers depending on whether one considers motion relative to the scene, relative to the other moving entity, or relative to some arbitrary world coordinate system.

Human expert review and refinement. All template-generated VQA pairs undergo human review, filtering, and refinement. The paper states this step is designed to "ensure clarity and eliminate ambiguity." This is a critical quality control measure because template-based generation can produce questions that are technically correct according to the annotations but confusing or misleading to a human (or model) viewer — for example, a question about observer motion in a video where the dominant visual signal is object motion, making it genuinely hard to perceive the observer's movement.

The human review step also catches cases where the template populated answer options with values that don't match the video content (e.g., a template might generate "rotating clockwise" as an option when the actual motion is a combination of translation and rotation, making the simplified rotation-only option misleading even if technically incorrect). By filtering and refining, the human experts ensure that each VQA pair tests the intended capability without introducing confounds from poorly worded questions or ambiguous answer choices.

Spatio-Temporal Flip Augmentation

This is the methodological heart of DSI-Bench's bias-detection capability. The augmentation serves a dual purpose: it balances the motion-type distribution in the dataset (mitigating the forward-motion overrepresentation that the paper identifies as a source of model bias) and it creates paired test cases where a model with genuine spatial understanding should produce consistent, appropriately transformed answers, while a model relying on superficial heuristics will produce inconsistent answers.

The four-variant generation process. For each original video in the benchmark, the paper produces exactly four variants through two binary transformations applied in combination:

Horizontal flip: the video is mirrored left-to-right. Visually, this means that objects on the left of the frame now appear on the right, and motion direction — importantly — is mirrored. An observer moving left in the original now appears to move right in the flipped version.
Temporal reversal: the video is played backwards. The first frame becomes the last, and the last becomes the first. Motion trajectories are reversed: an observer that moved toward a building now appears to move away from it; an object that rotated clockwise now rotates counterclockwise.

These two transformations are independent and commutative, producing four variants:

Standard: the original video, unmodified.
Horizontal flip: mirrored left-to-right.
Reverse: played backwards in time.
Reverse + horizontal flip: both transformations applied (order does not matter since they commute).

The paper states this design is inspired by 3DSR-Bench's Evalflip strategy (Ma et al., 2024) and Ori-Bench's orientation analysis (Wang et al., 2024b). The innovation in DSI-Bench is extending this methodology from static images to dynamic videos by adding the temporal reversal dimension — static benchmarks can only flip spatially, but DSI-Bench can also reverse temporally, exposing a new class of temporal biases.

Rule-based answer-option updating. For each of the four variants, the questions remain unchanged, but the answer options are symmetrically adjusted using rule-based substitution to maintain correct ground-truth labels. The paper provides concrete examples:

After temporal reversal: "moving forward" → "moving backward" (because playing the video in reverse flips the direction of motion).
After horizontal flipping: "rotating clockwise" → "rotating counterclockwise" (because mirroring the video flips the apparent rotation direction).

This rule-based updating is applied automatically wherever the mapping is deterministic. The key insight is that a model with genuine spatial understanding — one that actually perceives the motion in the video rather than guessing based on the question text or semantic priors — should produce appropriately transformed answers across the four variants. For example, if the correct answer for the standard video is "moving forward," the correct answer for the reversed video should be "moving backward," and a model that says "moving forward" for both is exploiting a forward-motion bias rather than perceiving the video content.

Manual correction for reference-frame shifts. The paper acknowledges a subtle complication: temporal reversal shifts the reference frame from the first frame of the standard video to the last frame of the standard video. This means that for some questions — particularly those involving relative position or orientation — the rule-based substitution alone cannot produce the correct answer options because the reference point has changed in a way that depends on the specific video content.

The paper handles these cases through manual inspection and correction by human experts. The exact proportion of VQA pairs requiring manual correction is not reported, but the acknowledgment is important: it signals that the paper recognizes a genuine ambiguity in its augmentation methodology and addresses it through human judgment rather than ignoring it or applying an incorrect rule.

What the four variants enable analytically. The four-variant design directly supports the group-wise evaluation metric (described below). If a model answers correctly on the standard video but incorrectly on one or more of the flip/reverse variants, the most likely explanation is that the model is not performing genuine spatial reasoning — it is exploiting a bias, a heuristic, or a statistical regularity in the training data. The specific pattern of failures can reveal which bias the model is exploiting:

Correct on standard, incorrect on horizontal flip → left/right spatial bias (the model relies on absolute frame position rather than relative motion direction).
Correct on standard, incorrect on reverse → forward-motion temporal bias (the model defaults to "forward" regardless of actual motion direction).
Correct on standard and horizontal flip, incorrect on reverse and reverse+flip → the model correctly perceives spatial direction but fails at temporal direction (can tell left from right but not forward from backward in time).
Incorrect on all variants equally → the model genuinely cannot perform the task at this difficulty level (not a bias problem, a capability problem).

This analytical power — the ability to diagnose why a model fails rather than just that it fails — is the primary methodological contribution of the spatio-temporal augmentation design.

Evaluation Protocols for VLMs

The paper evaluates VLMs under carefully controlled settings that are designed to isolate the effect of reasoning on dynamic spatial task performance and to enable comparison across model families and scales.

Two answering modes. All VLMs are evaluated under two distinct protocols, adapted from the MindCube protocol (Yin et al., 2025):

RAWQA (Direct Answering): the model receives the video clip, the question, and the answer options, and must output only the final answer with no intermediate reasoning, explanation, or commentary. The system prompt (reproduced in Appendix A.2) explicitly instructs: "You only need to provide ONE correct answer selecting from the options listed below... your response should only be '<answer>A</answer>'." The temperature is set to 0 for deterministic output. This mode measures the model's direct perceptual capability — its ability to extract the correct answer from the visual evidence without the confounding influence of explicit language-based reasoning.
FFR (Free-Form Reasoning): the model receives the same input but must "provide your reasons step by step in details, then provide ONE correct answer selecting from the options." The system prompt instructs the model to output a thinking block containing its reasoning, followed by a response block with the final <answer> tag. This mode measures whether explicit reasoning improves dynamic spatial task performance — that is, whether the model can think its way to the correct answer even when its initial perceptual impression might be wrong.

The comparison between RAWQA and FFR accuracy (Table 2) is one of the paper's key experimental results: if FFR substantially outperforms RAWQA, it suggests that the visual encoder does capture sufficient information and the bottleneck is in downstream processing; if FFR provides minimal or negative benefit, it suggests that the visual encoder fails to extract the necessary spatial information in the first place, and no amount of language-based reasoning can compensate.

Sampling and generation parameters. To ensure fair comparison across models, the paper standardizes several evaluation parameters:

Temperature: 0 (deterministic decoding). This eliminates sampling variance as a confound in accuracy measurements.
Maximum output length: 2,048 tokens. This is generous enough to accommodate extended chain-of-thought reasoning in FFR mode without truncation.
Video sampling rate: 5 fps. For a standardized 3-second video, this yields 15 frames as input to the VLM. The choice of 5 fps balances temporal resolution (enough frames to perceive motion) against computational cost (not so many frames that context length or inference time become prohibitive).

Answer format enforcement. The system prompts (reproduced verbatim in Appendix A.2) enforce a strict XML-style output format:

RAWQA: <answer>[A, B, C, or D]</answer>
FFR: thinking[reasoning text] response <answer>[A, B, C, or D]</answer>

This structured format enables reliable automatic answer extraction without requiring an external language model or parser. The paper measures accuracy by directly comparing the extracted answer letter with the ground truth, with no intermediate grading model.

Model selection rationale. The paper evaluates 14 models spanning four categories:

Proprietary VLMs: Nova-pro-v1 (Amazon, 2025), GPT-4o (OpenAI, 2025a), GPT-5 (OpenAI, 2025b), Gemini-2.5-Pro (Comanici et al., 2025), Seed-1.6, Seed-1.6-vision (ByteDance, 2025). These represent the current state-of-the-art in commercial VLM performance and provide an upper bound on what is achievable.
Open-source VLMs (Qwen2.5-VL series): QwenVL2.5-32B and QwenVL2.5-72B (Bai et al., 2025). These represent the leading open-source VLM family and enable the within-architecture scaling analysis (32B vs. 72B).
Open-source VLMs (InternVL3.5 series): InternVL3.5-8B, InternVL3.5-30B-A3B, InternVL3.5-38B, and InternVL3.5-241B-A28B (Wang et al., 2025b). This family includes both standard dense models and mixture-of-experts architectures (the -A3B and -A28B suffixes indicate the number of activated parameters per token), providing additional within-family scaling data.
3D expertise models: VGGT (Wang et al., 2025a) and SpatialTrackerV2 (Xiao et al., 2025). These are not VLMs in the traditional sense — they do not answer natural language questions. Instead, they produce geometric outputs (camera poses, 3D point trajectories) that are mapped to answer choices via a rule-based procedure (described below).

The inclusion of four InternVL3.5 variants with different scales and architectures enables the paper to test whether scaling laws observed in general VLM benchmarks (bigger model → better performance) hold for dynamic spatial reasoning or whether fundamentally different architectural properties are required.

Evaluation Protocols for 3D Expertise Models

Unlike VLMs, which are evaluated end-to-end (video in, answer letter out), the 3D expertise models require a multi-stage pipeline that extracts geometric outputs and maps them to the discrete answer choices of DSI-Bench.

Orientation calibration and mask refinement. Before evaluation, the paper performs two calibration steps:

Orientation calibration: human experts manually calibrate the orientations of observed objects in the videos. This involves specifying the initial pose of each annotated object so that the expertise model's orientation estimates can be compared against the ground-truth annotation. Because orientation reasoning depends on this calibration, the paper removes benchmark samples that specifically test orientation reasoning (i.e., VQA pairs where the answer depends on knowing whether the object is rotating clockwise vs. counterclockwise) — presumably because the manual calibration process introduces some ambiguity or unreliability that would make those evaluations unfair to the expertise models.
Mask refinement: the paper uses Segment Anything to produce initial masks for observed objects, then refines these masks to align keypoints with the correct objects. This step is necessary because 3D expertise models like SpatialTrackerV2 track specific points in 3D space; if the tracked keypoints fall on background pixels or on the wrong object, the resulting trajectory will not correspond to the intended observed object. The refinement process — the details of which are not specified — ensures that the expertise models are evaluated on their ability to track the correct entities.

Trajectory and pose extraction. After calibration, the expertise models process each video through their native inference pipelines:

VGGT combines depth estimation, keypoint tracking, and camera pose estimation to reconstruct the 3D scene and estimate the observer's (camera's) trajectory through it.
SpatialTrackerV2 performs joint reasoning across scene geometry, camera poses, and object motion to track dynamic object trajectories in 3D.

For both models, the paper extracts the predicted trajectories of the observer and the observed objects — sequences of 3D positions and orientations over time. These trajectories are the geometric equivalent of the motion annotations that human experts produced for the benchmark's ground truth.

Rule-based trajectory-to-answer mapping. The final step maps the continuous trajectory estimates to the discrete answer choices of the VQA pairs. The paper describes this as a "rule-based procedure" without providing the specific rules. Presumably, for a question like "Is the camera moving forward or rotating in place?", the procedure checks whether the observer's trajectory shows significant translational displacement (→ "moving forward") or primarily angular change without displacement (→ "rotating in place"). The thresholds for "significant" displacement or angular change are not specified, but they are applied consistently across all videos.

This mapping step is a potential source of error: if the rule-based procedure misinterprets the trajectory (e.g., classifying a slow translation as rotation because the displacement falls below an arbitrary threshold), the expertise model could appear to fail on a question that it actually got right in geometric terms. The paper does not discuss the calibration or validation of this mapping procedure, which is a notable gap — the reported performance of expertise models is a joint function of their geometric accuracy and the quality of the rule-based answer mapper.

Two-Tier Scoring System

The paper evaluates all models using two complementary accuracy metrics that together distinguish superficial correctness from robust understanding.

Sample-wise accuracy. Each of the four augmented video variants (standard, horizontal flip, reverse, reverse+horizontal flip) is treated as an independent test sample. Accuracy is computed as the fraction of all variant-question pairs for which the model selects the correct answer:

$\text{Accuracy}_{\text{sample-wise}} = \frac{1}{4N} \sum_{i=1}^{N} \sum_{v \in \{\text{std}, \text{flip}, \text{rev}, \text{rev+flip}\}} \mathbb{1}[\text{model}(q_i, v_i^{(v)}) = a_i^{(v)}]$

where $N$ is the number of original video-question pairs, $q_i$ is the $i$ -th question (unchanged across variants), $v_i^{(v)}$ is the $v$ -th video variant for the $i$ -th video, $a_i^{(v)}$ is the ground-truth answer for that variant (transformed by the rule-based substitution), and $\mathbb{1}[\cdot]$ is the indicator function that equals 1 when the model's predicted answer matches the ground truth and 0 otherwise.

What it computes: the fraction of all test instances — across all videos, all questions, and all four augmented variants — on which the model correctly identifies the answer. This is the standard accuracy metric used in most benchmarks, and it measures average performance without distinguishing between a model that is consistently correct on all four variants of a video and a model that succeeds on some variants but fails on others.

Group-wise accuracy. The four augmented variants of each original video-question pair are grouped into a single test instance. The group is counted as correct only if the model answers correctly on at least 3 of the 4 variants:

$\text{Accuracy}_{\text{group-wise}} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}\left[ \sum_{v \in \{\text{std}, \text{flip}, \text{rev}, \text{rev+flip}\}} \mathbb{1}[\text{model}(q_i, v_i^{(v)}) = a_i^{(v)}] \geq 3 \right]$

where the inner sum counts the number of correct answers across the four variants for video-question pair $i$ , and the outer indicator returns 1 if this count is at least 3 (i.e., 3 or 4 out of 4 correct).

What it computes: the fraction of original video-question pairs for which the model demonstrates consistent understanding across spatial and temporal perturbations. A model that gets 2 of 4 variants correct — possibly by exploiting biases that work on some variants but not others — receives no credit for that group under this metric.

Why this form: the 3-of-4 threshold is a deliberate choice that balances strictness against practical tolerance. Requiring 4-of-4 would penalize models for single errors that might arise from annotation noise, ambiguous motion, or genuine borderline cases — a standard that even human annotators might not meet with perfect consistency. Requiring 2-of-4 would be too lenient — a model guessing randomly would achieve 50% accuracy on 4-option multiple-choice questions, which would produce some groups passing by chance. The 3-of-4 threshold hits a middle ground: it demands better-than-chance consistency while acknowledging that perfect consistency across all four transformations is an extremely high bar.

What the gap between sample-wise and group-wise reveals. When sample-wise accuracy is substantially higher than group-wise accuracy, it means the model is inconsistent across the four variants — it sometimes gets the right answer on the standard video but fails on the flipped or reversed versions. This pattern is a signature of brittle heuristics rather than robust spatial understanding. A model that genuinely perceives motion direction would recognize that horizontal flipping reverses left-right motion and temporal reversal reverses forward-backward motion, and would adjust its answers accordingly. A model operating on heuristics (e.g., "most things move forward, so answer 'forward'") would produce the same answer regardless of the augmentation, succeeding on some variants (where "forward" happens to be correct) and failing on others (where the augmentation makes "backward" the correct answer).

The paper observes (Table 1, comparing upper and lower halves) that VLMs show substantial drops from sample-wise to group-wise accuracy, while 3D expertise models like SpatialTrackerV2 show "minimal degradation." This is direct evidence that VLMs rely on brittle pattern matching while expertise models encode more robust geometric understanding — validating the two-tier scoring design as a diagnostic tool.

Equation handling note. The equations are not provided explicitly in the paper — I have constructed them from the textual description of the metrics to make the computation explicit. This is standard practice when the paper describes a scoring procedure in prose rather than formal notation. If the original paper contains specific mathematical definitions that differ from these formulations, those should take precedence.

4. Key Insights and Innovations

Innovation 1: Dynamic Spatial Intelligence as a Distinct Cognitive Category, Not "Static Spatial Reasoning Plus Time"

The paper's most fundamental intellectual move is defining Dynamic Spatial Intelligence as a capability that is qualitatively different from static spatial reasoning, not an incremental extension of it. Before this work, the implicit assumption in the VLM evaluation community was that spatial reasoning benchmarks — VSI-Bench (Yang et al., 2025a), MMSI-Bench (Yang et al., 2025b), VLM4D (Zhou et al., 2025) — could stand as proxies for spatial understanding in general, with dynamic scenarios representing a harder-but-fundamentally-similar task that better static models would naturally handle. The paper demolishes this assumption with direct empirical evidence.

What makes this insight distinctive is not the observation that "moving scenes are harder" — that would be unsurprising — but rather the specific cognitive demand that makes them harder: the requirement to decouple two independent, simultaneous motion streams (the observer's and the object's) and attribute visual change to the correct source. This is not a harder version of a known skill; it is a different skill entirely. A model that correctly identifies "the chair is to the left of the table" in a static image is solving a single-frame spatial relationship problem. That same model, when shown a video where the camera pans right while the table slides left, must perform a fundamentally different computation: it must simultaneously estimate camera ego-motion, track object motion, and compute the relative change by factoring out one from the other. The paper's three-entity task taxonomy — Object-Scene, Observer-Scene, Observer-Object — operationalizes this distinction by creating question categories that can only be answered correctly by disentangling these motion sources.

The significance extends beyond benchmark design to how the field conceptualizes spatial intelligence in artificial systems. Prior work treated spatial reasoning as a monolithic capability: a model either "understands space" or doesn't. DSI-Bench's task decomposition reveals that spatial understanding fractionates into distinct sub-capabilities that do not compose automatically. A model can score well on Observer-Scene tasks (estimating its own motion) and Object-Scene tasks (estimating object motion against a static background) yet fail catastrophically on Observer-Object tasks (estimating the relative relationship between two moving entities). This implies that current VLM architectures learn correlated motion features from training data but never develop the explicit factorization that decoupled understanding requires.

The paper does not claim to solve this factorization — it provides no new training method, no architectural innovation, no decoupling module. What it provides is a diagnostic framework that makes the factorization failure visible and measurable. This is a conceptual contribution of the kind that restructures a research agenda: rather than asking "how can we make VLMs better at spatial reasoning?", the field should now ask "how can we make VLMs decouple self-motion from object motion?" The former question invites incremental improvements to existing architectures; the latter may require fundamentally different inductive biases. By naming and taxonomizing the capability gap rather than proposing a solution, the paper creates an intellectual target that future work can aim at. This is a fundamental reframing, not an incremental refinement.

The evidence anchoring this claim appears most directly in Figure 5, where the static-to-dynamic performance drop is broken out by task type, and in Figure 8, where the coupled motion reasoning failures are visualized. Seed-1.6-vision's 11.55% drop in observer motion perception when objects are dynamic (vs. static) is not just a quantitative degradation — it is a qualitative signature that the model's observer-motion estimation pathway is contaminated by object-motion signals that should be factored out. This is an empirical demonstration that static and dynamic spatial reasoning engage different underlying mechanisms, supporting the paper's core conceptual claim.

Innovation 2: Spatio-Temporal Symmetry as a Bias Diagnostic for Dynamic Scenes

The paper extends a methodology — perturbation-based bias detection — from static images to dynamic video in a way that is conceptually novel rather than merely technically incremental. Prior work (3DSR-Bench by Ma et al., 2024; Ori-Bench by Wang et al., 2024c) established that horizontally flipping static images and checking whether model predictions change can reveal orientation hallucinations and semantic biases. This is a spatial perturbation: the image content is mirrored, so a model with genuine spatial understanding should produce correspondingly mirrored predictions. If it doesn't, it's exploiting non-spatial heuristics (e.g., "cars usually face right in ImageNet-style photos").

DSI-Bench adds temporal reversal as a second perturbation dimension, creating a 2×2 grid of four variants per video: standard, horizontal flip, temporal reverse, and reverse+flip. This is not just "spatial flipping plus temporal flipping" — it is a compositional diagnostic where the interaction between spatial and temporal transformations reveals distinct failure modes that neither transformation alone could expose. Specifically:

A model that correctly answers on standard and horizontal flip but fails on reverse and reverse+flip demonstrates intact spatial perception but broken temporal perception — it can tell left from right but not forward from backward in time.
A model that fails on horizontal flip but succeeds on reverse demonstrates a left/right spatial bias that is independent of temporal direction.
A model that exhibits a specific answer bias (e.g., always selecting "forward") will succeed on variants where that bias happens to align with the correct answer and fail where it doesn't, producing a distinctive error signature across the four variants.

This compositional diagnostic capability is the conceptual innovation. The four-variant design does not just increase the number of test instances — it creates a structured error space where specific failure patterns map to specific underlying biases. The forward-motion bias documented in Figure 6 is not simply "models prefer the answer 'forward'" — it is revealed by the fact that models select "forward" far more often than the ground-truth distribution, and this over-selection persists across horizontally flipped and temporally reversed variants where "forward" is sometimes correct and sometimes incorrect. The inconsistency manifests as a large sample-wise-to-group-wise accuracy gap (Table 1), which the two-tier scoring system makes quantitative.

The significance of this innovation lies in what it enables analytically. Without symmetric perturbations, a benchmark can report that "Model X achieves 62% accuracy on dynamic spatial reasoning." With symmetric perturbations and group-wise scoring, DSI-Bench can report that "Model X achieves 62% sample-wise accuracy but only 38% group-wise accuracy, indicating that roughly 24 percentage points of its performance comes from brittle heuristics rather than robust spatial understanding." The gap between these two numbers — which the paper shows is substantial for VLMs and minimal for 3D expertise models — is a new metric that benchmarks lacking symmetric perturbation design cannot produce. It quantifies not just what models get right, but how they get it right: through understanding or through exploitation.

This is a methodological innovation of the kind that changes evaluation standards. Just as adversarial evaluation (Jia and Liang, 2017) and contrast sets (Gardner et al., 2020) reshaped NLP benchmarking by revealing that high leaderboard scores often reflected dataset artifacts rather than genuine capability, DSI-Bench's spatio-temporal symmetry design reshapes spatial intelligence evaluation by revealing that high sample-wise accuracy often reflects motion-pattern biases rather than dynamic understanding. The methodology is replicable: other benchmarks can adopt spatio-temporal perturbations to add robustness diagnostics to their own evaluations. The paper explicitly credits 3DSR-Bench and Ori-Bench as inspiration but advances the methodology from static to dynamic and from single-axis to compositional perturbation — a fundamental extension rather than a minor variant.

Innovation 3: The Forward-Motion Bias as a Systematically Documented Visual-Linguistic Prior

The paper's identification and quantification of forward-motion bias — the tendency of VLMs to select "forward" as the answer to motion-direction questions far more frequently than the ground-truth distribution warrants — is a specific empirical finding that reveals a deep structural problem in how current VLMs process dynamic visual information.

This is not merely an observation that models make a particular kind of mistake. The paper's analysis shows that this bias is systematic, large in magnitude, and resistant to reasoning. Figure 6 quantifies it: the proportion of "forward" selections by models far exceeds the true proportion of ground-truth answers containing "forward." The paper provides a concrete example (the "fixed statues" case in Figure 7) where this bias directly causes multimodal hallucination — the model "sees" forward motion in a video where no motion exists because "forward" is the default motion prediction that the language model component imposes on ambiguous visual evidence.

What makes this finding intellectually distinctive is its implications for how we understand VLM failure modes. The paper hypothesizes that this bias originates from imbalanced training data distributions — "animals, characters, vehicles, and similar entities" predominantly move forward in visual datasets (Section 5.1). This is a visual-linguistic prior: a statistical regularity in training data that becomes an inappropriate default in evaluation. The model is not "confused" about what it sees; it is applying a reasonable statistical inference ("most things in training data moved forward, so this thing probably moves forward too") in a setting where that inference is wrong. This is fundamentally different from a perceptual failure — the model can see the video perfectly well but overrides its visual evidence with its language-model prior.

The connection to the broader hallucination literature is clear but the finding advances it in a specific direction. Prior work on VLM hallucinations has largely focused on object presence (models describing objects that aren't in the image) and attribute binding (models incorrectly associating properties with the wrong object). The forward-motion bias is a temporal hallucination: the model hallucinates a dynamic property (motion direction) that contradicts the visual evidence. This category of hallucination was invisible to static-image benchmarks and represents a new axis along which VLM reliability must be evaluated for any deployment involving video understanding.

The finding also carries a concrete implication for training: mitigating this bias likely requires explicit balancing of motion directions in pretraining data, not just architectural improvements or better prompting. The paper's free-form reasoning results (Table 2) show that reasoning does not fix this bias — Gemini-2.5-Pro and InternVL3.5-241B actually perform worse under reasoning mode, suggesting that when models articulate their reasoning, they double down on their statistical priors rather than correcting them. This is strong evidence that the bias lives in the visual encoder's feature extraction, not in downstream reasoning, and that language-based debiasing (chain-of-thought, self-consistency) will be insufficient. The finding is incremental in its empirical specifics — it documents one particular bias — but fundamental in its implications: it suggests that current VLM training pipelines systematically embed spurious motion priors that no amount of test-time intervention can fully correct.

Innovation 4: The Robustness-Accuracy Tradeoff Under Model Scaling

The paper's observation that larger models within the same architecture family achieve higher sample-wise accuracy but lower group-wise accuracy (i.e., worse robustness) is a counterintuitive finding that challenges the dominant "scale is all you need" narrative in VLM research. Specifically, QwenVL2.5-72B outperforms QwenVL2.5-32B by 2.8% in sample-wise accuracy but underperforms it in group-wise accuracy. The same pattern holds across InternVL3.5 variants: InternVL3.5-38B beats the 8B version by 2.69% sample-wise but loses on group-wise, and InternVL3.5-241B-A28B beats InternVL3.5-30B-A3B by 2.35% sample-wise but again loses on group-wise (Table 1).

This is not a marginal or noisy result — it is a consistent pattern across two model families and multiple scale comparisons. And it is not what scaling laws would predict. The standard expectation, derived from pretraining scaling literature (Kaplan et al., 2020; Hoffmann et al., 2022) and validated on most VLM benchmarks, is that larger models are strictly better: they achieve higher accuracy, better calibration, and greater robustness to distribution shift. DSI-Bench reveals a setting where this monotonic relationship reverses on a specific metric: larger models become more brittle even as they become more accurate.

The intellectual significance of this finding lies in what it reveals about what scaling actually improves. The paper hypothesizes that "larger models enhance perceptual ability, they do not eliminate inherent biases in spatial perception and reasoning patterns" (Section 4.2). In other words, scaling improves the model's ability to extract fine-grained visual details — which boosts sample-wise accuracy by enabling correct answers on cases where the visual signal is subtle — but simultaneously amplifies the confidence of spurious heuristics that the model has learned from imbalanced training data. A larger model with stronger forward-motion bias will answer "forward" more consistently, scoring correctly on all variants where forward happens to be the right answer (boosting sample-wise accuracy) but incorrectly on all variants where augmentation makes backward the right answer (destroying group-wise accuracy). The bias becomes more entrenched, not less, with scale.

This is a negative result with positive implications. It tells the field that scaling model parameters is not the path to robust dynamic spatial intelligence — at least not within current architectures and training paradigms. Some other intervention is needed: balanced training data, architectural inductive biases for motion decoupling, explicit symmetry constraints, or test-time consistency enforcement. The finding provides a concrete counterexample to the claim that "bigger models fix everything" and identifies dynamic spatial reasoning as a domain where scale amplifies rather than resolves fundamental capability gaps.

The finding also validates the paper's two-tier scoring methodology. If DSI-Bench used only sample-wise accuracy — as most benchmarks do — the conclusion would be "larger models are better at dynamic spatial reasoning, consistent with scaling laws." The group-wise metric reveals the opposite conclusion. This means that the diagnostic value of the group-wise metric is not just that it's a "stricter" accuracy measure — it's that it captures a qualitatively different property of model behavior (consistency under symmetric perturbation) that diverges from raw accuracy under scaling. This methodological contribution — demonstrating that a robustness-focused metric can invert the scaling trend — is itself a significant insight for benchmark design beyond the specific domain of spatial reasoning.

The evidence is clear in Table 1 and the accompanying analysis, but the paper does not fully explore the mechanistic explanation for why this reversal occurs. It offers the hypothesis about amplified biases but does not, for example, decompose the sample-wise accuracy of larger models into "cases solved through genuine understanding" versus "cases solved through amplified heuristics." This is a limitation of the current analysis that future work can address — but the identification of the phenomenon itself is a substantive contribution.

5. Experimental Analysis

Evaluation Methodology

Dataset. DSI-Bench comprises 943 dynamic scene videos yielding over 1,700 manually annotated VQA pairs, drawn from CameraBench (Lin et al., 2025), Kinetics-700 (Smaira et al., 2020), SynFMC (Shuai et al., 2025), LLaVA-178K (Zhang et al., 2024), and additional online sources. Videos are standardized to 480p resolution and 3-second duration, with each original video expanded into four augmented variants (standard, horizontal flip, temporal reverse, reverse+horizontal flip) for a total evaluation set of 943 × 4 = 3,772 video instances.
Base models. The paper evaluates 14 models spanning four categories: proprietary VLMs (Nova-pro-v1 from Amazon, GPT-4o, GPT-5, Gemini-2.5-Pro, Seed-1.6, and Seed-1.6-vision from ByteDance), open-source VLMs (Qwen2.5-VL at 32B and 72B scales, InternVL3.5 at 8B, 30B-A3B, 38B, and 241B-A28B scales including mixture-of-experts variants), and 3D expertise models (VGGT and SpatialTrackerV2). These models span the state-of-the-art in commercial, open-source, and specialized geometric approaches to spatial understanding (Table 1).
Metrics. Two accuracy metrics are computed. Sample-wise accuracy treats each of the four augmented video variants as an independent test instance, computing the fraction of all variant-question pairs where the model's selected answer matches ground truth. Group-wise accuracy groups the four variants of each original video-question pair into a single test instance, counting it as correct only if the model answers correctly on at least 3 of the 4 variants. The gap between sample-wise and group-wise accuracy quantifies robustness — a large gap indicates that correct answers arise from brittle heuristics that fail under symmetric perturbation rather than from genuine spatial understanding.
Baselines. The paper does not designate formal baselines in the sense of a "default method" to beat. Instead, it establishes comparative baselines through: (1) static vs. dynamic condition comparison within the same model, where accuracy on questions with stationary observer/object provides the baseline for measuring degradation under motion; (2) RAWQA vs. FFR comparison, where direct answering accuracy provides the baseline for measuring whether free-form reasoning improves performance; (3) sample-wise vs. group-wise comparison, where sample-wise accuracy provides the baseline for measuring robustness; and (4) cross-model comparison, where proprietary VLMs and 3D expertise models serve as upper-bound performance baselines for open-source models.
Generation budget / compute accounting. All VLMs are evaluated with temperature set to 0 (deterministic decoding), maximum output length of 2,048 tokens, and video sampling rate of 5 fps (producing 15 frames per standardized 3-second video). For 3D expertise models, the "budget" is the model's native inference pipeline — VGGT combines depth estimation, keypoint tracking, and pose estimation; SpatialTrackerV2 performs joint geometry-camera-object reasoning — with no generation budget in the VLM sense. The paper does not report FLOPs or inference time for any model.
Cross-validation / statistical protocol. The paper does not employ cross-validation, confidence intervals, or statistical significance testing. The benchmark is a fixed test set of 943 videos with human-annotated ground truth; there is no training or hyperparameter tuning component. This is standard for benchmark papers but means that the reported accuracy numbers should be understood as point estimates without quantification of variance.

Main Quantitative Results

Aggregate Model Performance Under Sample-Wise and Group-Wise Evaluation

Table 1 presents the comprehensive evaluation of all 14 models under both RAWQA (direct answering) and FFR (free-form reasoning) modes across the three task categories (Object-Scene, Observer-Scene, Observer-Object) and an overall average. The sample-wise results appear in the upper half of the table; the group-wise results in the lower half.

Headline sample-wise results (RAWQA mode). Across sample-wise evaluation, proprietary VLMs lead with Gemini-2.5-Pro achieving the highest overall accuracy (the exact number is not quoted in the text but visible in Table 1), followed by GPT-5 and Seed-1.6-vision. Among open-source models, QwenVL2.5-72B leads the dense models, with InternVL3.5-241B-A28B (a mixture-of-experts architecture) achieving competitive performance. The 3D expertise models — VGGT and SpatialTrackerV2 — achieve substantially lower sample-wise accuracy than the leading VLMs, which the paper attributes to the conservative nature of their rule-based trajectory-to-answer mapping rather than necessarily inferior geometric understanding.

Headline group-wise results (RAWQA mode). When evaluated under the stricter group-wise criterion (at least 3 of 4 augmented variants correct), all VLMs exhibit substantial performance degradation. The paper reports (Section 4.2): "By comparing sample-wise and group-wise accuracy, we observe substantial performance drops across all task categories for VLMs." The 3D expertise models, particularly SpatialTrackerV2, show "minimal degradation under group-wise evaluation" — their sample-wise and group-wise scores are nearly identical. This is the paper's central quantitative finding: VLMs achieve their sample-wise accuracy through brittle heuristics that fail under symmetric perturbation, while expertise models, despite lower absolute accuracy, exhibit genuine spatial consistency.

The paper does not quote exact group-wise accuracy numbers for individual models in the main text, relying on Table 1 for the complete data. The gap magnitudes are visually apparent in the table's layout.

Static vs. Dynamic Performance Gap

Figure 5 presents the critical analysis that anchors the paper's central claim: dynamic scenarios are qualitatively harder than static ones, not just a more difficult version of the same task. The figure shows the accuracy difference between static and dynamic conditions broken out by task type.

The paper reports two specific degradation magnitudes (Section 4.2):

"For example, in Seed-1.6-vision, the accuracy in understanding object dynamics decreases by 2.67% when the observer is moving compared with a static observer, while the accuracy in perceiving observer motion drops by as much as 11.55% under dynamic conditions relative to static ones."

This asymmetry — observer motion perception degrades more severely (11.55%) than object motion perception (2.67%) when the non-target entity is in motion — is a non-obvious result. It suggests that object motion acts as a distractor that contaminates observer motion estimation more strongly than the reverse. The paper does not provide this decomposition for other models, making it difficult to assess whether this asymmetric degradation pattern generalizes or is specific to Seed-1.6-vision.

The paper further notes that "most models experience a notable decline in accuracy on tasks that are otherwise equivalent" when moving from static to dynamic conditions, and that "this performance degradation is especially evident in models with relatively advanced spatial reasoning capabilities." This is a surprising pattern: it is not the weakest models that degrade most, but the strongest ones — suggesting that whatever mechanism produces high static spatial reasoning accuracy does not transfer to dynamic scenarios and may in fact interfere.

Effect of Free-Form Reasoning on Accuracy

Table 2 presents the difference between FFR (free-form reasoning with chain-of-thought) and RAWQA (direct answering) accuracy across all models. The headline finding is that reasoning provides marginal and inconsistent benefits.

The paper reports specific numbers (Section 4.2):

"For most models, enabling free-form reasoning leads to only minor and inconsistent improvements. For instance, QwenVL2.5-72B achieves merely a 0.17% increase in overall accuracy under free-form reasoning mode."

More strikingly, several models perform worse under reasoning:

"Some models even perform worse in free-form reasoning: Gemini-2.5-Pro and InternVL3.5-241B exhibit lower accuracy compared with directly producing answers."

The full detailed breakdown of FFR results across all task categories and models appears in Table 3 (Appendix A.1). The paper does not quote specific degradation magnitudes for Gemini-2.5-Pro or InternVL3.5-241B in the main text.

The paper attributes this pattern to the primacy of visual perception over language-based reasoning in dynamic spatial tasks (Section 4.2):

"We attribute this to the fact that current VLMs primarily ground their reasoning on information extracted by the visual encoder. As a result, language-based reasoning alone cannot compensate for errors that originate from inaccurate visual perception."

This is an important negative result because chain-of-thought reasoning has become a standard technique for improving VLM performance on complex tasks. The finding that it provides essentially zero benefit — and sometimes hurts — for dynamic spatial reasoning establishes a boundary condition on the effectiveness of test-time reasoning: when the bottleneck is in the visual encoder's feature extraction, no amount of language-based deliberation can compensate.

Additionally, the paper reports a qualitative observation about reasoning behavior: "In certain cases, models fail to terminate their reasoning process and instead generate incoherent text until reaching the maximum output token limit." This suggests that the dynamic spatial reasoning task can trigger degenerate behavior in the language generation component, possibly because the model cannot construct a coherent narrative from its confused spatial percepts.

Scaling Behavior: Accuracy vs. Robustness

Table 1 enables a within-family scaling analysis by comparing sample-wise and group-wise accuracy across model sizes. The paper reports (Section 4.2):

Sample-wise accuracy scales positively with model size:

"QwenVL2.5-72B surpasses its 32B counterpart by 2.8%, InternVL3.5-38B outperforms the 8B version by 2.69%, and InternVL3.5-241B-A28B exceeds InternVL3.5-30B-A3B by 2.35%."

These are consistent, modest improvements of 2.4–2.8 percentage points with substantial parameter scaling (2.25× for Qwen, 4.75× for InternVL dense, 8× for InternVL MoE in activated parameters).

Group-wise accuracy scales negatively with model size:

"However, group-wise evaluation, which places greater emphasis on robustness, reveals the opposite trend: QwenVL2.5-72B, InternVL3.5-38B, and InternVL3.5-241B-A28B all underperform their smaller counterparts (32B, 8B, and 30B-A3B, respectively)."

The paper does not quote the specific group-wise degradation magnitudes, but the inversion of the scaling trend is the key finding: larger models are more brittle, not less, in their dynamic spatial reasoning. The paper interprets this (Section 4.2):

"This suggests that model size may not be the principal bottleneck for video-based spatial intelligence in current VLMs. While larger models enhance perceptual ability, they do not eliminate inherent biases in spatial perception and reasoning patterns."

This is a concrete counterexample to the "scale solves everything" narrative. In dynamic spatial reasoning, scaling amplifies the confidence of spurious heuristics (forward-motion bias, left/right positional bias) without eliminating the underlying perceptual confusion, producing models that are more often correct on unperturbed videos but more consistently wrong on perturbed ones.

Forward-Motion Bias Quantification

Figure 6 provides the quantitative evidence for the forward-motion bias — the systematic tendency of VLMs to select "forward" as the answer to motion-direction questions at rates far exceeding the ground-truth distribution. The paper reports (Section 5.1):

"Our analysis reveals that the proportion of 'forward' selections by the models far exceeds the true proportion of ground truths containing 'forward,' indicating a strong selection bias."

The exact over-selection ratio varies by model, but the pattern is consistent: every VLM tested exhibits this bias. The paper hypothesizes (Section 5.1):

"We hypothesize that this bias is related to the imbalanced distribution of visual datasets, in which forward motion predominates over other movement patterns for animals, characters, vehicles, and similar entities."

The paper does not provide the specific forward-selection frequencies or ground-truth forward proportions in the text — these are shown in Figure 6's visualization. The absence of exact numbers in the prose makes it difficult to quantify the bias magnitude precisely without consulting the figure.

Confused Rotation-Translation Discrimination

The paper identifies a specific error pattern where VLMs "misinterpret in-place rotations as translations when inferring the observer's motion" (Section 5.1, Figure 7). The qualitative example in Figure 7 shows a model attempting to determine observer motion by reasoning about "which side of the scene enters the field of view," but then failing to distinguish whether the visual change arises from camera translation through space or in-place camera rotation — two physically distinct motions that can produce similar patterns of visual occlusion and flow at the image boundaries.

The paper does not report a quantitative frequency for this error type, characterizing it instead as a "specific type of error" identified through manual examination of model responses and reasoning processes. The paper notes that "in these tasks, 3D expertise models that leverage classical geometric constraints for camera pose estimation achieve superior performance" — a comparison visible in Table 1's Observer-Scene task category, where VGGT and SpatialTrackerV2 outperform many VLMs on observer motion estimation tasks that require distinguishing rotation from translation.

Coupled Motion Reasoning Failures

Figure 8 presents two representative failure patterns in Observer-Object reasoning where VLMs conflate the motion of the observer with the motion of the observed object. The paper identifies two distinct subtypes (Section 5.1):

Substitution error (Figure 8a): When asked about the observer's motion, the model "mistakenly substitutes the orientation and movement of the observed object for that of the observer, effectively assuming that the observer and the object are stationary relative to each other." This is a relative-to-absolute confusion: the model perceives that the observer and object maintain a fixed relative relationship and incorrectly infers that neither is moving in absolute terms.
Overgeneralization error (Figure 8b): The model correctly perceives relative motion between observer and object but "erroneously generalizes [this relative motion] to the reference frame of the scene." That is, the model attributes the relative change entirely to one entity (e.g., "the object is moving") when in fact both entities are moving relative to the scene.

The paper summarizes the core finding (Section 5.1): "We find that current VLMs are unable to independently infer the motion of the observer in 3D space and the motion of the observed object." This is the empirical basis for the paper's central conceptual claim — that decoupled motion understanding is the fundamental missing capability.

The paper does not report quantitative frequencies for these error patterns, characterizing them as "representative patterns" identified through manual examination. The absence of per-error-type statistics limits the ability to assess which failure mode dominates or how they covary across model families.

3D Expertise Model Performance

Table 1 includes results for VGGT and SpatialTrackerV2. The paper reports (Section 4.2):

"The 3D expert models exhibit robust camera pose estimation in dynamic scenes, showing the smallest performance degradation under group-wise symmetry validation."

This means that expertise models maintain consistent accuracy across the four augmented variants — their group-wise scores are close to their sample-wise scores. However, the paper also reports a limitation (Section 5.2):

"Viewpoint changes and foreground occlusions during motion can cause keypoint loss, leading to instability in the bundle adjustment stage. Experimental results further indicate inaccurate estimation of the observer–object relative distance in dynamic scenes, which may indirectly compromise the accuracy of object motion estimation."

The paper does not quote specific accuracy numbers or degradation magnitudes for expertise models in the text, nor does it break out expertise model performance by task category beyond what Table 1 shows. The key analytical point is that expertise models are more consistent but still imperfect — classical geometric constraints provide a partial solution to dynamic spatial reasoning but break down under dual-motion conditions where the fundamental assumption of scene rigidity is violated.

Ablation Studies and Robustness Checks

Spatio-temporal augmentation as a robustness diagnostic: The comparison between sample-wise and group-wise accuracy (Table 1, upper vs. lower halves) functions as the primary robustness check. The consistent gap across all VLMs — and its near-absence in expertise models — demonstrates that VLM performance is substantially driven by brittle heuristics that fail under symmetric perturbation. The paper reports (Section 4.2): "In contrast, 3D expertise models such as SpatialTrackerV2 show minimal degradation under group-wise evaluation."

Horizontal flip alone vs. temporal reversal alone: The paper does not report separate analyses for spatial-only versus temporal-only perturbation effects. The four-variant design enables this decomposition analytically — by comparing standard-to-flip changes (isolating spatial bias) with standard-to-reverse changes (isolating temporal bias) — but the paper does not present these disaggregated results. This is a missed opportunity: such an analysis could reveal whether the dominant source of VLM inconsistency is spatial (left/right bias) or temporal (forward/backward bias), which would have different implications for mitigation strategies.

Task category analysis: Table 1 disaggregates all results by the three task categories (Object-Scene, Observer-Scene, Observer-Object). The paper does not quote per-category numbers in the text, but the table structure enables the reader to identify which task types are most challenging and where the static-to-dynamic degradation is most severe. Observer-Object tasks — which directly test decoupled motion understanding — consistently show the lowest accuracies, consistent with the paper's claim that this is the hardest sub-capability.

Model scale within architecture families: The comparison of QwenVL2.5-32B vs. 72B, InternVL3.5-8B vs. 38B, and InternVL3.5-30B-A3B vs. 241B-A28B (Table 1) serves as an ablation of model size. The sample-wise improvement but group-wise degradation pattern is consistent across both families, suggesting it is a general property of current VLM architectures rather than an artifact of a specific training pipeline.

RAWQA vs. FFR as a reasoning ablation: Table 2 (and the fuller Table 3 in Appendix A.1) ablates the effect of explicit chain-of-thought reasoning on task performance. The near-zero and sometimes negative benefit demonstrates that reasoning does not compensate for perceptual failures — a finding that would be invisible if only FFR results were reported.

Proprietary vs. open-source models: Table 1 includes both categories, enabling a rough comparison of the performance frontier. The paper does not explicitly discuss this comparison, but the table shows that proprietary models (GPT-5, Gemini-2.5-Pro) achieve higher sample-wise accuracy than open-source alternatives at comparable parameter counts, though the group-wise degradation pattern affects both categories similarly.

Missing ablations: The paper does not report several analyses that would strengthen its robustness assessment: (1) sensitivity to video sampling rate (5 fps was chosen — would 2 fps or 10 fps change results?), (2) sensitivity to video resolution (480p standardized — would 720p improve accuracy?), (3) per-model analysis of which specific motion patterns (the 3×3 observer-object grid) drive failures, (4) inter-annotator agreement on the human motion annotations that serve as ground truth, (5) calibration of 3D expertise model rule-based mapping thresholds, (6) whether the forward-motion bias is stronger for certain object categories (vehicles vs. animals vs. people), and (7) separate spatial-only and temporal-only perturbation analyses.

Critical Assessment

Do the Experiments Support the Claim That Dynamic Spatial Intelligence Is Qualitatively Distinct?

The paper's central conceptual claim is that dynamic spatial reasoning requires decoupled understanding of self-motion and object motion — a capability that is not simply "static spatial reasoning made harder." The experiments partially support this claim, but with important limitations in the evidence.

The static vs. dynamic performance gap (Figure 5) establishes that dynamic conditions reduce accuracy, which is necessary but not sufficient for the "qualitatively distinct" claim — a harder task would also show reduced accuracy without requiring different underlying mechanisms. The paper does not provide the kind of analysis that would distinguish between "dynamic is harder" and "dynamic is different": for example, showing that static spatial reasoning accuracy does not predict dynamic spatial reasoning accuracy across models (a double dissociation), or that specific error types in dynamic conditions have no static analog. The coupled motion reasoning failures (Figure 8) are the strongest evidence for qualitative distinctness because they have no static equivalent — you cannot substitute observer motion for object motion when neither is moving. But these are presented as qualitative examples without quantitative frequency data across the full benchmark, making it difficult to assess whether they represent the dominant failure mode or rare edge cases.

The task taxonomy (Object-Scene, Observer-Scene, Observer-Object) is well-motivated conceptually, but the paper does not demonstrate that Observer-Object tasks — which should require the most decoupled understanding — show disproportionately worse performance than would be predicted from general difficulty scaling. Without this normalization, the lower Observer-Object accuracy could reflect that these questions are simply harder in an absolute sense (more answer options, more ambiguous visual evidence) rather than tapping a fundamentally different cognitive demand.

Do the Experiments Support the Claim That Spatio-Temporal Symmetry Enables Bias Detection?

Yes, with strong evidence. The sample-wise vs. group-wise gap in Table 1 is a clean demonstration that VLM accuracy is substantially driven by heuristics that fail under symmetric perturbation. The fact that this gap is near-zero for 3D expertise models — which use geometric constraints that are inherently symmetric-consistent — provides a validity check: the group-wise metric is not just harder, it specifically penalizes models that lack robust spatial representations.

The forward-motion bias (Figure 6) is clearly documented by comparing model answer distributions to ground-truth distributions, and the spatio-temporal augmentation makes this bias visible by creating test cases where the biased answer is incorrect. The qualitative error examples (Figures 7, 8) provide face validity for the claim that models confuse rotation with translation and conflate observer-object motion.

The key limitation is that the paper does not provide a decomposition of the sample-wise-to-group-wise gap by error type. We can see that VLMs are inconsistent, but we cannot tell from the reported results whether the inconsistency is predominantly due to forward bias, left/right bias, rotation-translation confusion, or coupled motion reasoning failures. This decomposition is analytically possible within the four-variant framework — each error signature maps to a specific bias — but the paper does not perform it. Consequently, the benchmark demonstrates that biases exist without fully characterizing their relative importance.

Do the Experiments Support the Claim That Free-Form Reasoning Provides Marginal Benefit?

Strongly, with quantified evidence. Table 2 shows that QwenVL2.5-72B gains 0.17% from reasoning and that Gemini-2.5-Pro and InternVL3.5-241B perform worse with reasoning than without. This is a clean demonstration that test-time reasoning does not help when the bottleneck is in visual perception rather than logical inference. The full FFR results in Table 3 (Appendix A.1) provide the per-task-category breakdown.

The paper's interpretation — "language-based reasoning alone cannot compensate for errors that originate from inaccurate visual perception" — is consistent with the data but would be strengthened by analyzing the content of the reasoning traces. If models consistently produce logically sound reasoning from incorrect perceptual premises, that supports the interpretation. If they produce incoherent or contradictory reasoning, the bottleneck might be in the reasoning process itself rather than in perception. The paper mentions that "models often rely on commonsense knowledge within the language modality, which sometimes introduces additional biases and hallucinations into visual reasoning" (Section 4.2), but does not provide a systematic analysis of reasoning trace quality.

Do the Experiments Support the Claim That Larger Models Improve Accuracy But Reduce Robustness?

Yes, with consistent evidence across two model families. The pattern — QwenVL2.5-72B beats 32B on sample-wise but loses on group-wise; InternVL3.5-38B beats 8B on sample-wise but loses on group-wise; InternVL3.5-241B-A28B beats 30B-A3B on sample-wise but loses on group-wise — appears in Table 1 and is reported explicitly in Section 4.2.

The causal mechanism for this reversal — the hypothesis that scaling amplifies bias confidence without resolving perceptual confusion — is plausible but not directly tested. The paper does not, for example, show that larger models exhibit stronger forward-motion bias (higher forward-selection rates than smaller models), which would be the expected signature of amplified bias. Without this intermediate evidence, alternative explanations are possible: larger models might simply be more sensitive to the specific perturbation types used (horizontal flipping and temporal reversal being out-of-distribution transformations that larger models are worse at generalizing across), or the group-wise threshold (3 of 4) might interact with model calibration in a way that disadvantages larger models even if they are not more biased per se.

Do the Experiments Support the Claim About 3D Expertise Model Limitations in Dynamic Scenes?

Partially. The paper reports (Section 5.2) that expertise models exhibit "keypoint loss" and "inaccurate estimation of the observer–object relative distance in dynamic scenes," but provides no quantitative evidence for these claims beyond what is visible in Table 1. The expertise models achieve their (lower) accuracy scores in the table, but the paper does not decompose these into "failures caused by dynamic scene breakdown of geometric constraints" versus "failures caused by limitations of the rule-based trajectory-to-answer mapping" versus "failures on genuinely hard examples that no method solves."

The claim that "classical 3D constraints fail to consistently characterize relative pose relationships in continuous dynamic scene videos" (abstract) is stated as a finding, but the evidence for causal attribution to constraint breakdown rather than other factors is thin. The paper would need controlled experiments — comparing expertise model performance on static vs. dynamic versions of the same scene with otherwise identical geometry — to isolate the constraint breakdown effect from general difficulty.

Genuine Weaknesses in the Experimental Design

Single evaluation metric (accuracy) without calibration analysis. The paper reports only accuracy (fraction correct). It does not report calibration error, confidence scores, or any measure of model uncertainty. For a benchmark about bias and hallucination, knowing whether models are confidently wrong or uncertainly wrong would be valuable — forward-motion bias with high confidence indicates a different problem (entrenched prior overriding perception) than forward-motion bias with low confidence (genuine uncertainty resolved by guessing).

No human performance baseline. The paper does not report human accuracy on DSI-Bench, making it impossible to assess whether the observed model failures represent "sub-human performance" or "genuinely difficult tasks." The motion annotation process demonstrates that human experts can correctly label motion patterns, but annotation with unlimited time and focused attention is different from answering VQA questions under time pressure. A human baseline would contextualize the 60–80% sample-wise accuracies reported for top VLMs — are these near human-level or far below?

Small number of questions per fine-grained category. With 1,700+ questions distributed across three task categories, five observer motion types, five object motion types (the taxonomy described in Section 3.1), and four augmented variants, the number of questions in any specific cell (e.g., Observer-Object tasks with observer translating forward and object rotating clockwise) is small — likely single digits. This limits the statistical reliability of fine-grained analysis, which may explain why the paper reports only aggregate results and qualitative error examples rather than per-motion-pattern breakdowns.

No analysis of failure consistency across models. The paper reports per-model accuracies but does not analyze whether different models fail on the same questions (suggesting genuinely ambiguous or difficult examples) or on different questions (suggesting model-specific biases). Item-level agreement analysis would strengthen the benchmark's diagnostic value.

Unvalidated rule-based mapping for expertise models. The paper does not describe how the rule-based procedure that maps expertise model trajectory outputs to discrete answer choices was calibrated or validated. If this mapping introduces systematic errors — for example, by using displacement thresholds that misclassify slow translations as rotations — the reported expertise model accuracies are lower bounds on their true geometric understanding. The paper removed orientation-reasoning samples to avoid calibration issues but does not discuss whether similar issues affect other task categories.

Absence of confidence intervals. With 943 unique videos (not 3,772 independent samples given the correlation among augmented variants), the effective sample size for group-wise accuracy is 943 groups. The paper does not report confidence intervals or standard errors, making it impossible to assess whether the 2.8% gap between QwenVL2.5-72B and 32B is statistically significant or within sampling noise.

Missing experiments that would have strengthened the paper:

Per-motion-pattern accuracy breakdown: mapping the 3×3 observer-object motion grid to accuracy would directly show which decoupled motion patterns are most challenging.
Separate spatial-only and temporal-only perturbation analysis: decomposing the sample-wise-to-group-wise gap into the portion attributable to horizontal flipping failures versus temporal reversal failures.
Confidence calibration analysis: measuring whether models are confidently wrong (strong bias) or uncertainly wrong (guessing) when they fail.
Human performance baseline: establishing the ceiling for DSI-Bench accuracy.
Cross-model error consistency analysis: measuring whether different models fail on the same video-question pairs.
Bias-by-scale analysis: measuring whether forward-motion bias magnitude increases with model size within the same architecture family, directly testing the amplification hypothesis.
Controlled expertise model experiments: comparing VGGT/SpatialTrackerV2 performance on static-only vs. dynamic versions of the same scenes to isolate the effect of geometric constraint breakdown.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Not Accounted for in the Practical Deployment Budget

The assumption or constraint. The entire spatio-temporal symmetry methodology — the four-variant generation, the rule-based answer-option updating, and the manual correction for reference-frame shifts — is described as a benchmark construction technique, not a deployment strategy. The paper does not claim that real-world systems should apply horizontal flipping and temporal reversal at inference time. However, the paper also does not account for the human annotation cost that makes the benchmark possible: motion pattern labeling, question generation, answer-option updating, and manual correction of reference-frame ambiguities all require expert human effort.

This matters because the benchmark's diagnostic value — its ability to distinguish genuine spatial understanding from brittle heuristics — depends on having ground-truth motion annotations that enable the symmetric answer transformation. In a deployment scenario where a VLM answers questions about a novel video, there is no equivalent procedure for testing whether its answer reflects understanding or bias exploitation. The benchmark diagnoses the disease but provides no test that can be applied at inference time.

The consequence. The paper's central finding — that VLMs achieve their sample-wise accuracy through brittle heuristics rather than robust understanding — is revealed through an evaluation methodology that is not replicable in deployment. A system builder who reads the paper and worries that their VLM might exhibit forward-motion bias cannot apply the group-wise evaluation protocol to their own videos without first annotating ground-truth motion for every video, which defeats the purpose. The diagnostic tool is a research instrument, not an operational safeguard.

What evidence exists in the paper. The paper acknowledges the human annotation workload implicitly by describing the benchmark construction process (Section 3.2): human experts "manually annotated the motion patterns," "reviewed, filtered, and refined" VQA pairs, "determined the starting and end points of each video," and "manually inspected and corrected" cases where rule-based substitution failed. The paper does not report the total person-hours required, the number of annotators, or inter-annotator agreement metrics. The throughput — 943 videos, 1,700+ QA pairs, four variants each — represents a substantial annotation investment that is not characterized.

Mitigation status. The paper does not address this limitation. It does not propose a method for automated group-wise consistency checking at deployment time, nor does it discuss whether training-time interventions (balanced motion data, symmetry-aware data augmentation) could reduce the sample-wise-to-group-wise gap without requiring symmetric perturbations at test time. The benchmark establishes that the gap exists and that it is large; it does not provide tools for closing it or detecting it in the wild.

Dynamic Spatial Reasoning Failure on the Hardest Motion Patterns Is Not Characterized

The assumption or constraint. DSI-Bench classifies observer and object motion into three types — translation, rotation, and combination — producing a 3×3 grid of nine decoupled motion patterns. The paper reports results aggregated across all motion patterns (Table 1) and presents qualitative examples of specific failure modes (forward bias, rotation-translation confusion, coupled motion reasoning; Figures 6–8), but it does not provide a per-motion-pattern accuracy breakdown. The paper does not report which of the nine decoupled motion patterns are hardest, which failure modes are most common, or whether certain patterns are essentially unsolved by all models.

The consequence. Without per-pattern results, the paper cannot identify capability boundaries — the specific dynamic scenarios where current models completely fail versus those where they show partial competence. A practitioner deciding whether to deploy a VLM in a dynamic environment (e.g., a drone navigation system, an AR headset) cannot determine from the paper whether their specific motion profile (e.g., "observer rotates while object translates") is within current model capabilities or fundamentally out of reach. The paper reports that "current VLMs are unable to independently infer the motion of the observer in 3D space and the motion of the observed object" (Section 5.1), but this is a binary characterization — "unable" — when the reality is likely graded: some motion patterns may be substantially easier than others.

This limitation is particularly significant because the paper frames its contribution partly as a motion taxonomy — the nine decoupled patterns are presented as a systematic decomposition of the dynamic spatial intelligence problem space. A decomposition that is not analyzed at the level of its own categories is taxonomy without diagnosis.

What evidence exists in the paper. The paper presents qualitative examples: the "fixed statues" case demonstrating forward bias (Figure 7), the cow-rotation example demonstrating rotation-translation confusion (Figure 7), and the two coupled motion reasoning cases (Figure 8). These are illustrative but not systematic. Figure 2 displays the motion distribution of the dataset (how many videos have each motion type), but not accuracy as a function of motion type. Table 1 disaggregates by task category (Object-Scene, Observer-Scene, Observer-Object) but not by the specific motion patterns within each category. The paper's claim that "classical 3D constraints fail to consistently characterize relative pose relationships in continuous dynamic scene videos" (abstract) is stated as a finding about expertise models but is not tied to specific motion patterns where constraint breakdown occurs.

Mitigation status. The paper does not address this limitation or acknowledge it as a gap. The motion taxonomy is presented as a design feature of the benchmark (Section 3.2: "both the observer and the observed objects exhibit three types of motion"), but the analysis does not exploit the taxonomy's granularity. A per-pattern accuracy table — 9 rows (motion combinations) × 3 task categories — would substantially increase the benchmark's diagnostic value and is feasible within the existing dataset, but the paper does not provide it.

The Benchmark Is Restricted to Short Video Clips with Three-Second Duration

The assumption or constraint. Every video in DSI-Bench is standardized to a duration of 3 seconds and sampled at 5 fps, producing 15 frames per clip. The paper states (Section 3.2): "overly short clips were slowed down to a duration of 3 seconds." This means the benchmark evaluates dynamic spatial reasoning on brief, atomic motion segments — a single continuous motion event rather than extended sequences with multiple motion phases, changes in direction, or long-term spatial memory requirements.

The consequence. The 3-second duration means DSI-Bench cannot evaluate several capabilities that are central to real-world dynamic spatial intelligence:

Motion phase transitions: a viewer walking forward for 5 seconds, then turning right and walking for another 5 seconds, produces a fundamentally different spatial reasoning challenge than a single 3-second forward walk. The model must segment the motion trajectory, maintain spatial state across the transition, and reason about the composite path.
Long-term spatial memory: in extended navigation, the observer's position relative to previously seen landmarks must be maintained over tens of seconds or minutes. A 3-second clip requires only immediate perception, not memory.
Acceleration and deceleration: short clips at fixed playback speed capture constant-velocity motion patterns (or their slowed-down equivalents), not the natural variations in speed that characterize real observer and object motion.

The consequence is that DSI-Bench measures a subset of dynamic spatial intelligence — the ability to perceive and classify a single, short motion segment — and should not be interpreted as evaluating the full spectrum of capabilities that the term "dynamic spatial intelligence" implies in natural language. A model that performs well on DSI-Bench might still fail catastrophically on longer, multi-phase dynamic sequences. Conversely, a model that performs poorly on DSI-Bench might handle longer sequences better if its weakness is specifically in rapid motion classification from minimal temporal evidence.

What evidence exists in the paper. The paper explicitly describes the 3-second standardization in Section 3.2 and the 5 fps sampling rate in Section 4.1. The duration choice is motivated practically: short clips ensure that each video contains a single, interpretable motion pattern suitable for unambiguous annotation and evaluation. The paper does not claim to evaluate long-term dynamic spatial reasoning and does not discuss the duration constraint as a limitation. It frames static cases as "a special form of dynamic scenarios" (Section 2.2) but does not similarly frame the 3-second clips as a special (short) form of the broader dynamic reasoning challenge.

Mitigation status. The paper does not address this scope limitation or propose extensions to longer-duration evaluation. The choice of 3 seconds is a practical tradeoff — it enables clean annotation and controlled evaluation at the cost of ecological validity for deployment scenarios involving extended navigation or surveillance — but the paper does not characterize this tradeoff explicitly.

The Benchmark Evaluates Classification, Not Continuous Spatial Reasoning

The assumption or constraint. Every question in DSI-Bench is a multiple-choice classification with a fixed set of answer options. The paper describes six question types (Section 3.1) covering observer motion classification, object motion classification, and relative distance/orientation changes, but all are operationalized as selecting from discrete options (e.g., "A. Translating forward B. Translating backward C. Rotating clockwise D. Rotating counterclockwise"). The benchmark does not include continuous estimation tasks: predicting the exact distance between observer and object, estimating the speed of relative motion, or generating a trajectory in 3D coordinates.

The consequence. The multiple-choice format makes DSI-Bench a perception-plus-classification benchmark, not a spatial reasoning benchmark in the fullest sense. A model that can distinguish "moving forward" from "rotating clockwise" with 80% accuracy may have no ability to answer "how far has the observer moved?" or "at what rate is the distance decreasing?" — questions that are central to spatial intelligence in robotic navigation, autonomous driving, and AR applications. The discrete classification format also creates an artificial ceiling: a model with perfect dynamic spatial understanding but poor multiple-choice test-taking ability (e.g., due to unfamiliarity with the answer format) would underperform on DSI-Bench relative to its true capability.

This design choice also interacts with the bias findings in a potentially confounding way. The paper documents strong forward-motion selection bias (Section 5.1, Figure 6): models disproportionately select answer options containing "forward." In a multiple-choice format, a model with this bias will select "forward" when it is one of the options, regardless of the video content. In a continuous estimation format — e.g., "describe the observer's motion direction in your own words" — the bias might manifest differently (overusing the word "forward" in free-text descriptions) or might be mitigated by the absence of a constrained option set that primes the biased response. The paper's finding that VLMs exhibit forward bias is therefore potentially confounded with the multiple-choice format — the bias may be partly a test-taking artifact rather than a pure perceptual phenomenon.

What evidence exists in the paper. The paper does not discuss this limitation. The template-based question generation approach (Section 3.2) is inherently classification-oriented, and the paper's evaluation metrics (sample-wise and group-wise accuracy) assume discrete answer selection. The paper does not include open-ended or continuous estimation questions, nor does it discuss the relationship between classification accuracy and continuous spatial reasoning capability.

Mitigation status. The paper does not address this scope limitation. The abstract and introduction describe DSI-Bench as evaluating "dynamic spatial intelligence" and "spatial reasoning" without qualifying that the evaluation is restricted to discrete classification. The template-based construction method could in principle be extended to generate continuous estimation questions (e.g., "estimate the observer's displacement in meters"), but this is not explored.

No Human Baseline Establishes the Performance Ceiling

The assumption or constraint. DSI-Bench evaluates 14 computational models — VLMs and 3D expertise systems — but does not report human accuracy on any subset of the benchmark. The paper does not describe a human evaluation protocol, does not report how many human participants were tested, and does not provide human scores for comparison against model scores in Table 1 or Figure 5.

The consequence. Without a human baseline, the reported model accuracies are uninterpretable in absolute terms. When Gemini-2.5-Pro achieves a certain sample-wise accuracy on DSI-Bench (visible in Table 1), we cannot tell whether this represents near-human performance (the task is easy for humans and the model is approaching the ceiling), substantially sub-human performance (humans achieve near-perfect accuracy and the model is far below), or super-human performance (the task is genuinely difficult for humans and the model exceeds human capability). Each interpretation has different implications for deployment readiness and research prioritization.

This limitation is particularly significant given the paper's framing. The introduction argues that dynamic spatial intelligence is "an intuitive ability rooted in basic spatial intelligence" for humans (Section 1), implying that humans should perform well on the benchmark. If humans achieve near-ceiling accuracy, the model errors documented in the paper represent a clear capability gap that demands architectural innovation. If humans also struggle on many DSI-Bench questions — for example, distinguishing camera rotation from translation in ambiguous monocular video is known to be difficult even for humans — then the model errors may reflect a fundamentally hard perceptual problem rather than a model-specific deficiency.

What evidence exists in the paper. Human experts annotated the ground-truth motion patterns (Section 3.2), which establishes that humans with unlimited time and focused attention on annotation can correctly classify the motion in each video. However, annotation performance under ideal conditions does not translate to VQA performance under time pressure — an annotator who carefully studies a video frame-by-frame to label motion type may not achieve the same accuracy when asked to answer a multiple-choice question after watching the video once at normal speed. The paper provides no human VQA accuracy data and does not discuss this distinction.

Mitigation status. The paper does not acknowledge the absence of a human baseline as a limitation. It does not suggest a human evaluation protocol for future work. The claim that dynamic spatial intelligence is "intuitive" for humans (Section 1) is asserted rather than empirically demonstrated on the DSI-Bench task format. A human evaluation on even a 50-video subset would substantially strengthen the benchmark's interpretability and is a standard practice in benchmark papers (e.g., VQA-v2, NLVR2, and many VL benchmarks report human accuracy).

The 3D Expertise Model Evaluation Pipeline Introduces Unmeasured Sources of Error

The assumption or constraint. The evaluation of VGGT and SpatialTrackerV2 on DSI-Bench requires a multi-stage pipeline that is qualitatively different from the end-to-end VLM evaluation. Specifically (Section 4.1): human experts manually calibrate object orientations, the paper removes benchmark samples testing orientation reasoning, masks predicted by Segment Anything are refined to align keypoints with observed objects, and extracted trajectories are mapped to discrete answer choices through a rule-based procedure whose details and calibration are not described.

The paper implicitly assumes that this pipeline faithfully translates the expertise models' geometric outputs into answers that are comparable to VLM answer selections, and that any errors in the expertise models' final answers reflect failures of geometric understanding rather than failures of the pipeline.

The consequence. The reported expertise model accuracies in Table 1 represent a joint function of (1) the model's geometric estimation quality, (2) the accuracy of the Segment Anything masks, (3) the correctness of the manual mask refinement, (4) the appropriateness of the rule-based mapping thresholds, and (5) any residual calibration errors from the orientation annotation. An expertise model that produces a perfectly correct 3D trajectory could still be scored as incorrect if the rule-based mapping misclassifies a slow translation as rotation (because the displacement falls below an arbitrary threshold), or if the Segment Anything mask includes background pixels that contaminate the keypoint trajectory.

This makes the comparison between VLMs and expertise models in Table 1 asymmetric in validity. VLMs are evaluated end-to-end with no intermediate processing — their answers are a direct function of what the model outputs. Expertise models pass through a human-designed pipeline that introduces its own failure modes. The paper's finding that expertise models show "minimal degradation under group-wise evaluation" (Section 4.2) could reflect genuine geometric consistency — or it could reflect that the rule-based mapping is deterministic and therefore produces the same (potentially wrong) answer for all four augmented variants, artificially inflating group-wise accuracy relative to a noisy but unbiased estimator. The paper does not provide the evidence needed to distinguish these interpretations.

What evidence exists in the paper. The paper acknowledges several components of the calibration requirement — removing orientation-reasoning samples (Section 4.1), refining Segment Anything masks, manually calibrating orientations — but does not report validation metrics for any of these steps. The rule-based trajectory-to-answer mapping is mentioned only as "a rule-based procedure" (Section 4.1) without specification of the rules, their thresholds, or their validation against human-judged answers for the same trajectories. The paper reports (Section 5.2) that "experimental results further indicate inaccurate estimation of the observer–object relative distance in dynamic scenes," but does not specify whether this inaccuracy is measured in geometric terms (meters of error in 3D distance estimates) or in classification terms (error in the discrete answer selection), making it unclear whether the problem is in the geometric estimation or the classification mapping.

Mitigation status. The paper does not characterize this as a limitation. The expertise model evaluation is presented as a straightforward component of the benchmark evaluation (Section 4.1: "3D Expertise Model Evaluation Details") without discussion of the failure modes introduced by the pipeline. The paper could mitigate this concern by: (1) reporting expertise model 3D error metrics directly (e.g., mean absolute error in camera translation estimation, in degrees) on a subset of videos where ground-truth 3D motion is available from the synthetic SynFMC source; (2) validating the rule-based mapping against human judgments of the same trajectories; or (3) reporting the sensitivity of expertise model accuracy to the mapping thresholds. None of these validation steps are reported.

7. Implications and Future Directions

How This Work Changes the Landscape

DSI-Bench does not introduce a new model, training method, or architectural innovation. What it introduces is a diagnostic instrument that makes visible a capability gap the field has systematically overlooked: the inability of current vision-language models to decouple self-motion from object motion when both occur simultaneously. This is a reframing contribution rather than a paradigm shift — it does not propose a new theory of spatial intelligence, but it provides the measurement framework that makes the reframing actionable. The paper's impact will be measured by whether future work adopts its motion taxonomy, its spatio-temporal symmetry methodology, and its two-tier accuracy metrics as standard evaluation practices, not by whether anyone builds "the DSI model."

Reconciling a hidden contradiction. Prior to DSI-Bench, the VLM evaluation landscape contained a quiet tension that the paper resolves. On one side, benchmarks like VSI-Bench (Yang et al., 2025a) and MMSI-Bench (Yang et al., 2025b) showed that VLMs possess nontrivial spatial reasoning capabilities — they can identify relative positions, estimate distances, and reason about orientations in static scenes. On the other side, anyone who has deployed VLMs in video settings knows anecdotally that they produce bizarre spatial errors when the camera moves — confusing left turns for right pans, claiming objects are "approaching" when they are receding, and generally failing at the kind of dynamic spatial reasoning that humans perform effortlessly. These two perspectives seemed contradictory: how can models be "good at spatial reasoning" yet fail so conspicuously on dynamic video?

DSI-Bench resolves this by showing that these are not contradictory observations — they measure different capabilities. The static spatial reasoning benchmarks measure a model's ability to extract spatial relationships from a single, fixed geometric configuration. DSI-Bench measures the additional demand of attributing visual change to the correct motion source when two independent motion streams (observer and object) are superimposed. The paper's central finding — that models systematically conflate observer and object motion (Section 5.1, Figure 8) — explains the anecdotal deployment failures as failures of decoupling, not failures of spatial perception per se. A model can correctly answer "the chair is to the left of the table" in a static image while simultaneously being unable to determine whether the chair moved left or the camera moved right when both possibilities produce the same visual change. This reconciles the contradiction by demonstrating that "spatial reasoning" is not a unitary construct — it fractionates into sub-capabilities with different difficulty profiles, and current benchmarks only measure the easier ones.

Shifting the research agenda from scale to structure. Perhaps the paper's most consequential implication is its challenge to the dominant "scale solves everything" narrative in VLM research. The finding that larger models within the same architecture family achieve higher sample-wise accuracy but lower group-wise accuracy (Table 1, Section 4.2) is a concrete counterexample to scaling-as-panacea. QwenVL2.5-72B outperforms its 32B counterpart by 2.8% in raw accuracy yet underperforms it in robustness. InternVL3.5-38B beats 8B by 2.69% on sample-wise but loses on group-wise. The same pattern holds for InternVL3.5's mixture-of-experts scaling.

This is not a marginal or noisy result — it is a consistent inversion across two model families and three scale comparisons. And it carries a specific implication: if scaling model parameters amplifies bias confidence without resolving the underlying perceptual confusion, then the path to robust dynamic spatial intelligence does not run through larger versions of current architectures. Some structural change is needed — architectural inductive biases for motion factorization, training objectives that enforce consistency under symmetric perturbation, or data augmentation strategies that explicitly balance motion directions. The paper does not specify what that structural change should be, but it establishes that continuing to scale current architectures is not the answer, which is a valuable negative result that can redirect research investment.

Making verifier over-optimization a first-class concern in spatial evaluation. The paper's two-tier scoring system — sample-wise accuracy measuring raw performance, group-wise accuracy measuring consistency under symmetric perturbation — introduces a methodological innovation with implications beyond dynamic spatial reasoning. The gap between these two metrics quantifies something that standard benchmarks cannot measure: what fraction of a model's correct answers arise from genuine understanding versus brittle heuristics. A model achieving 70% sample-wise accuracy with only 40% group-wise accuracy (a 30-percentage-point gap) is getting nearly half its "correct" answers through mechanisms that fail under trivial transformations. This is a form of evaluation over-optimization — models learn to exploit statistical regularities in training data (predominantly forward motion, left/right positional biases) that produce correct answers on standard test distributions but collapse under systematic perturbation.

This insight generalizes beyond spatial reasoning. Any domain where training data contains systematic biases — and every domain contains systematic biases — is vulnerable to the same dynamic: models achieve high standard accuracy by learning the bias, benchmark designers report the high accuracy as evidence of capability, and the capability gap remains invisible until someone designs the right perturbation test. DSI-Bench's spatio-temporal symmetry methodology provides a template for constructing such perturbation tests: identify the bias (forward motion overrepresentation), design symmetric transformations that invert the bias (horizontal flip, temporal reversal), and measure consistency across transformations. This template can be applied to action recognition ("raising right hand" vs. "raising left hand" under horizontal flip), navigation instruction following ("turn left at the corner" under map rotation), and temporal reasoning ("before" vs. "after" under video reversal) — a family of diagnostic benchmarks that DSI-Bench's methodology enables but does not itself provide.

What becomes more attractive as a research direction. The paper makes several lines of inquiry newly compelling:

Motion-balanced pretraining data. The forward-motion bias documented in Figure 6 — models selecting "forward" far more often than the ground-truth distribution — is hypothesized to originate from imbalanced training data where forward motion predominates for animals, vehicles, and characters. A direct intervention would be curating or synthesizing video pretraining data where all motion directions (forward, backward, left, right, up, down, and all rotation senses) appear with equal frequency, then measuring whether this reduces both the forward-selection bias and the sample-wise-to-group-wise accuracy gap. The paper provides the measurement tool (DSI-Bench) that makes this experiment evaluable.
Architectural motion factorization. The coupled motion reasoning failures (Figure 8) — where models substitute the observed object's motion for the observer's motion or overgeneralize relative motion to the scene reference frame — suggest that current VLM architectures process observer and object motion through shared feature pathways that do not naturally support factorization. An architecture that explicitly separates ego-motion estimation from object-motion estimation, perhaps through parallel processing streams with a cross-attention mechanism for computing relative relationships, is a natural structural intervention that the paper's task taxonomy directly motivates.
Consistency training objectives. The group-wise accuracy metric measures a property — consistency under symmetric perturbation — that is not explicitly optimized by any standard VLM training objective. A training-time intervention that penalizes inconsistent predictions across augmented variants (e.g., a contrastive loss that pushes the model toward giving appropriately transformed answers for horizontally flipped inputs) could directly target the robustness gap without requiring balanced pretraining data. The paper's four-variant augmentation design provides the paired training data (standard video with answer A, flipped video with transformed answer A′) that such an objective would require.

What becomes less attractive. The paper's findings also diminish the appeal of several popular approaches:

Chain-of-thought reasoning as a general-purpose fix. The finding that free-form reasoning provides near-zero benefit (QwenVL2.5-72B: +0.17%) and sometimes hurts (Gemini-2.5-Pro and InternVL3.5-241B perform worse under FFR than RAWQA) establishes a clear boundary condition on the effectiveness of test-time reasoning. When the bottleneck is in visual perception — the encoder fails to extract the necessary spatial information — no amount of language-based deliberation can compensate. This suggests that research effort spent on more sophisticated reasoning strategies for spatial tasks is misallocated; the bottleneck is earlier in the pipeline.
Scaling model parameters as the primary axis of improvement. The within-family scaling results (larger models improve sample-wise accuracy but degrade group-wise accuracy) indicate that parameter count is not the binding constraint on dynamic spatial intelligence. Research programs organized around "train a bigger VLM and it will eventually understand dynamic scenes" are empirically contraindicated by the paper's results. This does not mean scaling is useless — it improves perceptual precision, which matters — but it means scaling alone cannot solve the decoupling problem.
Static spatial reasoning benchmarks as proxies for spatial intelligence. The paper demonstrates that a model's static spatial reasoning capability (as measured by existing benchmarks) does not predict its dynamic spatial reasoning capability (as measured by DSI-Bench). This means that static benchmarks are not just incomplete — they are positively misleading as measures of spatial intelligence, because they create a false impression of capability that collapses when motion is introduced. Researchers and practitioners should treat high scores on static spatial benchmarks as necessary but not sufficient evidence of spatial understanding, and should supplement them with dynamic evaluations.

Follow-Up Research This Work Enables

Decomposing the sample-wise-to-group-wise accuracy gap by error type. The paper demonstrates that VLMs exhibit a large gap between sample-wise and group-wise accuracy (Table 1) and identifies several candidate biases driving this gap — forward-motion selection bias, rotation-translation confusion, and coupled motion reasoning failures. However, it does not quantify which of these biases contributes most to the gap. A direct follow-up would compute, for each of the four augmented variants separately, the confusion matrix between model predictions and ground truth. The pattern of errors across variants would reveal the dominant failure mode: if forward bias dominates, errors should concentrate on reversed videos where "forward" is incorrect; if left/right bias dominates, errors should concentrate on horizontally flipped videos; if rotation-translation confusion dominates, errors should concentrate on videos where the observer undergoes rotation (misclassified as translation) or translation (misclassified as rotation). This decomposition is analytically straightforward within the existing DSI-Bench framework — it requires only per-variant accuracy reporting, which the paper already has the data to compute — and would transform the benchmark from a diagnostic that says "VLMs are inconsistent" to one that says "VLMs are inconsistent primarily because of X, secondarily because of Y."

Per-motion-pattern accuracy analysis on the 3×3 observer-object grid. DSI-Bench classifies observer and object motion into three types (translation, rotation, combination), producing nine decoupled motion patterns. The paper reports only aggregate accuracy (Table 1) and qualitative error examples. A systematic follow-up would produce a 9-row × 3-task-category table showing accuracy for each motion combination. Specific hypotheses that this analysis would test: (1) Are patterns where observer and object move in the same direction (both translating forward) easier or harder than patterns where they move in opposite directions? (2) Does observer rotation systematically impair object motion perception more than observer translation, consistent with the rotation-translation confusion finding? (3) Are combination motions (simultaneous translation and rotation) disproportionately hard, suggesting that motion-type disentanglement within a single entity is itself a bottleneck? The paper's existing dataset supports this analysis — the motion annotations are already recorded — and the results would provide granular guidance for deployment (which motion patterns are safe) and research prioritization (which patterns need new architectural solutions).

Training a difficulty predictor that eliminates the annotation bottleneck. The paper's central diagnostic — group-wise evaluation under spatio-temporal perturbation — depends on having ground-truth motion annotations for every video, which requires expensive human labeling. This makes the methodology a research instrument rather than a deployment tool. A natural extension would train a difficulty-and-bias predictor that takes a novel video as input and predicts: (1) the motion pattern (observer translation vs. rotation vs. combination, and same for objects), (2) the expected VLM accuracy on that motion pattern (using DSI-Bench's per-pattern results as training labels), and (3) whether the video is likely to trigger specific biases (forward bias, rotation-translation confusion). The predictor could be a lightweight model trained on DSI-Bench's annotated videos, perhaps using SpatialTrackerV2's trajectory estimates as input features rather than raw pixels, making it computationally cheap enough for deployment. Such a predictor would enable adaptive test-time strategies: for videos predicted to trigger forward bias, suppress "forward" answer options or require higher confidence thresholds; for videos with predicted high difficulty, escalate to a specialized spatial reasoning module or human review. This turns DSI-Bench from a static benchmark into a dynamic deployment safeguard.

Human performance baseline with time-pressured VQA. The paper's motion annotations demonstrate that humans can correctly classify motion patterns with unlimited time and focused attention on annotation. This does not establish the ceiling for human performance under the actual DSI-Bench evaluation protocol — watching a 3-second video once at normal speed and answering a multiple-choice question. A critical follow-up would replicate DSI-Bench's VQA evaluation on a sample of human participants (even 20–30 participants on a 50-video subset would be informative), measuring both sample-wise and group-wise accuracy under the same answer format. Specific results that would be informative: (1) Whether humans achieve near-ceiling accuracy on all task categories, confirming that the model errors represent a genuine capability gap rather than an intrinsically hard problem. (2) Whether humans exhibit any forward-motion bias (suggesting it is partly a perceptual phenomenon rather than purely a training-data artifact). (3) Whether the sample-wise-to-group-wise gap for humans is near-zero (as it should be if humans use genuine spatial understanding rather than heuristics), validating the group-wise metric as a measure of "human-like" spatial reasoning. The absence of this baseline is the single largest interpretability gap in the paper, and closing it would substantially increase confidence in all reported findings.

Motion-balanced pretraining intervention with DSI-Bench as the evaluation target. The forward-motion bias finding (Figure 6) suggests a specific intervention: curate or synthesize a video pretraining dataset where all motion directions (forward, backward, left, right, up, down, clockwise rotation, counterclockwise rotation) appear with equal frequency for common object categories (vehicles, animals, people). Fine-tune a base VLM on this balanced dataset and evaluate on DSI-Bench, measuring: (1) whether the forward-selection frequency drops to match the ground-truth distribution, (2) whether the sample-wise-to-group-wise accuracy gap narrows, (3) whether performance on static spatial reasoning benchmarks (VSI-Bench, MMSI-Bench) is maintained or degraded (testing for catastrophic forgetting). A positive result — balanced training reduces bias without harming static capability — would provide a concrete recipe for improving dynamic spatial intelligence. A negative result — balanced training fails to reduce bias, suggesting the bias is architectural rather than data-driven — would redirect attention toward structural interventions (motion factorization architectures, consistency training objectives). Either outcome advances the research agenda.

Testing whether the 3×3 motion grid transfers to real-world deployment domains. DSI-Bench draws its videos from curated datasets (CameraBench, Kinetics-700, SynFMC) and web sources, which may not reflect the motion distributions encountered in specific deployment domains. A domain-transfer study would collect videos from a target application — autonomous driving footage from nuScenes or Waymo Open Dataset, egocentric video from EPIC-Kitchens, drone surveillance footage — annotate them using DSI-Bench's motion taxonomy (or an adapted version if the domain introduces new motion types), and evaluate the same 14 models. The key question is whether the biases documented in DSI-Bench (forward bias, rotation-translation confusion, coupled motion errors) replicate in domain-specific data with different motion statistics. If the biases are universal (appearing across all domains), the case for architectural intervention is strengthened. If the biases are domain-specific (forward bias dominates in egocentric video but not in drone footage), the solution may be domain-specific data balancing rather than general-purpose architectural change.

Practical Applications and Downstream Use Cases

Autonomous driving and advanced driver assistance systems (ADAS). Modern autonomous driving stacks rely on a combination of specialized perception modules (object detection, lane detection, depth estimation) and trajectory prediction systems. VLMs are increasingly being explored as unified perception-and-reasoning components that could handle edge cases — "is that pedestrian about to step into the road?" or "is that vehicle merging into my lane or continuing straight?" — where rigid classification pipelines fail. DSI-Bench's findings have direct implications for whether this approach is safe: the paper demonstrates that current VLMs conflate observer motion (ego-vehicle motion) with object motion, which in a driving context means the model might attribute the visual expansion of a nearby car to that car approaching when it is actually the ego-vehicle accelerating toward it. The 11.55% degradation in observer motion perception when objects are dynamic (Seed-1.6-vision, Figure 5) suggests that VLMs become substantially less reliable at estimating their own motion precisely when other moving entities are present — which is essentially always the case in driving. A practical deployment decision rule emerging from the paper: if a VLM scores below some threshold on DSI-Bench's Observer-Object task category (the tasks directly testing decoupled motion understanding), it should not be trusted as the sole source of spatial reasoning in safety-critical dynamic environments, regardless of its performance on static perception benchmarks.

Augmented reality (AR) spatial anchoring. AR systems must maintain consistent spatial anchors — virtual objects that appear fixed in the physical world — as the user moves through an environment. If the system misestimates its own motion (camera ego-motion), the virtual objects drift, breaking the illusion and potentially causing disorientation. Current AR systems use specialized SLAM (Simultaneous Localization and Mapping) pipelines for this, but there is growing interest in using VLMs for higher-level spatial understanding — recognizing that "the user has entered the kitchen" or "the user is walking toward the table." DSI-Bench's finding that VLMs confuse rotation with translation (Figure 7, Section 5.1) is directly relevant: in an AR context, confusing a head rotation (looking around the room) with a translation (walking across the room) would cause the system to update the user's position incorrectly, breaking spatial anchoring. The 3D expertise models (VGGT, SpatialTrackerV2) show near-zero group-wise degradation (Table 1), suggesting they maintain geometric consistency under perturbation — precisely the property needed for AR anchoring. The practical implication is that AR systems should use geometric SLAM for low-level pose estimation (where expertise models excel) and restrict VLMs to semantic spatial reasoning (recognizing rooms, objects, spatial relationships) that does not depend on precise metric estimation — a functional split that the paper's results directly motivate.

Video understanding for content moderation and search. Platforms that process user-uploaded video at scale — social media, video sharing sites, surveillance systems — increasingly use VLMs for automated content understanding: detecting policy violations, generating captions, enabling natural language video search. DSI-Bench reveals that VLMs exhibit systematic forward-motion bias (Figure 6) and coupled motion reasoning failures (Figure 8), which means that a VLM tasked with describing a video might systematically mischaracterize motion direction or attribute motion to the wrong entity. For content moderation, this could mean misclassifying an approaching person as a receding person (or vice versa), which might be the difference between flagging a video as "harassment" (someone approaching aggressively) versus "incidental" (someone walking away). For video search, a query like "videos where the camera pans left while the subject moves right" would systematically fail because VLMs cannot decouple these motion streams. The practical mitigation — until models improve — is to distrust VLM-generated motion descriptions for any application where motion direction carries semantic meaning, and to rely on specialized trackers (CoTracker, SpatialTrackerV2) for motion-relevant metadata.

Robotic navigation in dynamic human environments. Service robots operating in hospitals, warehouses, or homes must navigate spaces shared with humans who are also in motion. The robot must simultaneously track its own position (observer motion), track moving humans (object motion), and compute relative relationships (is this person walking toward me or away from me? Will our paths intersect?). DSI-Bench evaluates exactly this capability through its Observer-Object task category (Section 3.1). The paper's finding that "current VLMs are unable to independently infer the motion of the observer in 3D space and the motion of the observed object" (Section 5.1) means that a robot using a VLM as its spatial reasoning module would be blind to relative motion relationships — it might correctly detect a person but fail to determine whether a collision is imminent. The 3D expertise models show better robustness (minimal group-wise degradation) but are also imperfect, with the paper noting "inaccurate estimation of the observer–object relative distance in dynamic scenes" (Section 5.2). The practical implication is that current VLMs are not ready for closed-loop spatial reasoning in dynamic environments and should be restricted to open-loop tasks (scene description, object identification) where motion attribution errors do not directly cause unsafe actions. The paper does not provide a threshold for "ready," but the gap between VLM and expertise model group-wise accuracy in Table 1 provides a quantitative benchmark against which future improvements can be measured — a VLM that matches SpatialTrackerV2's group-wise consistency on DSI-Bench would be a candidate for closed-loop deployment.