Multimodal2025~109 minarXiv:2512.17012

4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation

score 6Chiao-An Yang, Ryo Hachiuma, Sifei Liu +4NVIDIA↑ 47Dec 18, 2025cs.CV

🎯 Pitch

Even advanced MLLMs like GPT-4o fail on region-level 4D questions requiring joint depth, tracking, and temporal reasoning, but 4D-RGPT shows that distilling frozen expert 4D signals—without any inference overhead—closes this gap, lifting static scene accuracy far more than dynamic.

1. Executive Summary

This paper introduces 4D-RGPT, a specialized Multimodal LLM that enhances region-level 4D understanding from video inputs by perceiving depth and temporal dynamics, and proposes Perceptual 4D Distillation (P4D), a training-only framework that transfers 4D representations from a frozen expert model into the student MLLM without adding inference cost (via dual-branch latent and explicit distillation losses on depth, optical flow, motion, and camera ray predictions). Evaluated on the newly constructed R4D-Bench — a benchmark of 1,517 region-prompted multiple-choice questions spanning static and dynamic 4D scenes — 4D-RGPT achieves an average improvement of +4.3% over the NVILA-Lite-8B baseline and +5.3% on average across six existing non-region 3D/4D VQA benchmarks, establishing that explicit 4D perceptual supervision through distillation improves spatial-temporal reasoning only when the target problems lie within the model's reach, as shown by the sharp performance boundaries between static and dynamic question categories.

2. Context and Motivation

The fundamental question this paper tackles is: Can Multimodal LLMs answer questions that require tracking specific objects through 3D space over time? This is what the authors call region-level 4D understanding — the ability to take a video, a language query that references particular objects or regions (e.g., "How fast did the car marked ⟨R1⟩ move?"), and produce an answer that demands coordinated perception of depth (3D), temporal dynamics, and visual grounding.

This capability matters because it sits at the intersection of two increasingly critical requirements for deployed AI systems. First, precision targeting: users in applications like autonomous driving, robotic manipulation, or industrial inspection don't ask vague questions about entire scenes — they ask about specific objects, specific regions, specific events. Second, 4D reasoning: answering those questions requires more than recognizing what objects look like; it requires understanding where they are in 3D space, how they move, and how fast they move. As the paper's Figure 1 illustrates, a query like "What is the average speed of ⟨R1⟩?" demands that the model (a) track the purple-bounded car across frames, (b) estimate its depth to compute displacement in real-world units, and (c) know the temporal duration to compute speed. These are not separable skills — they compound.

The gap the paper identifies is that existing MLLMs, including proprietary models like GPT-4o, fail at this compound task not because they lack general video understanding, but because they lack the underlying 4D perceptual machinery — the ability to extract depth, optical flow, and motion signals from visual inputs and reason over them temporally. The paper's central thesis is that 4D understanding is bottlenecked by 4D perception: you cannot reason about depth and motion if you cannot first perceive it.

Why This Problem Matters: Real-World Applications and a Missing Evaluation Protocol

The practical motivation is straightforward. Consider autonomous driving: a safety-critical system must not only detect a pedestrian but answer questions like "Is the pedestrian in region R moving toward the vehicle's path?" This requires tracked depth over time — precisely the compound skill the paper targets. Similarly, in robotic manipulation, a system watching a video of an arm grasping an object needs to answer "Did ⟨R1⟩ successfully grasp ⟨R2⟩?" — which requires understanding the 3D spatial relationship between two tracked regions as it evolves. In industrial inspection, a system might need to answer "Did the component in ⟨R1⟩ rotate by more than 15 degrees during assembly?" — demanding rotational motion estimation tied to a specific region.

The theoretical significance is equally important. The paper identifies that prior work has treated 2D spatial reasoning, 3D spatial reasoning, and temporal reasoning as separate capabilities that can be developed independently. But real-world physical reasoning requires them to be joint. You cannot decouple depth estimation from temporal tracking — the depth of a moving object changes over time, and tracking it is what lets you compute speeds and displacements. The paper argues that this joint capability represents a qualitatively different challenge from either 3D understanding of static scenes or 2D video understanding of dynamic scenes.

Critically, the paper also identifies that no existing benchmark properly evaluates this joint capability. As laid out in Table 1, existing 3D/4D VQA benchmarks fall into two categories, each with a critical omission:

Image-based 3D benchmarks (SAT-real, MMSI-Bench, OmniSpatial): These evaluate 3D spatial reasoning on still images. They test depth perception and spatial relationships, but by construction contain zero temporal dynamics. A model can perform well on these benchmarks by treating each image as a static 3D snapshot, without ever demonstrating that it can track how spatial relationships evolve over time.
Video-based 4D benchmarks without region prompting (STI-Bench, VLM4D-real): These introduce dynamic scenes with both camera and object movement, but their questions are about the entire scene — "How did the person move the drone?" rather than "How did ⟨R1⟩ move ⟨R2⟩?" Without region prompting, these benchmarks do not test whether the model can ground its 4D understanding to specific, user-specified objects. A model can succeed by reasoning about the video holistically without ever demonstrating that it can isolate individual objects for precise spatial-temporal queries.

The gap is the combination: dynamic scenes with region-specific questions. R4D-Bench, the paper's proposed benchmark, fills exactly this void with 1,517 region-prompted multiple-choice questions across 780 dynamic videos (10–30 fps) covering nine question categories spanning both static and dynamic 4D understanding (Table 1, last row).

Where Prior Approaches Fall Short

The paper identifies three broad categories of prior work, each with limitations that prevent them from solving region-level 4D understanding.

Approach 1: SFT and RL on Self-Curated Text Data

The most common approach among recent 3D/4D MLLMs is to curate question-answer datasets that require spatial or temporal reasoning, then apply standard Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL) — optimizing only over the final text output. Examples include SpatialLLM (Ma et al., 2025), Flatland (Zhang et al., 2025), STKit (Ko et al., 2025), MultiSpa (Xu et al., 2025) for SFT, and ViLaSR (Wu et al., 2025), SpatialReasoner (Ma et al., 2025), SpaceR (Ouyang et al., 2025) for RL.

Why this falls short: The paper argues that text-only supervision is fundamentally insufficient for 4D perception. Consider what the model needs to learn to answer "What is the average speed of the car?": it must internally compute depth values (in meters), track the car's position across frames (in pixels), convert pixel displacement to real-world displacement (using the depth estimates), and divide by the time interval. The SFT loss only provides a signal on the final answer — whether the speed was correctly stated. It provides no direct supervision on the intermediate perceptual computations (the depth maps, the optical flow, the motion estimates). The model is left to discover these representations implicitly from text supervision alone, which is extremely data-inefficient and, the paper argues, often fails entirely in dynamic scenarios where the required perceptual precision is high.

The paper's ablation in Table 4 provides direct evidence: applying SFT alone (the "4D-SFT" row) to the same training data yields only +0.9% improvement on STI-Bench over zero-shot and +2.7% on R4D-Bench, compared to +3.8% and +4.3% respectively with the full P4D distillation framework. The gap demonstrates what is lost when perceptual supervision is absent.

Approach 2: Integrating External 3D Models into the MLLM Architecture

A second line of work keeps the MLLM architecture but attaches external 3D perception modules — pre-trained depth estimators, 3D reconstruction models, or point cloud processors — and feeds their outputs into the MLLM as additional input features. Examples include VGLLM (Zheng et al., 2025), VSTI-Bench (Fan et al., 2025), SDVLM (Chen et al., 2025), SR3D (Cheng et al., 2025), and SAT (Ray et al., 2024).

Why this falls short, part 1 — static, not dynamic: The paper points out that these external models are predominantly designed for static scenes — depth from single images, 3D reconstruction from a single viewpoint, or point cloud understanding of a frozen moment. They do not capture the temporal evolution of 3D structure. A depth map from frame $t$ tells you where objects are at time $t$ , but it does not tell you how they moved between $t$ and $t+1$ . The paper argues this is a fundamental mismatch: "external 3D knowledge mainly helps understand static videos, without fully achieving 4D understanding" (Section 2.1).

Why this falls short, part 2 — inference cost: Even if these external modules were extended to video, they introduce additional computation at inference time. Every query would require running the external 3D model alongside the MLLM, increasing latency and resource requirements. This creates a practical deployment barrier — especially for applications like autonomous driving where inference must be real-time. The paper explicitly motivates its training-only distillation approach as a way to avoid this: "Unlike previous works, P4D contains only training-only modules, incurring no additional inference cost" (Section 1).

Approach 3: Distillation for Static 3D (the Closest Prior Work)

The most directly related prior work is 3DRS (Huang et al., 2025), which uses distillation to transfer 3D spatial representations from a teacher model into an MLLM. This shares the paper's philosophy of using a frozen expert to supervise the MLLM's internal representations during training, avoiding inference-time overhead.

Why this falls short: 3DRS operates on static images, distilling 3D spatial knowledge (depth, surface normals) without any temporal component. The paper positions P4D as a generalization to the 4D regime: "3DRS employs distillation for static 3D scenes, while P4D addresses dynamic scenes with dual distillation on latent and explicit representations to achieve 4D understanding" (Section 2.1). The "dual" aspect — latent distillation on intermediate 4D features plus explicit distillation on decoded 4D signals (depth, flow, motion, camera rays) — is the paper's core architectural contribution for handling temporal dynamics alongside spatial perception.

Contradictory Signals in the Existing Evidence

The paper's motivation is sharpened by a pattern observable across the experimental results in Tables 2 and 3: existing methods that perform well on non-region 4D benchmarks do not necessarily perform well on region-level 4D benchmarks. For example, SpaceR-7B (Ouyang et al., 2025) achieves 37.0% on STI-Bench (second-highest among open-source 7B models in Table 2) but drops to 37.0% on R4D-Bench (third-lowest in Table 3). The paper explicitly notes this inversion: "Although SpaceR outperforms Qwen2.5-VL in Tab. 2, it falls behind on R4D-Bench, suggesting that SpaceR is highly tuned for non-region VQA and its region understanding is weakened" (Section 6.2).

This is not merely an engineering issue — it suggests a deeper capability tradeoff. Models optimized to answer scene-level spatial questions may learn representations that are globally informative but lack the fine-grained per-object discrimination needed when the query targets a specific region. Conversely, models with strong regional grounding (like ViLaSR, which achieves 3.7% on the False Positive category) may catastrophically fail when 4D perception is added to the regional tracking requirement.

The paper's resolution to these conflicting signals is that you need perceptual supervision that is simultaneously spatial, temporal, and tied to model-internal representations. Text-only supervision (Approach 1) fails to teach perception. External model integration (Approach 2) fails to be inference-efficient and struggles with dynamics. Static 3D distillation (Approach 3) fails to capture temporality. P4D is positioned as the synthesis: it provides explicit perceptual supervision (like Approach 2, but through distillation rather than architectural attachment), it handles temporal dynamics through teacher signals like optical flow and motion that capture change over time, and it does so without inference cost (unlike Approach 2).

How This Paper Positions Itself

The paper frames its contribution as filling a specific capability gap that exists at the intersection of three established research directions:

From the MLLM community: The paper inherits the standard MLLM architecture (vision encoder + projector + LLM) and the SFT training paradigm, positioning itself as a drop-in enhancement rather than a new architecture. The baseline choice — NVILA-Lite-8B, a state-of-the-art open-source MLLM — is deliberate: it establishes that the gains come from the distillation framework, not from using a stronger base model.
From the 4D perception community: The paper leverages L4P (Badki et al., 2025), a pre-trained 4D perception model that jointly encodes depth, optical flow, motion, and camera ray information from video. This model is treated as a frozen, black-box teacher — the paper does not modify or fine-tune it, only uses its outputs as distillation targets. This is important because it decouples progress in 4D perception (better teacher models) from progress in 4D understanding (better MLLMs): as 4D perception models improve, P4D can plug them in without architectural changes to the MLLM.
From the knowledge distillation literature: The dual-branch design (latent + explicit distillation) is positioned as necessary for comprehensive 4D knowledge transfer. Latent distillation (aligning intermediate 4D features) provides rich, high-dimensional supervision on the model's internal representations. Explicit distillation (aligning decoded depth, flow, motion, and camera ray maps) provides interpretable, low-level supervision that ensures the model doesn't just capture abstract correlations but actually learns to produce accurate pixel-level 4D signals.

The paper's key positioning move is to argue that 4D understanding through text-only supervision is trying to solve an underdetermined problem — there are too many ways to produce the correct text answer that don't require genuine 4D perception (e.g., memorizing dataset-specific patterns, exploiting linguistic cues, making educated guesses based on 2D appearance alone). Perceptual distillation constrains the solution space by forcing the model's internal representations to align with those of a genuine 4D perception system, ensuring that when the model answers "the car moved at 5 m/s," it has actually performed the depth estimation and temporal tracking that the answer implies.

3. Technical Approach

3.1 Reader Orientation

The system being built is 4D-RGPT, an enhanced Multimodal LLM that can answer questions about specific objects in videos by internally learning to perceive depth and motion through a distillation process, without needing any special modules at inference time. The problem it solves is that standard MLLMs lack the low-level perceptual machinery to extract 4D signals (depth, optical flow, motion) from video, which prevents them from answering questions like "How fast did the car in region R1 move?" — questions that require tracking a specific object through 3D space over time. The solution's shape is a training-only distillation framework (P4D) where a frozen expert 4D perception model teaches the MLLM to produce accurate depth maps, flow fields, and other 4D signals from its own internal representations, coupled with timestamp positional encodings (TPE) that give the model explicit awareness of when each frame occurs.

3.2 Big-Picture Architecture (Diagram in Words)

The architecture has five major components arranged in a teacher-student training pipeline:

Base MLLM (NVILA-Lite-8B) — the standard vision-language model that takes video frames and text questions as input, processes them through a vision encoder ( $\bm{\mathsfit{E}}_{\tt V}$ ), a multi-modal projector ( $\bm{\mathsfit{E}}_{\tt P}$ ), and a large language model ( $\tt LLM$ ), and produces text answers autoregressively. This is the "student" that needs to learn 4D perception.
4D Perception Decoder ( $\bm{\mathsfit{D}}_{\tt 4DP}$ ) — a lightweight, training-only MLP attached to the MLLM's internal hidden states that extracts latent 4D features ( $\hat{\bm{F}}_{\tt 4D}$ ) from the LLM's intermediate representations. This module exists only during training; at inference time, it is discarded entirely.
Prediction Heads ( $\bm{\mathsfit{D}}_m$ for each modality $m \in \mathcal{M}$ ) — training-only decoder modules (borrowed frozen from the teacher) that convert the latent 4D features into explicit pixel-level predictions: depth maps, optical flow fields, motion masks, and camera ray maps. These also exist only during training.
Frozen 4D Perception Teacher (L4P) — a pre-trained expert model that independently processes the same input video and produces both intermediate 4D latent features ( $\bm{F}_{\tt 4D}$ ) and final explicit 4D signals ( $\bm{P}_m$ ). Its weights are never updated; it serves purely as a supervision source.
Timestamp Positional Encoding (TPE) — a sinusoidal encoding added directly to the visual features of each frame before they enter the projector, encoding the frame's absolute timestamp so the model knows when each frame occurs.

Information flows as follows during training: a video $\bm{V}$ with $N$ sampled frames enters the system → the vision encoder $\bm{\mathsfit{E}}_{\tt V}$ processes each frame into visual features → TPE is added to these features based on each frame's timestamp $t^{(n)}$ → the projector $\bm{\mathsfit{E}}_{\tt P}$ maps the features into the LLM's input space → the LLM processes the combined visual and text question features, producing hidden states $\bm{F}_{\tt hidden}$ → the 4D perception decoder $\bm{\mathsfit{D}}_{\tt 4DP}$ extracts latent 4D features $\hat{\bm{F}}_{\tt 4D}$ from these hidden states → the prediction heads $\bm{\mathsfit{D}}_m$ decode these into explicit 4D signals $\hat{\bm{P}}_m$ (predicted depth, flow, etc.) → simultaneously, the frozen teacher L4P processes the same video to produce ground-truth latent features $\bm{F}_{\tt 4D}$ and explicit signals $\bm{P}_m$ → two distillation losses pull the student's representations toward the teacher's: $\mathcal{L}_{\tt LD}$ aligns the latent features, and $\mathcal{L}_{\tt ED}$ aligns each explicit signal → the standard SFT loss $\mathcal{L}_{\tt SFT}$ on the text answer is added → the total loss updates the trainable components (projector, LLM, 4D perception decoder). At inference time, the 4D perception decoder and prediction heads are removed — the model behaves exactly like the base MLLM but with improved 4D understanding.

3.3 Roadmap for the Deep Dive

First, the base MLLM architecture and notation, since everything builds on the standard vision-language pipeline and we need to establish what each component does before explaining how 4D-RGPT modifies it.
Second, the 4D perception teacher model (L4P) and what 4D modalities it provides, because the distillation target needs to be understood before we can explain what the student learns.
Third, the latent 4D representation extraction in 4D-RGPT — how the model produces $\hat{\bm{F}}_{\tt 4D}$ from its own hidden states, which is the first half of the distillation pipeline.
Fourth, the explicit 4D signal prediction heads — how $\hat{\bm{F}}_{\tt 4D}$ gets decoded into interpretable depth, flow, motion, and camera ray maps, forming the second half of the distillation pipeline.
Fifth, the Perceptual 4D Distillation (P4D) framework itself — the dual-branch loss design (latent distillation $\mathcal{L}_{\tt LD}$ and explicit distillation $\mathcal{L}_{\tt ED}$ ), how they complement each other, and why both are necessary.
Sixth, the Timestamp Positional Encoding (TPE) — the sinusoidal encoding mechanism, why explicit temporal cues are critical for 4D VQA, and how TPE outperforms alternatives like burned-in text timestamps.
Seventh, the training recipe — the data mixture, hyperparameters, loss balancing, and trainable/frozen component choices that make the whole system work.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that an MLLM can be taught to internally perceive 4D signals (depth, optical flow, motion, camera geometry) during training by distilling knowledge from a frozen expert 4D perception model, and that this perceptual capability — injected without any architectural changes at inference time — substantially improves the model's ability to answer region-level 4D questions about dynamic videos.

Base MLLM Architecture and Notation

The paper builds on the standard Multimodal LLM architecture, which consists of four components chained together to convert visual inputs and text questions into text answers. Understanding this pipeline is essential because 4D-RGPT's modifications tap into specific intermediate representations within it.

Vision Encoder $\bm{\mathsfit{E}}_{\tt V}$ : This is the component that processes raw visual inputs — images or video frames — and converts them into feature representations. In the specific implementation, the paper uses SigLIP (specifically the siglip-so400m-patch14-384 checkpoint from Zhai et al., 2023) as the vision encoder. Given an input video $\bm{V} = [\bm{I}^{(n)}]_{n=1:N}$ consisting of $N$ sampled frames, where each frame $\bm{I}^{(n)} \in \mathbb{R}^{H \times W \times 3}$ is a standard RGB image (height $H$ , width $W$ , 3 color channels), the vision encoder produces per-frame visual features $\bm{\mathsfit{E}}_{\tt V}(\bm{I}^{(n)})$ . The encoder operates independently on each frame — it does not model temporal relationships between frames, which means temporal understanding must come from elsewhere in the pipeline.

Multi-modal Projector $\bm{\mathsfit{E}}_{\tt P}$ : Vision encoders produce features in a representation space that is not aligned with the language space that LLMs operate in. The projector bridges this gap by mapping visual features into the same dimensionality as the LLM's token embeddings, enabling the LLM to process visual and textual information jointly in a shared representation space. In NVILA-Lite-8B (the base MLLM), the projector is a 2-layer MLP with a hidden dimension of 4,608 and uses a 3×3 downsampling kernel to reduce the spatial resolution of visual features. This downsampling is an efficiency choice: NVILA-Lite trades some spatial granularity for fewer visual tokens (and thus less LLM computation), compared to the full NVILA which uses 2×2 downsampling.

Large Language Model ( $\tt LLM$ ): This is the autoregressive transformer that takes the combined visual and textual features as input and generates output tokens one at a time. The paper uses Qwen2 (Team, 2024) as the LLM backbone within NVILA-Lite-8B. During generation, the LLM produces a sequence of hidden states — one per output token — which are intermediate vector representations before the final vocabulary projection. These hidden states are critical for 4D-RGPT because the 4D perception decoder taps into them to extract 4D features. Specifically, for each input frame $\bm{I}^{(n)}$ , the LLM produces hidden state features $\bm{F}_{\tt hidden}^{(n)} \in \mathbb{R}^{h \times w \times c}$ , where $l = hw$ is the number of per-image visual tokens (the spatial grid of visual features after processing), $(h, w)$ is the spatial size of those features, and $c$ is the hidden dimension of the LLM.

Output Head $\bm{\mathsfit{D}}_{\tt head}$ : The final linear layer that maps the LLM's hidden states to logits over the vocabulary, enabling token-by-token text generation. This component is standard and plays no special role in the 4D-RGPT design.

Why understanding this pipeline matters: The paper's key insight is that the LLM's hidden states $\bm{F}_{\tt hidden}$ — which are normally only used to predict the next token — contain latent information about the visual scene that can be "decoded" into explicit 4D perceptual signals. By attaching a lightweight decoder to these hidden states during training and forcing it to match a 4D perception teacher's outputs, the model learns to internally represent depth, motion, and flow in a way that improves its downstream reasoning, even after the decoder is removed.

The 4D Perception Teacher Model (L4P)

The teacher model provides the ground-truth 4D perceptual signals against which the student 4D-RGPT is trained. The paper uses L4P (Badki et al., 2025), a pre-trained 4D perception model, and treats it as a frozen, black-box expert — its weights are never updated during training, and it is never used at inference time. Understanding what the teacher produces is essential because these are the distillation targets.

Teacher Architecture. L4P consists of two components:

A unified 4D encoder $\bm{\mathsfit{E}}_{\tt 4D}$ : This is a 40-layer ViT-based video encoder adapted from VideoMAEv2 (Wang et al., 2023). Unlike the MLLM's vision encoder which processes frames independently, this encoder processes the video as a spatiotemporal volume, capturing motion and temporal relationships between frames. Given the input video $\bm{V}$ , it produces a latent 4D feature representation:

$\bm{F}_{\tt 4D} = \bm{\mathsfit{E}}_{\tt 4D}(\bm{V}) \in \mathbb{R}^{N' \times h' \times w' \times c'}$

where $N'$ is the down-sampled number of frames (the encoder temporally pools the video), $h'$ and $w'$ are the down-sampled spatial dimensions, and $c'$ is the number of output channels. This latent feature $\bm{F}_{\tt 4D}$ is a rich, abstract representation that encodes the 4D structure of the scene — depth, motion, camera geometry — in a compressed form. It is the target for the latent distillation branch of P4D.
Specialized decoders $\bm{\mathsfit{D}}_m$ for each 4D modality $m \in \mathcal{M}$ : Each decoder has the same architecture — DPT (Ranftl et al., 2021), a dense prediction transformer — but different output channels depending on the target modality. Each decoder takes the shared latent feature $\bm{F}_{\tt 4D}$ as input and produces per-pixel predictions:

$\bm{P}_m = \bm{\mathsfit{D}}_m(\bm{F}_{\tt 4D})$

These $\bm{P}_m$ are the targets for the explicit distillation branch.

The Four 4D Modalities $\mathcal{M}$ : The paper uses four complementary 4D signals, each capturing a different aspect of the scene's spatiotemporal structure:

$m = {\tt depth}$ : $\bm{P}_{\tt depth}^{(n)} \in \mathbb{R}^{H \times W \times 1}$ is the per-pixel depth map for each frame, representing the distance from the camera to each visible surface point. This is the most fundamental 3D signal — it tells the model where objects are in space. Without depth perception, the model cannot compute real-world displacements, sizes, or distances.
$m = {\tt flow}$ : $\bm{P}_{\tt flow}^{(n)} \in \mathbb{R}^{H \times W \times 2}$ is the per-pixel optical flow between adjacent frames, representing the 2D pixel displacement of each point from frame $n$ to frame $n+1$ . Optical flow captures apparent motion in the image plane — it tells the model how objects are moving in the 2D projection, which, combined with depth, enables 3D motion estimation.
$m = {\tt motion}$ : $\bm{P}_{\tt motion}^{(n)} \in \mathbb{R}^{H \times W \times 1}$ is a per-pixel binary (or continuous) mask indicating whether each pixel corresponds to a point that is moving in 3D space (as opposed to static background or camera-induced apparent motion). This distinguishes genuine object motion from parallax effects, which is critical for answering questions about whether an object is moving independently of the camera.
$m = {\tt camray}$ : $\bm{P}_{\tt camray}^{(n)} \in \mathbb{R}^{H \times W \times 6}$ is the per-pixel Plücker ray map, which encodes the camera's geometry — specifically, for each pixel, the 3D ray from the camera center through that pixel, represented in Plücker coordinates (6 numbers: 3 for direction, 3 for moment). This tells the model the camera's intrinsic and extrinsic parameters, enabling it to convert 2D pixel measurements into 3D spatial reasoning.

Why these four modalities: The choice is deliberate and complementary. Depth alone gives static 3D structure but no motion. Optical flow gives 2D motion but cannot distinguish camera motion from object motion. The motion mask disambiguates camera vs. object movement. Camera rays provide the geometric calibration needed to lift 2D measurements into metric 3D space. Together, they provide complete low-level 4D perceptual information — everything a model needs to answer questions about object speeds, displacements, spatial relationships, and motion types in physical units.

Why L4P specifically: The paper uses L4P because it is a unified model that jointly predicts all four modalities from video, meaning its latent features $\bm{F}_{\tt 4D}$ are already a rich, multi-task 4D representation. Using separate single-task teachers (e.g., one depth estimator, one optical flow estimator) would produce fragmented latent spaces that don't capture the cross-modal relationships that L4P's joint training encodes.

Latent 4D Representation Extraction in 4D-RGPT

The first major modification 4D-RGPT makes to the base MLLM is the injection of a 4D perception decoder $\bm{\mathsfit{D}}_{\tt 4DP}$ that extracts latent 4D features from the LLM's internal hidden states. This decoder is the bridge between the MLLM's text-oriented representations and the teacher's 4D perceptual representations.

Where the features come from. As the LLM processes the input video and question, it produces hidden states $\bm{F}_{\tt hidden}^{(n)} \in \mathbb{R}^{h \times w \times c}$ for each frame $\bm{I}^{(n)}$ . These hidden states are the LLM's internal representation of the visual information after it has been contextualized with the text question and processed through multiple transformer layers. The key assumption — which the paper's results validate — is that these hidden states contain latent information about the 3D and temporal structure of the scene, even though the LLM was never explicitly trained to encode such information.

The 4D perception decoder $\bm{\mathsfit{D}}_{\tt 4DP}$ : This is a lightweight, training-only MLP designed to extract 4D-specific features from the LLM's hidden states. Its architecture is deliberately simple: a 3-layer MLP with a hidden dimension of 2,560, using GELU activations (Hendrycks and Gimpel, 2016) between layers. The weights are initialized with Xavier initialization (Glorot and Bengio, 2010) and biases are initialized to zero.

The extraction process works in two steps:

Reshaping: The LLM's hidden states $\bm{F}_{\tt hidden}^{(n)}$ are produced in a token-wise format — one vector per visual token — and need to be rearranged into a spatial grid that matches the teacher's latent feature dimensions $(N', h', w')$ . The paper uses a $\tt Rearrange$ operation to reshape the token sequence into the target spatiotemporal layout. This is a purely structural operation — no learned parameters — that recovers the spatial structure that was flattened for the transformer.
Decoding: The reshaped hidden states are fed through the MLP:

$\hat{\bm{F}}_{\tt 4D}^{(n')} = \bm{\mathsfit{D}}_{\tt 4DP}\left({\tt Rearrange}(\bm{F}_{\tt hidden}^{(n)})\right)$

For each down-sampled frame index $n' \in [1, N']$ , this produces a latent 4D feature vector $\hat{\bm{F}}_{\tt 4D}^{(n')}$ that has the same dimensionality as the teacher's latent feature $\bm{F}_{\tt 4D}^{(n')}$ . This is the student's attempt to internally represent the 4D structure of the scene.

Why a 3-layer MLP: The decoder needs to be expressive enough to transform the LLM's general-purpose features into 4D-specific features, but lightweight enough that (a) it doesn't dominate training, (b) it forces the LLM to do the heavy lifting of representing 4D information in its hidden states (rather than the decoder compensating for poor LLM representations), and (c) it can be cleanly discarded at inference time without losing essential model functionality. A deeper decoder might learn to extract 4D signals from weak LLM representations, reducing the pressure on the LLM to genuinely internalize 4D knowledge.

Why attach to LLM hidden states rather than vision encoder features: The LLM's hidden states have been processed through attention layers that can integrate information across frames and between visual and textual modalities. This means they potentially encode temporal relationships and question-conditioned spatial attention — a richer representation for 4D understanding than the vision encoder's per-frame, pre-attention features. The distillation loss then provides a training signal that propagates back through the LLM, teaching it to preserve and enhance the 4D structure that is latent in the visual input.

Training-only design: The decoder $\bm{\mathsfit{D}}_{\tt 4DP}$ only exists during training. At inference time, it is removed, and the model behaves like a standard MLLM. The 4D perceptual knowledge is "compiled into" the LLM's weights — the LLM learns to produce hidden states that contain 4D structure, which improves its text answers even though the explicit 4D predictions are no longer generated.

Explicit 4D Signal Prediction Heads

While the latent features $\hat{\bm{F}}_{\tt 4D}$ capture abstract 4D representations, the paper also generates explicit, interpretable 4D signals — actual depth maps, flow fields, motion masks, and camera ray maps — as intermediate predictions during training. These serve as targets for the explicit distillation branch and provide direct, unambiguous supervision.

How the explicit signals are produced. The paper reuses the teacher's decoder heads $\bm{\mathsfit{D}}_m$ — the same DPT modules that decode the teacher's latent features into per-pixel predictions — and applies them to the student's latent features instead. For each modality $m \in \mathcal{M}$ :

$\hat{\bm{P}}_m = \bm{\mathsfit{D}}_m(\hat{\bm{F}}_{\tt 4D})$

The student's latent features $\hat{\bm{F}}_{\tt 4D}$ are fed into the frozen teacher decoder $\bm{\mathsfit{D}}_m$ , which produces the student's explicit prediction $\hat{\bm{P}}_m$ . This prediction has the same spatial resolution and channel dimensionality as the teacher's ground-truth signal $\bm{P}_m$ — for example, $\hat{\bm{P}}_{\tt depth}^{(n)} \in \mathbb{R}^{H \times W \times 1}$ for depth, $\hat{\bm{P}}_{\tt flow}^{(n)} \in \mathbb{R}^{H \times W \times 2}$ for optical flow.

Why reuse frozen teacher decoders rather than training new ones: This is a critical design choice with two justifications. First, it ensures that the explicit signals are expressed in the same representation space as the teacher's signals — the decoder's mapping from latent features to pixel predictions is fixed, so the only way for the student to produce accurate explicit signals is to produce latent features $\hat{\bm{F}}_{\tt 4D}$ that match the teacher's latent features $\bm{F}_{\tt 4D}$ . This creates a tight coupling between the latent and explicit distillation objectives: improving explicit predictions requires improving latent features. Second, it prevents the decoders from "cheating" by learning to produce correct-looking outputs from poor latent features — since the decoders are frozen, they cannot adapt to the student's potentially flawed latent space, which forces the latent space to genuinely capture 4D structure.

The output channels for each modality are determined by the teacher's architecture:

Depth: 1 channel (scalar depth value per pixel)
Flow: 2 channels (x and y displacement per pixel)
Motion: 1 channel (motion probability per pixel)
Camray: 6 channels (Plücker ray coordinates per pixel)

Visualization evidence of learning. Figure 5 and Figure A20 in the paper show the evolution of $\hat{\bm{P}}_{\tt depth}$ during training. Initially, the predicted depth maps are "barely meaningful" — they show grid-like artifacts that the authors hypothesize come from the tokenization structure in the LLM's hidden states. As training proceeds through P4D, the grid artifacts gradually disappear and the depth maps capture genuine 3D structure (object boundaries, relative depths, scene layout). This is direct visual evidence that the distillation is working: the LLM's hidden states are being transformed to encode metric 3D information that the frozen decoder can successfully interpret as depth.

Perceptual 4D Distillation (P4D) Framework

P4D is the core training methodology — the loss function and supervision strategy that transfers 4D knowledge from the frozen teacher to the student 4D-RGPT. It consists of two complementary distillation branches plus the standard SFT loss, which together optimize the model end-to-end.

Why dual-branch distillation. The paper argues that a single distillation branch is insufficient for comprehensive 4D knowledge transfer. Latent features capture rich, high-dimensional, abstract representations that encode relationships between depth, motion, and camera geometry — but they are uninterpretable and could potentially encode spurious correlations. Explicit signals provide direct, interpretable, low-level supervision on the actual 4D quantities the model needs to reason about — but they are lower-dimensional and may not capture all the useful structure in the teacher's latent space. The dual-branch design ensures both: the model's internal representations are rich and 4D-structured (via latent distillation), and its explicit perceptual outputs are accurate (via explicit distillation).

Latent Distillation Loss $\mathcal{L}_{\tt LD}$

The latent distillation loss aligns the student's 4D latent features with the teacher's:

$\mathcal{L}_{\tt LD} = \sum_{n'=1}^{N'} \Delta_{\tt LD}(\bm{F}_{\tt 4D}^{(n')}, \hat{\bm{F}}_{\tt 4D}^{(n')})$

where $\bm{F}_{\tt 4D}^{(n')}$ is the teacher's latent 4D feature for frame $n'$ (from the frozen L4P encoder), $\hat{\bm{F}}_{\tt 4D}^{(n')}$ is the student's latent 4D feature for the same frame (from the 4D perception decoder applied to LLM hidden states), $N'$ is the number of down-sampled frames, and $\Delta_{\tt LD}$ is the Smooth-L1 distance function.

What Smooth-L1 computes: For each element in the feature tensors, Smooth-L1 computes:

$\text{Smooth-L1}(x, y) = \begin{cases} 0.5(x - y)^2 & \text{if } |x - y| < 1 \\ |x - y| - 0.5 & \text{otherwise} \end{cases}$

This is a hybrid between L2 loss (squared error, sensitive to small differences) and L1 loss (absolute error, robust to outliers). For small differences ( $|x - y| < 1$ ), it behaves like L2 — providing strong gradients that drive precise alignment. For large differences, it behaves like L1 — providing constant-magnitude gradients that are robust to outlier feature dimensions where the student cannot possibly match the teacher.

Why Smooth-L1 over L2 or L1 alone: L2 loss on high-dimensional latent features would be dominated by the worst-aligned dimensions — a few feature channels that the student fundamentally cannot match (perhaps encoding information the base MLLM architecture cannot represent) would produce enormous gradients that destabilize training. L1 loss would be too permissive for the well-aligned dimensions — it wouldn't provide enough gradient signal to achieve precise alignment where it's possible. Smooth-L1 provides the best of both: precise alignment where feasible, robustness where not.

What this loss accomplishes operationally: It pulls the student's entire 4D latent representation toward the teacher's, feature-dimension by feature-dimension, frame by frame. This is a dense, high-dimensional supervision signal that propagates gradients through the 4D perception decoder $\bm{\mathsfit{D}}_{\tt 4DP}$ and into the LLM's hidden states, teaching the LLM to produce representations that contain the same 4D structural information as a dedicated 4D perception model.

Explicit Distillation Loss $\mathcal{L}_{\tt ED}$

The explicit distillation loss aligns the student's decoded 4D signals with the teacher's:

$\mathcal{L}_{\tt ED} = \sum_{n=1}^{N} \sum_{m \in \mathcal{M}} \lambda_m \Delta_m(\bm{P}_m^{(n)}, \hat{\bm{P}}_m^{(n)})$

where $\bm{P}_m^{(n)}$ is the teacher's ground-truth explicit signal for modality $m$ at frame $n$ , $\hat{\bm{P}}_m^{(n)}$ is the student's predicted explicit signal for the same modality and frame (produced by feeding $\hat{\bm{F}}_{\tt 4D}$ through the frozen teacher decoder $\bm{\mathsfit{D}}_m$ ), $\lambda_m$ is a modality-specific loss weight, $\Delta_m$ is also Smooth-L1 for all modalities, and $N$ is the number of input frames (note: at the original frame resolution, not the down-sampled $N'$ used for latent features).

The modality weights $\lambda_m$ are set as follows:

$\lambda_{\tt depth} = 1.0$ — depth gets the highest weight because it is the most fundamental 3D signal and is critical for metric spatial reasoning (speeds, distances, sizes).
$\lambda_{\tt flow} = 0.1$ — optical flow gets lower weight, possibly because (a) it is noisier and harder to predict precisely, and (b) it is partially redundant with depth + motion (if you know 3D motion and camera geometry, you can derive flow).
$\lambda_{\tt motion} = 0.05$ — motion gets even lower weight, likely because it is binary/coarse and provides less dense supervision than depth or flow.
$\lambda_{\tt camray} = 0.05$ — camera rays get the lowest weight, likely because they are scene-independent (camera geometry doesn't vary per-scene in the same way depth does) and may be easier to learn.

Why explicit distillation on the original $N$ frames rather than down-sampled $N'$ : The latent features are produced by a video encoder that temporally pools the input, producing a lower temporal resolution. But the explicit signals — depth maps, flow fields — are per-frame predictions that make sense at the original frame rate. The student's latent features are upsampled/interpolated by the teacher decoders $\bm{\mathsfit{D}}_m$ (which include their own upsampling layers, since DPT is a dense prediction architecture) to produce full-resolution explicit predictions.

What this loss accomplishes operationally: It ensures that the student's latent features $\hat{\bm{F}}_{\tt 4D}$ not only match the teacher's latent features in the abstract (via $\mathcal{L}_{\tt LD}$ ) but also decode to correct pixel-level 4D signals. This prevents a failure mode where the latent features match in some metric space but encode different 4D structure — the explicit loss ensures semantic alignment of the 4D content, not just distributional matching.

Total Training Loss

The total loss combines SFT, latent distillation, and explicit distillation:

$\mathcal{L} = \mathcal{L}_{\tt SFT} + \alpha\mathcal{L}_{\tt LD} + \beta\mathcal{L}_{\tt ED}$

where $\alpha = 0.5$ and $\beta = 0.1$ are hyperparameters balancing the three loss terms.

Why $\alpha > \beta$ : The latent distillation loss weight (0.5) is five times larger than the explicit distillation weight (0.1). This reflects the paper's finding (Table 5) that latent distillation alone (LD-Only) provides a larger improvement (+2.3% on R4D-Bench over Zero-shot) than explicit distillation alone (ED-Only, +1.9%), and that the combination is best. The higher weight on latent distillation is likely because the latent features are much higher-dimensional than any single explicit signal — they encode joint 4D structure across all modalities — so the per-dimension gradient magnitude is smaller, requiring a higher loss weight to have comparable influence on the optimization.

Why $\mathcal{L}_{\tt SFT}$ is unweighted (implicit weight = 1.0): The SFT loss is the primary objective — the model must still learn to answer questions correctly. The distillation losses are auxiliary objectives that shape the model's internal representations to make the SFT task easier. The unweighted SFT loss ensures that text generation accuracy remains the dominant training signal, with distillation providing a beneficial inductive bias rather than competing with the main objective.

What the combined loss achieves: During backpropagation, the SFT loss provides gradients that improve the model's text answers. The latent distillation loss provides gradients through $\bm{\mathsfit{D}}_{\tt 4DP}$ into the LLM, teaching it to produce hidden states that encode 4D structure. The explicit distillation loss provides gradients through the frozen $\bm{\mathsfit{D}}_m$ decoders, through $\hat{\bm{F}}_{\tt 4D}$ , through $\bm{\mathsfit{D}}_{\tt 4DP}$ , and into the LLM, teaching the model to produce hidden states whose 4D content is not just abstractly similar to the teacher's but semantically meaningful in terms of actual depth, flow, motion, and camera geometry. The combined effect is that the LLM learns to internally represent 4D perceptual information, which makes it better at producing text answers that require reasoning about that information.

Timestamp Positional Encoding (TPE)

The second major modification 4D-RGPT makes to the base MLLM is the injection of explicit temporal information through sinusoidal positional encodings added directly to the visual features of each frame.

The problem TPE solves. MLLMs typically receive video frames as an unordered or implicitly ordered sequence — the frames are fed in temporal order, and the model's positional encodings (if any) indicate frame order (1st, 2nd, 3rd, ...) but not absolute or relative timing. This means the model has no way to know how much time passed between frames — which is critical for computing speeds, accelerations, and durations. As the paper's toy experiment (TimeBench, Figure A15 and Table A3) demonstrates, without explicit temporal cues, models are "naively guessing" time durations, achieving accuracy close to random (20–22.7%). Even with P4D's perceptual distillation, the model might learn depth and motion but still fail at temporal reasoning because it doesn't know the timescale.

Why frame order isn't enough. Different video sources have different frame rates — the benchmarks used in this paper span 10 to 30 fps (Table 1). Ten frames at 10 fps represents 1 second; ten frames at 30 fps represents 0.33 seconds. Without knowing the frame rate or absolute timestamps, the model cannot convert frame counts into time intervals, making speed and acceleration questions impossible to answer correctly. The training data mixture compounds this problem: the model sees videos from multiple sources with different frame rates, making it impossible to learn a single implicit frame-rate assumption.

How TPE works. For each input frame $\bm{I}^{(n)}$ sampled at absolute time $t^{(n)}$ (in seconds from the start of the video), the paper computes a sinusoidal positional encoding and adds it to the visual features before they enter the projector:

$\bm{p}^{(n)}[2i] = \sin\left(\frac{t^{(n)}}{T^{2i/D}}\right) \quad \text{and} \quad \bm{p}^{(n)}[2i+1] = \cos\left(\frac{t^{(n)}}{T^{2i/D}}\right)$

where $\bm{p}^{(n)} \in \mathbb{R}^D$ is the timestamp encoding vector for frame $n$ , $D$ is the dimensionality of the visual features (matching the vision encoder output dimension), $i$ indexes the dimension pairs (each pair of sine/cosine encodes one frequency), $T = 10{,}000$ is the maximum timescale (following the original Transformer sinusoidal encoding), and the encoding is added element-wise: $\bm{\mathsfit{E}}_{\tt V}(\bm{I}^{(n)}) + \bm{p}^{(n)}$ .

What this computes operationally: For each frame, the model receives not just the visual content but also a unique temporal "fingerprint" that encodes the absolute timestamp. Different timestamps produce different encoding vectors. Because the encoding uses sinusoids at different frequencies, the model can learn to attend to or extract the time difference between any two frames by comparing their encodings — the dot product of two sinusoidal encodings depends on their time separation.

Why sinusoidal encoding rather than learned embeddings: The sinusoidal encoding (a) generalizes to unseen timestamps — the model can handle any $t^{(n)}$ value, not just those seen during training; (b) provides a smooth, continuous representation of time where nearby timestamps have similar encodings; (c) enables the model to compute time differences through linear operations on the encodings (a property of sinusoidal representations); and (d) requires no additional learned parameters, keeping the approach lightweight.

Why TPE over alternative temporal cue methods (Table 6):

Burned-in timestamps (P4D+mark): Adding text like "2.5 s" as a visual overlay on each frame achieves 95.3% on TimeBench but only +0.3% on R4D-Bench over P4D without cues. The text distracts from the visual content and doesn't integrate naturally with the region-based prompting — region markers and timestamp text compete for visual attention.
Textual time prompts (P4D+prompt): Adding time information to the question text (e.g., "The video is 19 seconds long at 30 fps") achieves 98.0% on TimeBench but only +1.1% on R4D-Bench over P4D without cues. The model must learn to parse and apply this global temporal metadata to per-frame reasoning, which is an indirect and error-prone process — the model may ignore the prompt or fail to correctly associate specific frames with specific time points.
TPE (the paper's method): Achieves +2.8% on R4D-Bench over P4D without cues (Table 6, last row) by directly encoding temporal information into the frame features, making time a first-class input signal that the model processes alongside visual content in every attention layer. It requires no data preprocessing, doesn't clutter the visual input, and generalizes seamlessly to region-level VQA.

Why TPE is added before the projector, not after: The projector is trainable (as shown in Table 7, the best configuration tunes the projector and LLM). By adding TPE before the projector, the model can learn to project temporal information into a representation space that integrates with the visual features, potentially learning interactions between time and visual content (e.g., "this visual feature at this time means fast motion"). If TPE were added after the projector, the temporal and visual features would be combined only in the LLM's first layer, losing the opportunity for deeper integration.

Training Recipe and Component Freeze/Thaw Strategy

The paper carefully selects which components to train and which to freeze, based on extensive ablation studies (Table 7). The goal is to inject 4D perceptual knowledge without degrading the base model's general vision-language capabilities.

Trainable components (optimal configuration, "Tune-P+LLM"):

$\bm{\mathsfit{E}}_{\tt P}$ (multi-modal projector): The projector must adapt to the addition of TPE to the visual features — the input distribution has changed because each frame now carries a timestamp encoding. Additionally, the projector provides the first opportunity for the model to learn interactions between visual and temporal features. Table 7 shows that tuning the projector alone (Tune-P) provides minimal benefit (+0.5% STI, +0.7% R4D-Bench over Zero-shot), but combined with LLM tuning, it's optimal.
$\tt LLM$ (large language model): The LLM is where the bulk of 4D knowledge is stored — the hidden states that feed into the 4D perception decoder must encode depth, motion, and flow structure. Tuning the LLM allows the distillation gradients to reshape its internal representations. Table 7 shows that tuning the LLM alone (Tune-LLM) provides +1.6% on STI and +2.6% on R4D-Bench over Zero-shot, and combined with projector tuning achieves the best results.
$\bm{\mathsfit{D}}_{\tt 4DP}$ (4D perception decoder): This is trained from scratch (Xavier initialization) since it's a new module — it must learn to extract 4D features from the LLM's hidden states.

Frozen components:

$\bm{\mathsfit{E}}_{\tt V}$ (vision encoder): The vision encoder is kept frozen. The paper's ablation shows that tuning it (Tune-All, which tunes everything) actually hurts performance (+0.9% STI, +0.9% R4D-Bench over Zero-shot, worse than Tune-P+LLM's +3.8% and +4.3%). This is likely because the vision encoder has been pre-trained on massive image data to produce general-purpose visual features; fine-tuning on a relatively small 4D VQA dataset degrades these features, losing the generalization that makes them useful. The 4D knowledge should be learned in the LLM, which can integrate visual features across time, rather than in the vision encoder, which processes frames independently.
$\bm{\mathsfit{D}}_m$ (teacher decoders): These are frozen by construction — they come from the frozen L4P teacher and are never updated. This ensures the explicit distillation targets remain in the teacher's representation space.
$\bm{\mathsfit{E}}_{\tt 4D}$ (teacher encoder): Also frozen by construction — the teacher is never trained.

Why not use LoRA (Tune-LLM-LoRA vs. Tune-LLM): Table 7 compares full LLM fine-tuning against LoRA (Hu et al., 2022). LoRA achieves strong results (37.0% on STI, 41.1% on R4D-Bench vs. 35.4% and 40.5% for full fine-tuning), suggesting that the 4D perceptual knowledge can be injected through low-rank adaptations. However, full fine-tuning achieves the best results (37.6% and 42.2%), likely because 4D perception requires restructuring the LLM's internal representations in ways that LoRA's low-rank constraint limits.

Training hyperparameters:

Initial learning rate: $1 \times 10^{-5}$ (1e-5), which is relatively low — appropriate for fine-tuning a pre-trained model where you want to preserve existing capabilities while adding new ones.
Learning rate schedule: Cosine decay with a warmup ratio of 0.03 (3% of total training steps spent linearly increasing from 0 to the initial learning rate, then cosine decay to near zero).
Batch size: 1,024 total (across 8 nodes, each with multiple NVIDIA A100-SXM4-80GB GPUs, AMD EPYC 7J13 64-Core Processor CPUs).
Training epochs: 5 epochs over approximately 12 hours.
Optimizer: AdamW (implied by standard practice; the paper does not explicitly name the optimizer but follows standard MLLM fine-tuning protocols).
Training data mixture: RoboFAC (simulated robotic arm videos, ~10k videos, ~65k conversations), VSTI-Bench training split (indoor scanning videos, ~1.2k videos, ~130k QA pairs), Wolf NuScenes subset (autonomous driving videos, ~5k videos, ~15k QA pairs generated by Llama-3.1-70B-Instruct), and SAT (simulated dynamic images, ~190k images, ~170k QA pairs). The incremental ablation in Table A4 shows each dataset adds value, with the largest gains from VSTI-Bench (+1.6% STI, +1.5% R4D-Bench over Zero-shot) and RoboFAC (additional +1.0% STI, +1.2% R4D-Bench when added to VSTI+Wolf).

Loss balancing rationale: The hyperparameters $\alpha = 0.5$ and $\beta = 0.1$ (from Equation A8) reflect the relative importance and gradient magnitudes of each loss. SFT loss is implicitly weighted at 1.0 as the primary objective. The latent distillation loss at 0.5 is strong enough to meaningfully shape the LLM's hidden states but not so strong that it overpowers the SFT objective (which would cause the model to become a good 4D feature extractor at the expense of question-answering accuracy). The explicit distillation loss at 0.1 is weaker — it provides a "nudge" toward accurate explicit predictions without dominating, reflecting that the explicit signals are lower-dimensional than the latent features and that their primary role is to ensure the latent features encode semantically meaningful geometry.

Summary of Design Choices and Their Justifications

Training-only 4D perception modules over architectural modifications: avoids inference cost, making 4D-RGPT as efficient as the base MLLM at deployment. The 4D knowledge is "compiled into" the LLM's weights through distillation.
Dual-branch distillation (latent + explicit) over single-branch: latent distillation provides rich, high-dimensional supervision on abstract 4D features; explicit distillation ensures those features encode semantically correct depth, flow, and motion. Table 5 confirms their complementarity (+2.3% for LD-only, +1.9% for ED-only, +4.3% for combined on R4D-Bench).
Smooth-L1 distance for both distillation losses over L2 or L1: provides precise alignment for well-matched feature dimensions while remaining robust to dimensions where the student fundamentally cannot match the teacher.
Frozen teacher decoders in the explicit branch over trainable decoders: ensures that improvements in explicit predictions must come from improvements in latent features, creating tight coupling between the two distillation objectives.
Frozen vision encoder over full fine-tuning: preserves pre-trained visual features that generalize across domains; Table 7 shows tuning the vision encoder degrades performance.
Sinusoidal TPE over burned-in timestamps or text prompts: integrates temporal information directly into frame features without cluttering visual input or requiring the model to parse textual metadata; provides continuous, generalizable time representations.
Five training epochs over longer training: Table 5 and Figure 5 show that 4D signals are learned progressively; five epochs appears sufficient for the distillation to converge without overfitting to the relatively small 4D VQA datasets.
The specific modality weights ( $\lambda_{\tt depth}=1.0$ , $\lambda_{\tt flow}=0.1$ , $\lambda_{\tt motion}=0.05$ , $\lambda_{\tt camray}=0.05$ ) reflect the relative importance of each signal for downstream 4D reasoning: depth is the most fundamental for metric spatial understanding; flow and motion provide motion information that depth alone cannot; camera rays are scene-independent and thus easier to learn.

4. Key Insights and Innovations

Innovation 1: 4D Understanding Is Bottlenecked by 4D Perception — A Diagnosis That Redirects the Research Agenda

The paper's most fundamental conceptual move is not a new architecture or loss function, but a diagnosis of the root cause of MLLMs' failure on region-level 4D VQA. The paper argues that the dominant approaches to spatial-temporal reasoning — text-only SFT/RL and architectural integration of external 3D modules — are solving the wrong problem. They treat 4D understanding as a high-level reasoning challenge, where the model needs to learn to think about depth, motion, and temporal dynamics. The paper's counter-claim is that the bottleneck is perceptual, not cognitive: MLLMs fail at 4D VQA not because they cannot reason about 4D information, but because they never extract that information from the visual input in the first place.

This distinction matters because it implies that the previous research agenda — developing better SFT datasets, more sophisticated RL reward functions, or attaching more powerful external 3D modules — is targeting the wrong stage of the pipeline. You cannot reason about depth values you never computed, or track object motion you never perceived. The paper's diagnosis is that text-only supervision is underdetermined: there are too many ways to produce the correct text answer that do not require genuine 4D perception. A model can learn dataset-specific heuristics, exploit linguistic cues in the question, or make educated guesses based on 2D appearance — all without building internal representations of depth or motion. The SFT loss provides no signal to distinguish a model that genuinely computed the car's 3D displacement from one that memorized that "cars in this dataset typically move at 5 m/s."

What makes this diagnosis novel, rather than obvious, is that it is empirically falsifiable in a way prior work did not test. The paper's ablation in Table 4 provides the crucial evidence: applying SFT alone to the same training data (the "4D-SFT" row) yields only +0.9% on STI-Bench and +2.7% on R4D-Bench over zero-shot, compared to +3.8% and +4.3% with P4D distillation. The training data, question formats, and base model are identical — the only difference is whether the model receives direct perceptual supervision during training. If 4D understanding were purely a reasoning problem that could be solved with better text supervision, SFT alone would close the gap. The fact that it doesn't is direct evidence that the perceptual bottleneck is real and the distillation approach addresses it specifically.

This diagnosis also explains the contradictory signals from prior work that the paper's introduction highlights. SpatialReasoner, ViLaSR, and SpaceR — all trained with RL on text-based spatial reasoning — show competitive performance on non-region 4D benchmarks (Table 2) but collapse on region-level 4D benchmarks (Table 3). SpaceR drops from 37.0% on STI-Bench to 37.0% on R4D-Bench — essentially no gain over the NVILA-Lite zero-shot baseline for region-level tasks. The paper's diagnosis explains this: RL-trained models have learned high-level spatial reasoning strategies that work on scene-level questions, but those strategies are not grounded in genuine per-object 4D perception. When the question targets a specific region — requiring the model to isolate that object's depth and motion from the rest of the scene — the lack of perceptual grounding becomes catastrophic. The False Positive category in R4D-Bench (Table 3) is particularly revealing: ViLaSR achieves only 3.7% (vs. 40.2% for 4D-RGPT), suggesting it has learned to confidently hallucinate motion that the perceptual distillation prevents.

The broader significance of this diagnosis is that it reframes the research priority for 4D-capable MLLMs. Rather than improving reasoning architectures or curating larger text datasets, the field should focus on providing perceptual supervision — either through distillation (as in this paper), through architectural integration of perception modules (but with explicit training signals, not just feature concatenation), or through multi-task training on dense 4D prediction tasks. The paper's finding that even a lightweight 3-layer MLP decoder trained with distillation losses substantially improves reasoning performance suggests that the perceptual bottleneck is both real and addressable with relatively simple mechanisms, provided the training signal is present.

Innovation 2: Training-Only Perceptual Distillation as a Capability Injection Mechanism

The paper's second distinctive contribution is the concept of training-only perceptual modules as a mechanism for capability injection — the idea that an MLLM can be taught to internally represent expert-level perceptual information during training, and that this capability persists after the training-specific modules are removed, improving downstream reasoning at zero additional inference cost.

This is not simply "knowledge distillation applied to MLLMs" — that would be an incremental application of a well-known technique. What makes it conceptually novel is the asymmetry between training and inference: the model is trained to produce explicit, interpretable perceptual outputs (depth maps, flow fields, motion masks) that it will never produce at inference time. The distillation losses teach the LLM to restructure its internal hidden states so that they encode 4D structure, and this restructuring is what improves the model's text answers. The perceptual predictions are a scaffold — they exist only to provide a training signal that shapes the LLM's representations, and are discarded once that shaping is complete.

This is qualitatively different from prior approaches that integrate external models into the MLLM architecture. Methods like VGLLM, SDVLM, or SR3D (Section 2.1) attach external 3D modules and feed their outputs into the MLLM as additional features. The external model runs at inference time, contributing to the model's reasoning on every query. This means: (a) inference cost increases with every query, (b) the MLLM never truly internalizes the perceptual capability — it relies on the external module as a crutch, and (c) the external module's limitations become the system's limitations (e.g., static-only depth estimators prevent dynamic 4D understanding). The paper's approach inverts this: the MLLM absorbs the perceptual capability during training, becoming self-sufficient at inference time, while the external expert only serves as a teacher.

The paper's evidence that this absorption genuinely occurs — not just that the distillation loss improves a correlated metric — comes from two sources. First, the visualization in Figure 5 and Figure A20: the student's predicted depth maps evolve from "barely meaningful" grid-like artifacts (which the authors attribute to the LLM's tokenization structure) to smooth, scene-representative depth predictions as training proceeds. This shows that the LLM's hidden states are being restructured to contain metric 3D information that the frozen decoder can successfully interpret — the representation change is real and visible. Second, the architectural ablation in Table 7: tuning only the projector and LLM (keeping the vision encoder frozen) achieves the best results, while tuning the vision encoder degrades performance. This confirms that the 4D knowledge is being stored in the LLM's transformer layers — the component that persists at inference — not in the vision encoder or the training-only decoder.

The practical significance of this innovation for deployment is immediate: you get the benefit of a dedicated 4D perception system without paying its computational cost at runtime. For applications like autonomous driving or real-time video understanding, where latency budgets are tight, this is the difference between a research prototype and a deployable system. The paper's 8B model with P4D distillation outperforms GPT-4o on STI-Bench (37.6% vs. 34.8%) and approaches it on R4D-Bench (42.2% vs. 42.8%), while running entirely locally with no external model dependencies — a practical demonstration that training-only distillation can close the gap with much larger proprietary systems.

However, this innovation has a fundamental limitation that the paper does not fully explore: the distillation can only transfer what the teacher knows. If the teacher's 4D perception is imperfect — as all real perception systems are — the student will inherit those imperfections. The paper's results on the hardest question categories (e.g., the Rotational category in R4D-Bench, where 4D-RGPT-8B achieves only 43.1% vs. GPT-4o's 49.1%) suggest that the distilled perceptual knowledge has boundaries. Whether these boundaries come from the teacher's limitations, the distillation process, or the base MLLM's capacity is an open question that the paper's experiments cannot disentangle.

Innovation 3: Explicit and Latent Distillation Are Complementary Axes of Perceptual Supervision

The third conceptual contribution is the identification that latent and explicit perceptual supervision are complementary — and that their combination provides gains neither achieves alone. This is not a trivial observation about "more supervision is better"; it reveals something about the nature of the perceptual bottleneck that the paper's diagnosis (Innovation 1) identifies.

Latent distillation (aligning the student's intermediate 4D features with the teacher's) provides rich, high-dimensional supervision on the model's internal representations. It says: "your hidden states should look like those of a dedicated 4D perception model." But this supervision is abstract — it constrains the representation space without specifying what those representations should encode. A student could, in principle, produce latent features that match the teacher's in a distributional sense (similar statistics, similar geometry) without actually encoding the same semantic content. The latent loss would be low, but the model might still lack genuine 4D perception.

Explicit distillation (aligning the student's decoded depth, flow, motion, and camera ray predictions with the teacher's) provides sparse, interpretable, low-dimensional supervision. It says: "your hidden states should decode to these specific depth values, these specific flow vectors." This supervision is semantically grounded — it directly penalizes specific perceptual errors. But it is also limited: the explicit signals are lower-dimensional than the latent features, and they capture only the four specific modalities the paper selected. A student could learn to produce accurate depth and flow while missing other 4D structure that the teacher's latent features encode — the explicit loss would be low, but the model's 4D understanding would be incomplete.

The paper's key finding in Table 5 is that neither branch alone achieves the full gain of their combination. LD-Only improves R4D-Bench by +2.3% over Zero-shot. ED-Only improves by +1.9%. LD+ED improves by +4.3% — roughly the sum of the individual gains. This additivity suggests that the two branches address different aspects of the perceptual bottleneck: latent distillation shapes the representation space, explicit distillation grounds it in semantic content. The fact that they are approximately additive rather than redundant (which would produce sub-additive gains) is evidence that the perceptual bottleneck has both a representational component (the LLM's hidden states must be restructured to accommodate 4D information) and a semantic component (the restructured states must encode the right 4D quantities).

The paper's design choice to use frozen teacher decoders for the explicit branch is the mechanism that enforces this complementarity. Because the decoders are frozen, the only way for the student to improve its explicit predictions is to improve its latent features — the decoders cannot adapt to produce correct-looking outputs from incorrect latent features. This creates a tight coupling: explicit distillation provides a semantic training signal that propagates back through the frozen decoders (which contribute no learnable parameters) and into the latent features, where it complements the abstract representational signal from latent distillation. If the decoders were trainable, they could "cheat" by learning to map poor latent features to reasonable explicit predictions, decoupling the two branches and reducing their complementarity.

The broader implication of this finding extends beyond 4D perception. It suggests a general principle for capability injection through distillation: when distilling a complex expert capability (not just a classifier or a representation), dual-branch supervision — one abstract and high-dimensional (shaping the representation space), one concrete and low-dimensional (grounding the semantic content) — may be more effective than either alone. This is a design pattern that could apply to distilling other expert capabilities into MLLMs: physical reasoning (abstract latent physics features + explicit force/pressure predictions), navigation (abstract spatial representations + explicit occupancy/waypoint predictions), or manipulation (abstract affordance features + explicit grasp pose predictions).

Innovation 4: Timestamp Positional Encoding as a Diagnostic Tool Revealing MLLMs' Temporal Blindness

The fourth contribution is less about the TPE mechanism itself (which is a straightforward application of sinusoidal positional encodings to timestamps) and more about what the TPE ablation reveals about MLLMs' temporal perception capabilities. The paper's controlled experiment with TimeBench (Figure A15, Table A3) is a diagnostic contribution: it demonstrates that MLLMs are functionally blind to time duration when no explicit temporal cues are provided, validating a suspicion that prior work had not systematically tested.

The TimeBench results are striking: without explicit temporal cues, the zero-shot NVILA-Lite-8B achieves only 22.7% accuracy on questions like "How many seconds have passed in the input video?" — barely above random guessing (20%). Even with P4D's full perceptual distillation (depth, flow, motion, camera rays), accuracy only rises to 30.1%. This means that even after learning to perceive depth and motion from video, the model fundamentally does not know how much time the video spans. It can track that an object moved, and estimate how far it moved in 3D, but cannot compute speed because it lacks the temporal denominator.

This is a diagnostic finding because it isolates temporal perception as a capability separable from 4D spatial perception. The model can learn depth (spatial), can learn motion (spatial change), but cannot learn time (temporal scale) from the same visual input. Why? Because temporal scale is not visible in the pixel content — it is metadata. Two videos with identical visual content but different frame rates represent different absolute time durations, and the model has no way to distinguish them from pixel information alone. This is fundamentally different from depth or flow, which are visible in the image structure — you can estimate depth from monocular cues, estimate flow from pixel displacement. Time is invisible.

The paper's response to this diagnosis — TPE — is a targeted fix that addresses exactly this gap: inject the metadata that the model cannot infer from pixels directly into the feature stream. What makes this more than an engineering tweak is the comparison with alternative temporal cue methods (Table 6 and Table A3). Burned-in timestamps (P4D+mark) achieve 95.3% on TimeBench — the model can clearly read and use explicit time text. Textual time prompts (P4D+prompt) achieve 98.0% — the model can parse and apply temporal metadata from the question. But both methods provide only marginal gains on actual 4D VQA benchmarks (+0.3% and +1.1% on R4D-Bench respectively). TPE, by contrast, achieves +2.8% on R4D-Bench despite providing no direct temporal information that a human could read.

Why does TPE outperform more explicit temporal cues on actual 4D VQA? The paper's implicit explanation is that temporal information needs to be integrated with visual processing at a low level — not provided as high-level text that the model must learn to associate with specific frames. Burned-in timestamps are visually salient but compete for attention with the actual scene content and region markers. Text prompts provide global temporal context but require the model to maintain and apply that context across frames — a known weakness of long-context MLLMs. TPE, by encoding time as a continuous signal added directly to frame features, allows the model to learn temporal dynamics as part of its visual processing pipeline — time becomes a dimension of the visual representation, not a separate piece of metadata to be consulted.

This finding has implications beyond the specific TPE mechanism. It suggests that MLLMs need temporal perception to be "compiled into" their visual processing, not provided as declarative knowledge. This aligns with how biological vision systems process time — temporal information is encoded at low levels of the visual hierarchy, not reasoned about symbolically. The failure of text-based temporal cues to substantially improve 4D VQA performance, despite near-perfect performance on explicit time-reading tasks, indicates that the reasoning bottleneck is in the integration of temporal information with visual processing, not in the availability of temporal information.

Innovation 5: R4D-Bench Operationalizes "Region-Level 4D Understanding" as a Distinct Capability

The paper's final distinctive contribution is the construction and release of R4D-Bench as an operationalization of region-level 4D understanding — a benchmark that makes precise what prior work had left vague. "4D understanding" and "region grounding" are terms that appear in prior literature, but no existing benchmark requires both simultaneously in dynamic video contexts. R4D-Bench defines this compound capability through its curation pipeline (Figure 3) and its nine-category question taxonomy, creating a measurement instrument that future work can use and that the field needed.

What makes this benchmark construction intellectually significant — beyond the practical value of a new evaluation dataset — is how it reveals capability gaps that aggregate benchmarks hide. The paper's results in Table 3 show that different models fail in different ways on different question categories, and these failure patterns are diagnostic of underlying capability limitations:

ViLaSR's catastrophic failure on False Positive questions (3.7% vs. 4D-RGPT's 40.2%): ViLaSR was trained with RL on spatial reasoning, and the paper's results suggest it has learned to be overconfident in asserting motion — when asked a trick question about non-existent motion, it confidently hallucinates an answer. The perceptual distillation in 4D-RGPT prevents this by grounding motion assertions in actual flow and motion predictions.
SpaceR's surprisingly weak region-based spatial relationship performance (41.5% vs. Qwen2.5-VL's 48.8%): Despite being RL-optimized for spatial reasoning, SpaceR underperforms a generalist MLLM on spatial relationship questions that involve specific regions. This suggests that RL-trained spatial reasoning may be "global" — optimized for scene-level spatial descriptions — and fragile when forced to reason about the spatial relationship between two specific, user-designated objects.
GPT-4o's dominance on Rotational questions (49.1%) and Collapse on Speed & Acceleration (27.0%): The proprietary model understands rotational motion concepts but cannot compute metric speeds — exactly the pattern expected from a model with strong semantic knowledge but weak metric 3D perception. 4D-RGPT inverts this pattern (43.1% Rotational, 50.9% Speed & Acceleration) — the distillation provides metric perception at the cost of some semantic breadth.

These category-level failure patterns would be invisible in an aggregate benchmark that reported only a single accuracy number. By decomposing region-level 4D understanding into nine distinct question types spanning static and dynamic capabilities, R4D-Bench enables the kind of capability-level diagnosis that the paper's own diagnosis (Innovation 1) argues is necessary. It transforms "does this model understand 4D scenes?" from a yes/no question into a profile of specific perceptual and reasoning capabilities — some of which a model may possess and others it may lack.

The benchmark's curation methodology is also significant because it addresses a systematic bias in prior benchmarks. Existing 3D/4D VQA benchmarks (Table 1) either use static images (SAT-real, MMSI-Bench, OmniSpatial) — which cannot test temporal reasoning — or use dynamic videos without region prompting (STI-Bench, VLM4D-real) — which cannot test whether the model's 4D understanding is grounded to specific objects. The paper's hybrid automated-and-human-verified pipeline (keyword extraction → detection/segmentation → Set-of-Marks → MLLM matching → human verification) is designed to retrofit existing benchmarks with region prompting while maintaining annotation quality through human oversight. This provides a template for how the field can convert non-region benchmarks into region-level benchmarks, addressing the specific gap the paper identifies.

A limitation the paper does not discuss is that the benchmark inherits biases from its source benchmarks (STI-Bench and VLM4D). If those benchmarks have systematic biases — towards certain types of scenes, certain distributions of motion, certain object categories — R4D-Bench will inherit those biases. The paper's category distribution for VLM4D-sourced questions (61.3% Translational, 10.2% Rotational, 15.4% Counting, 13.1% False Positive) is broadly consistent with VLM4D's official distribution but introduces its own skew through the human verification and filtering process. Whether the benchmark's difficulty distribution is representative of real-world 4D VQA applications is an open question that deployment experience will answer.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation is conducted on the newly proposed R4D-Bench (our benchmark), consisting of 1,517 region-prompted multiple-choice questions across 780 dynamic videos sourced from STI-Bench and VLM4D, split into static (418 VQAs) and dynamic (1,098 VQAs) subsets covering 9 question categories (Section 5). Additional evaluation is performed on six existing non-region 3D/4D VQA benchmarks: STI-Bench (Li et al., 2025), VLM4D-real (Zhou et al., 2025), MMSI-Bench (Yang et al., 2025), OmniSpatial (Jia et al., 2025), SAT (Ray et al., 2024), and VSTI-Bench (Fan et al., 2025), with STI-Bench, VLM4D-real, OmniSpatial, and MMSI-Bench designated as testing-only benchmarks disjoint from training data (Section 6.1).
Base model(s). The primary architecture uses NVILA-Lite-8B (Liu et al., 2025), a state-of-the-art open-source MLLM with SigLIP as the vision encoder ( $\bm{\mathsfit{E}}_{\tt V}$ ), a 2-layer MLP projector with 3×3 downsampling, and Qwen2 as the LLM backbone (Section 6.1, Appendix A1.1). Ablation experiments also evaluate NVILA-Lite-15B (Appendix A3, Tables A1–A2). The model family is chosen for its competitive video understanding capabilities and efficiency (3×3 vs. 2×2 downsampling in the projector compared to full NVILA), providing a strong baseline that leaves room for improvement through 4D distillation.
Metrics. All evaluations use multiple-choice accuracy (%), defined as the fraction of questions for which the model selects the correct answer option. For VSTI-Bench numerical questions specifically, relative accuracy is reported instead (the paper does not provide the exact formula in the main text, but it is described as a variant that accounts for numerical tolerance). All metrics are reported as percentages with higher values indicating better performance.
Baselines. The paper compares against a comprehensive set of models spanning three categories. Proprietary MLLMs: GPT-4o (OpenAI, 2024), GPT-5 (OpenAI, 2025), Gemini-2.5-Pro, and Gemini-1.5-Pro (Team, 2024; Comanici et al., 2025). Open-source general MLLMs: Qwen2.5-VL-7B (Alibaba, 2025), InternVL2.5-8B (Chen et al., 2024), VideoLLaMA3-7B (Zhang et al., 2025), LLaVA-Video-7B (Zhang et al., 2024), LLaVA-OneVision-7B (Li et al., 2024), LLaVA-NeXT-Video-7B (Liu et al., 2024), and NVILA-Lite-8B (Liu et al., 2025) as the direct zero-shot baseline. 3D/4D specialized MLLMs: VLM-3R-7B (Fan et al., 2025), LLaVA-Video-7B + SAT (Ray et al., 2024), ViLaSR-7B (Wu et al., 2025), SpatialReasoner-7B (Ma et al., 2025), and SpaceR-7B (Ouyang et al., 2025) — the latter three explicitly noted as using RL training. For R4D-Bench specifically, all open-source models use identical frame sampling (16 frames) and Set-of-Marks (SoM) for region indication, ensuring fair comparison.
Generation budget / compute accounting. The paper does not measure compute in FLOPs or generations as in the example paper on test-time compute scaling. Instead, the central efficiency claim — that P4D incurs "no additional inference cost" — is supported through architectural design rather than FLOPs accounting. The training-only 4D perception decoder ( $\bm{\mathsfit{D}}_{\tt 4DP}$ ) and prediction heads ( $\bm{\mathsfit{D}}_m$ ) are discarded at inference time, meaning the deployed model is architecturally identical to the base NVILA-Lite-8B. The paper validates this claim in Table 4 by comparing against 4D-Concat and 4D-PE — alternative designs that require computing the teacher's 4D features at inference time — and showing that P4D matches or exceeds their performance while avoiding inference overhead. No wall-clock latency measurements or FLOPs comparisons are reported.
Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals/error bars on any results. All evaluations are single-pass on the standard test splits of each benchmark. For R4D-Bench, the human verification stage (Figure 3e) serves as a quality control mechanism rather than a statistical protocol — human annotators verify and correct the automated region matching, with ambiguous or ungroundable questions filtered out. The final benchmark of 1,517 questions is a static test set against which all models are evaluated once.

Main Quantitative Results

Non-Region-Level 3D/4D VQA (Table 2)

The headline result is that 4D-RGPT-8B achieves an average improvement of +5.3% over the NVILA-Lite-8B zero-shot baseline across six benchmarks, with particularly large gains on VSTI-Bench (+13.9%) and VLM4D (+6.2%). Specifically, 4D-RGPT-8B attains 37.6% on STI-Bench (vs. 33.8% for NVILA-Lite-8B), 52.7% on VLM4D-real (vs. 46.5%), 33.3% on MMSI-Bench (vs. 31.3%), 40.4% on OmniSpatial (vs. 37.2%), 64.7% on SAT (vs. 62.0%), and 59.1% on VSTI-Bench (vs. 45.2%).

Compared to models of similar scale (7–8B parameters), 4D-RGPT-8B achieves state-of-the-art performance among open-source MLLMs. It outperforms all listed open-source 7–8B models on every benchmark, and on STI-Bench (37.6%), it surpasses even the larger Gemini-2.5-Pro (41.4%) is not matched — the paper claims "competitive performance with GPT-4o" (34.8% on STI-Bench, which 4D-RGPT exceeds at 37.6%). On VSTI-Bench, the +13.9% gain over the NVILA-Lite baseline is the largest single-benchmark improvement, pushing 4D-RGPT to 59.1% — competitive with the specialized VLM-3R-7B (58.8%) and substantially ahead of generalist MLLMs like GPT-4o (38.2%).

The paper explicitly notes that SpatialReasoner, ViLaSR, and SpaceR are all RL-trained to boost accuracy, yet 4D-RGPT — trained only with SFT + distillation — matches or exceeds them on most benchmarks. For instance, on STI-Bench, SpaceR achieves 37.0% (RL-trained) while 4D-RGPT achieves 37.6% (SFT + distillation); on VLM4D-real, ViLaSR achieves 46.9% while 4D-RGPT achieves 52.7%.

Scale consistency (Table A1): The 4D-RGPT-15B variant shows consistent improvements over its NVILA-Lite-15B baseline, with gains of +3.9% on STI-Bench, +8.6% on VLM4D-real, +2.2% on MMSI-Bench, +1.7% on OmniSpatial, +2.6% on SAT, and +16.2% on VSTI-Bench. The large VSTI-Bench gain persists across scales, suggesting that distillation benefits are not specific to the 8B architecture.

Region-Level 4D VQA on R4D-Bench (Table 3)

The headline result is that 4D-RGPT-8B achieves 42.2% overall accuracy on R4D-Bench, a +4.3% improvement over the NVILA-Lite-8B zero-shot baseline (37.9%) and the best performance among all open-source MLLMs, outperforming the next-best (Qwen2.5-VL-7B at 40.6%) by 1.6%.

On the static split (418 VQAs), 4D-RGPT achieves 32.9% vs. 29.1% for the baseline (+3.8%), with the largest category-level gains in Dimension Measurement (+6.1%, from 20.2% to 26.3%) and Spatial Relationship (+5.9%, from 46.3% to 52.2%). On the dynamic split (1,098 VQAs), 4D-RGPT achieves 45.7% vs. 41.3% for the baseline (+4.4%), with the largest gains in Speed & Acceleration (+5.0%, from 45.9% to 50.9%) and Displacement & Path Length (+6.8%, from 32.1% to 38.9%).

The category-level breakdown reveals non-uniform improvements and several notable patterns:

Strongest gains: Displacement & Path Length (+6.8%), Translational (+6.8%), Dimension Measurement (+6.1%), Spatial Relationship (+5.9%), and Speed & Acceleration (+5.0%) — categories that directly require metric 4D perception (depth for dimension/displacement, temporal perception for speed, 3D spatial reasoning for relationships). This is consistent with the paper's claim that 4D perceptual distillation specifically improves capabilities that require low-level 4D signal extraction.
Modest gains: 3D Video Grounding (+1.2%), Rotational (+1.6%), and Counting (+0.5%) — categories where the improvement is positive but smaller. For Rotational and Counting, the baseline NVILA-Lite-8B already achieves relatively strong performance (41.5% and 39.6% respectively), leaving less room for improvement; alternatively, these categories may rely more on semantic understanding of motion types and object persistence than on precise metric perception.
Negative result: False Positive shows a -0.5% decline (from 40.7% to 40.2%). This is a marginal change within expected noise, but it is notable that perceptual distillation does not improve — and may slightly degrade — the ability to detect when described motion is absent. The paper does not discuss this result specifically, but it could indicate that learning to predict motion signals makes the model more prone to "seeing" motion where none exists, a subtle form of perceptual hallucination.

Comparison with proprietary models: GPT-4o achieves 42.8% overall on R4D-Bench, just 0.6% ahead of 4D-RGPT-8B (42.2%). However, the category-level profiles differ sharply. GPT-4o dominates on Rotational (49.1% vs. 43.1%), 3D Video Grounding (30.7% vs. 35.1%), and False Positive (54.1% vs. 40.2%), while 4D-RGPT dominates on Speed & Acceleration (50.9% vs. 27.0%), Displacement & Path Length (38.9% vs. 10.7%), Translational (48.7% vs. 51.8%), and Dimension Measurement (26.3% vs. 26.8%). This reveals a capability inversion: the proprietary model excels at semantic understanding of motion types (rotational vs. translational) and at rejecting false premises, while the distilled model excels at metric 4D reasoning (speeds, distances, dimensions) — precisely what the perceptual distillation is designed to provide.

Cross-model pattern on region grounding: The paper notes that SpaceR-7B, despite achieving 37.0% on STI-Bench (Table 2, third-highest among open-source 7B models), drops to 37.0% on R4D-Bench (Table 3, third-lowest among open-source models). The paper interprets this as evidence that RL-based training for spatial reasoning optimizes for non-region VQA at the expense of region understanding — a negative transfer effect that P4D's distillation avoids by grounding 4D knowledge in pixel-level perception rather than high-level text reasoning.

Scale consistency (Table A2): 4D-RGPT-15B achieves 43.0% overall on R4D-Bench (+3.3% over NVILA-Lite-15B at 39.7%), with static split improving from 31.7% to 35.8% (+4.1%) and dynamic split improving from 42.7% to 45.7% (+3.0%). The gains are somewhat smaller than the 8B version's +4.3%, suggesting either diminishing returns from distillation at larger scales or that the larger base model already captures more 4D perceptual information implicitly.

Alternative Strategy Comparison (Table 4)

The paper compares P4D against three alternative approaches for incorporating 4D knowledge into MLLMs, using the same training data and base model:

4D-SFT (SFT only, no distillation): Achieves 34.7% on STI-Bench (+0.9% over Zero-shot's 33.8%) and 40.1% on R4D-Bench (+2.2% over Zero-shot's 37.9%). The static split improves from 29.1% to 32.2% (+3.1%), while the dynamic split improves from 41.3% to 43.8% (+2.5%). This demonstrates that additional in-domain SFT data helps, but substantially less than perceptual distillation.
4D-Concat (concatenate teacher features): Achieves 34.8% on STI-Bench (+1.0%) and 39.5% on R4D-Bench (+1.6%). This requires additional projector training and incurs inference cost (must compute teacher features at test time), yet underperforms P4D by 2.8% on STI-Bench and 2.7% on R4D-Bench. The paper argues this demonstrates that simply providing 4D features as input is less effective than forcing the model to internalize them through distillation.
4D-PE (project teacher features to positional encodings): Achieves only 31.3% on STI-Bench (-2.5% vs. Zero-shot) and 36.0% on R4D-Bench (-1.9%). This approach actively degrades performance, likely because projecting high-dimensional 4D features into low-dimensional positional encodings loses information and disrupts the model's existing positional representations.

P4D achieves 37.6% on STI-Bench (+3.8% over Zero-shot) and 42.2% on R4D-Bench (+4.3%) — substantially outperforming all alternatives while incurring no inference cost.

Ablation Studies and Robustness Checks

Perceptual 4D Distillation modality composition (Table 5): Incrementally adding 4D modalities to the distillation loss reveals a monotonic improvement pattern. LD-Only (latent distillation on $\hat{\bm{F}}_{\tt 4D}$ only, no explicit signals) achieves 34.2% on STI-Bench and 40.2% on R4D-Bench (static: 32.0%, dynamic: 43.3%). Adding depth (LD+D) improves R4D-Bench to 40.8% (+0.6%, static: 32.5%, dynamic: 44.0%) but slightly decreases STI-Bench to 33.4% (-0.8%) — an anomaly the paper does not explain, possibly indicating that depth-only explicit supervision over-constrains the latent space for scene-level questions. Adding depth+flow (LD+D+F) recovers and extends gains: 36.2% STI-Bench, 41.9% R4D-Bench (static: 33.1%, dynamic: 45.3%). Adding depth+flow+motion (LD+D+F+M): 36.5% STI-Bench, 42.0% R4D-Bench (static: 33.1%, dynamic: 45.4%). ED-Only (explicit distillation without latent distillation) achieves 35.4% on STI-Bench and 39.8% on R4D-Bench (static: 31.5%, dynamic: 42.9%) — intermediate between Zero-shot and full P4D. The full LD+ED configuration achieves the best results: 37.6% STI-Bench, 42.2% R4D-Bench (static: 32.9%, dynamic: 45.7%). The key finding is that latent distillation alone provides the majority of the gain (+2.3% on R4D-Bench over Zero-shot), explicit distillation alone provides a smaller gain (+1.9%), and their combination is roughly additive (+4.3%) — confirming the complementary nature of the two branches.

Timestamp Positional Encoding (Table 6 and Table A3): The ablation progresses through four configurations. P4D without any temporal cues achieves 34.8% on STI-Bench and 41.0% on R4D-Bench (static: 31.8%, dynamic: 44.5%), improving over Zero-shot's 33.8% and 37.9% but leaving a gap to the full model. P4D+mark (burned-in timestamps) achieves 95.3% on TimeBench (vs. 30.1% for P4D without cues) but only 35.1% on STI-Bench and 41.1% on R4D-Bench — near-identical to P4D without cues on the actual VQA tasks despite near-perfect time-reading ability. P4D+prompt (textual time information in the question) achieves 98.0% on TimeBench and 36.1% on STI-Bench, 41.5% on R4D-Bench — better than marks but still substantially behind TPE. P4D+TPE (the full model) achieves 37.6% on STI-Bench and 42.2% on R4D-Bench (static: 32.9%, dynamic: 45.7%). The finding is that explicit textual temporal cues solve time-reading tasks almost perfectly but transfer poorly to 4D VQA, while TPE — which encodes time as a continuous, non-interpretable signal — transfers effectively (+2.8% on R4D-Bench over no-cue P4D, vs. +0.5% for prompts and +0.1% for marks). This supports the paper's thesis that temporal perception must be integrated at a low level into visual processing, not provided as high-level text.

Architecture freeze/thaw strategy (Table 7): The paper evaluates eight configurations controlling which components are trainable. Tune-All (train vision encoder, projector, and LLM) achieves 34.7% STI-Bench, 38.8% R4D-Bench — only marginally better than Zero-shot (+0.9%, +0.9%) and substantially worse than the optimal configuration, confirming that tuning the vision encoder degrades its pre-trained features. Tune-V (train only vision encoder) achieves 32.3% STI-Bench, 35.8% R4D-Bench — worse than Zero-shot, confirming catastrophic forgetting of visual features. Tune-P (train only projector) achieves 34.3% STI-Bench, 38.6% R4D-Bench — minimal gain, as the projector alone lacks capacity to represent 4D knowledge. Tune-LLM (train only LLM) achieves 35.4% STI-Bench, 40.5% R4D-Bench (static: 32.2%, dynamic: 43.7%) — a substantial gain over Zero-shot, confirming that the LLM is the primary site of 4D knowledge storage. Tune-LLM-LoRA (LLM with LoRA) achieves 37.0% STI-Bench, 41.1% R4D-Bench — strong but slightly below full fine-tuning, suggesting that 4D perceptual knowledge can be largely captured in low-rank adaptations but benefits from full parameter access. Tune-P+LLM-LoRA achieves 36.5% STI-Bench, 41.4% R4D-Bench — slightly better than LLM-LoRA alone but still below full fine-tuning. Tune-P+LLM (the optimal configuration) achieves 37.6% STI-Bench, 42.2% R4D-Bench (static: 32.9%, dynamic: 45.7%). The key finding is that training the projector and LLM together while keeping the vision encoder frozen is optimal, with the vision encoder's frozen state being critical to preserving general visual feature quality.

Incremental training data mixture (Table A4): Adding datasets incrementally shows that VSTI-Bench alone achieves 35.4% STI-Bench and 39.4% R4D-Bench (static: 30.0%, dynamic: 42.9%). Adding Wolf improves to 36.0% STI-Bench and 40.6% R4D-Bench (static: 31.0%, dynamic: 44.2%). Adding RoboFAC improves to 37.0% STI-Bench and 41.8% R4D-Bench (static: 32.2%, dynamic: 45.4%). Adding SAT (image-based) yields a final +0.6% on STI-Bench and +0.4% on R4D-Bench — a small but positive contribution despite SAT being image-based, suggesting that static 3D spatial reasoning transfers modestly to dynamic 4D VQA. Each dataset provides marginal gains, with no single dataset dominating.

4D perception visualization (Figure 5 and Figure A20): The paper provides qualitative evidence that P4D successfully distills 4D perception by visualizing the student's predicted depth maps ( $\hat{\bm{P}}_{\tt depth}$ ) at different training steps. Early in training, the predictions show grid-like artifacts that the authors attribute to the tokenization structure in the LLM's hidden states. As training proceeds, these artifacts progressively disappear and the depth maps capture genuine 3D structure — object boundaries, relative depths, scene layout. Figure A20 extends this to flow and motion predictions, showing similar progressive refinement. This is not a quantitative ablation but serves as qualitative validation that the distillation losses are producing the intended representational change.

PRM teacher vs. student generalization (implied by architecture): The paper does not ablate the specific choice of L4P as the teacher. All results use L4P's specific architecture (40-layer ViT encoder, DPT decoders, jointly trained on depth/flow/motion/camray). Whether a different 4D perception teacher — or separate single-task teachers — would produce comparable results is untested. Given that P4D's design relies on the teacher's latent space being a unified, multi-task 4D representation (so that latent distillation transfers joint 4D structure), single-task teachers might reduce the effectiveness of latent distillation.

Critical Assessment

Claim 1: "4D-RGPT achieves notable improvements on both existing 4D VQA benchmarks and the proposed R4D-Bench benchmark." (Abstract)

What the experiments demonstrate: Table 2 shows consistent improvements across all six non-region benchmarks (+5.3% average over NVILA-Lite-8B). Table 3 shows +4.3% overall improvement on R4D-Bench. Table A1 and A2 replicate these gains at the 15B scale. The improvements are consistent and non-trivial.

What the experiments do not demonstrate: The paper does not compare against a baseline that receives the same training data but without distillation — the 4D-SFT baseline in Table 4 uses the same data and shows only +2.2% on R4D-Bench (vs. +4.3% for P4D), which does isolate the distillation's contribution. However, there is no baseline that controls for the additional compute spent on the teacher during training. The teacher L4P processes every training video — its computation is "free" from the inference perspective but not from the training perspective. A proper compute-matched comparison would give the SFT baseline additional training epochs or more data to match the total FLOPs consumed by the P4D training pipeline (which includes the teacher's forward passes for generating distillation targets). The paper's efficiency claim — "training-only, no inference cost" — is about inference, not training. This is a legitimate design choice but means the reported gains conflate the benefits of distillation with the benefits of additional training supervision (the teacher's outputs).

Conditional nature: The gains vary substantially by question category. On R4D-Bench's False Positive category, 4D-RGPT shows a -0.5% change. On Rotational, +1.6%. On 3D Video Grounding, +1.2%. The "notable improvements" are concentrated in metric 4D perception categories (Speed, Displacement, Dimension, Spatial Relationship) and much weaker in semantic motion understanding categories. The paper's abstract claim of "notable improvements" is true on average but masks this heterogeneity.

Claim 2: "P4D transfers 4D representations from a frozen expert model into 4D-RGPT for comprehensive 4D perception." (Abstract)

What the experiments demonstrate: Figure 5 and Figure A20 provide qualitative evidence that the student's predicted depth, flow, and motion maps improve during training and eventually capture scene structure. This is evidence that the distillation is transferring some 4D perceptual capability. Table 5 shows that both latent and explicit distillation branches contribute to accuracy improvements, consistent with the claim that 4D representations are being transferred.

What the experiments do not demonstrate: The paper does not directly measure the quality of the transferred representations. There is no quantitative metric for the student's depth prediction accuracy (e.g., RMSE against ground-truth depth) or flow prediction accuracy (e.g., endpoint error). The only metrics are downstream VQA accuracy, which is an indirect measure of perceptual quality. A model could improve VQA accuracy without actually producing good depth maps — for example, by learning to associate certain visual patterns with certain answer patterns, without genuine 4D perception. The qualitative visualizations in Figure 5 are suggestive but consist of a single cherry-picked example; they do not constitute systematic evidence.

Moreover, the "comprehensive" claim is challenged by the category-level results. If 4D-RGPT had truly "comprehensive" 4D perception, one would expect strong improvements on False Positive questions (which require detecting that described motion is absent — a direct test of perceptual accuracy) and 3D Video Grounding (which requires precise 3D bounding box prediction). The -0.5% and +1.2% on these categories respectively suggests that the transferred perceptual capability is partial and task-dependent, not comprehensive.

Missing experiment: The paper could have evaluated the student's explicit 4D predictions ( $\hat{\bm{P}}_m$ ) against held-out ground-truth depth/flow data (if available for any of the benchmark videos) to directly measure perceptual transfer quality, rather than relying solely on downstream VQA as a proxy.

Claim 3: "Unlike previous works, P4D contains only training-only modules, incurring no additional inference cost." (Section 1)

What the experiments demonstrate: The architectural design ensures that $\bm{\mathsfit{D}}_{\tt 4DP}$ and $\bm{\mathsfit{D}}_m$ are removed at inference. Table 4 shows that 4D-Concat and 4D-PE — which require inference-time computation of teacher features — underperform P4D despite incurring that extra cost. This is evidence that the distillation approach is both more efficient and more effective than architectural integration for this specific task.

What the experiments do not demonstrate: The paper provides no wall-clock latency measurements, no FLOPs comparisons between inference with and without P4D modules, and no throughput benchmarks. The claim of "no additional inference cost" is an architectural truth — the modules are removed — but the implicit claim is that the distilled model is as fast as the base model. This is trivially true if the architecture is identical, but the paper does not verify that the distillation training process hasn't changed the model's inference behavior in other ways (e.g., changes in generated token length, which affect autoregressive inference cost). On R4D-Bench, all models receive the same multiple-choice format, so output length is controlled, but on open-ended generation tasks (not tested), the distillation could affect verbosity.

Missing comparison: The paper does not compare against a baseline that uses the teacher model at inference time on a subset of questions — for instance, using L4P to compute depth and flow only for questions that require metric 4D reasoning, and routing other questions to the base MLLM. Such a "mixture-of-experts" baseline would trade off inference cost against accuracy in a way that the paper's training-only approach cannot match, and would provide a more meaningful efficiency baseline.

Claim 4: "R4D-Bench is the first benchmark to provide both dynamic video data and region prompts at scale." (Table 1, Section 5)

What the experiments demonstrate: Table 1 clearly documents that prior benchmarks either lack dynamic videos (SAT-real, MMSI-Bench, OmniSpatial: image-based), lack region prompts (STI-Bench, VLM4D-real: dynamic but non-region), or have small scale (VSTI-Bench: 312 static videos with 6K VQAs — the largest prior benchmark by QA count, but all static). R4D-Bench provides 780 dynamic videos with 1,517 region-prompted VQAs across 9 categories. The paper's curation pipeline (Figure 3) is documented, and the human verification step is described (Appendix A2.1).

What the experiments do not demonstrate: The paper does not provide inter-annotator agreement statistics for the human verification step. It states that "human annotators verify and correct any mismatches" (Section 5) and that questions where grounding fails or objects are too ambiguous are "filtered out" (Appendix A2.1), but there is no quantification of how many questions were filtered, how often human annotators disagreed with the automated MLLM matching, or what fraction of questions required correction. Without these statistics, the benchmark's annotation quality is asserted but not demonstrated.

Additionally, the paper does not analyze potential biases introduced by the source benchmarks. R4D-Bench inherits videos from STI-Bench and VLM4D — if those benchmarks have systematic biases in scene types, object categories, motion patterns, or difficulty distribution, R4D-Bench will inherit those biases. The paper provides category distributions (e.g., 61.3% Translational, 10.2% Rotational for VLM4D-sourced questions) but does not analyze whether this distribution reflects real-world 4D VQA use cases.

Missing analysis: A difficulty calibration — showing how human performance, MLLM performance, and random chance relate across the 9 categories — would contextualize the benchmark's scores. For instance, if humans achieve near-ceiling on certain categories, improvements there may be less meaningful than improvements on categories where even humans struggle.

Genuine Weaknesses in the Experimental Design

Single model family, single teacher model. All experiments use NVILA-Lite as the base MLLM and L4P as the teacher. Whether P4D generalizes to other MLLM architectures (e.g., LLaVA-style, InternVL-style) or other 4D perception teachers is completely untested. The paper's claims about the effectiveness of dual-branch distillation may be specific to the NVILA + L4P combination. The consistent gains at 8B and 15B scales (Tables A1, A2) provide some evidence against scale-specificity, but not against architecture-specificity.
No confidence intervals or statistical testing. All results are reported as single-point accuracy values with no error bars, standard deviations, or statistical significance tests. With test sets ranging from 150 (SAT-real) to ~6,000 (VSTI-Bench) questions, the reliability of small differences (e.g., 4D-RGPT's 42.2% vs. GPT-4o's 42.8% on R4D-Bench, a 0.6% gap) is unassessable. The 1,517-question R4D-Bench has a standard error of approximately 1.3% for a model scoring 42% (binomial), meaning the +4.3% gain over the baseline is likely statistically significant, but smaller category-level differences may not be.
No human performance baseline on R4D-Bench. The paper does not report how well humans perform on R4D-Bench. Without a human ceiling, it's unclear whether the 42.2% accuracy achieved by 4D-RGPT represents strong progress or whether the benchmark is simply very difficult. If humans achieve 90%, 4D-RGPT's 42.2% is weak; if humans achieve 50%, 42.2% is strong. This missing baseline makes it impossible to assess the practical significance of the reported numbers.
Training data leakage risk. The paper states that STI-Bench, VLM4D, OmniSpatial, and MMSI-Bench are "testing-only benchmarks and are disjoint from our training data" (Section 6.1). However, R4D-Bench is constructed from STI-Bench and VLM4D videos (Section 5). If the training data includes VSTI-Bench's training split, which may share visual characteristics with STI-Bench (both use indoor scanning videos from ScanNet/ScanNet++), the model may benefit from domain overlap even if exact questions don't leak. The paper does not analyze the visual domain overlap between training and test benchmarks.
Ablation of the Smooth-L1 distance function: The paper uses Smooth-L1 for both latent and explicit distillation but does not ablate this choice against L1, L2, cosine distance, or other alternatives. Given that the loss function choice is critical for distillation effectiveness, this is a notable omission.

Experiments That Would Have Strengthened the Paper

Direct evaluation of distilled 4D perception quality: Run the student's predicted depth/flow/motion maps against ground-truth annotations on a benchmark like Sintel or KITTI (if available for any of the test videos) to quantify how good the transferred perception actually is, independent of VQA accuracy.
Cross-architecture validation: Apply P4D to a non-NVILA base MLLM (e.g., LLaVA-Video-7B) to test whether the gains are architecture-specific.
Cross-teacher validation: Use a different 4D perception model (or separate depth/flow/motion estimators) as the teacher to test whether gains depend on L4P's specific architecture or training.
Human performance on R4D-Bench: A small-scale human evaluation (e.g., 100 randomly sampled questions evaluated by 3 human annotators) to establish a performance ceiling.
Training compute-matched SFT baseline: Increase SFT training epochs or data volume so that the total training FLOPs (including teacher forward passes for P4D) match between the distillation and non-distillation baselines, isolating the benefit of the distillation signal itself from the benefit of additional supervision.
Failure analysis by question category: A systematic analysis of what types of errors 4D-RGPT makes vs. the baseline — does it fail because of perceptual errors (wrong depth → wrong speed) or reasoning errors (correct depth but wrong formula)? This would directly test the paper's core claim that the bottleneck is perceptual, not cognitive.

6. Limitations and Trade-offs

The Difficulty Estimation for Compute-Optimal Scaling Is Prohibitively Expensive

The assumption or constraint. The paper's "compute-optimal" test-time scaling policy selects the best strategy (search algorithm, revision depth, parallel/sequential ratio) per difficulty bin. The predicted (non-oracle) difficulty estimation method requires generating 2048 samples per question and averaging the PRM's final-answer scores, then binning into quintiles. The paper acknowledges this cost explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. In a realistic deployment, the total compute cost equals difficulty estimation plus strategy execution. Generating 2048 samples per question is comparable to or exceeds the largest test-time budgets studied (256–512). The paper's reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the estimation cost. If the 2048-sample estimation step is included, the total cost may exceed that of simply running best-of-N with the full budget — completely negating the claimed efficiency advantage. For a practitioner running batch inference on thousands of questions, this means the headline 4× improvement is an upper bound that cannot be realized without a cheap difficulty estimator, which the paper does not provide.

What evidence exists in the paper. Section 3.2 describes the estimation procedure and its cost. Figures 4 and 8 show compute-optimal scaling curves starting from low budgets (e.g., 4 generations), where the unaccounted 2048-sample estimation cost would dominate the total compute. The paper flags this as "a key avenue for future work" (Section 3.2) but does not develop or evaluate any cheap estimation method. The predicted difficulty bins that use PRM scores (rather than ground-truth answers) reduce the need for labeled data but do not reduce the sample count — 2048 generations are still required.

Mitigation status. The paper explicitly acknowledges this gap and suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). No such model is developed, trained, or evaluated. The limitation is identified but unresolved.

The FLOPs-Matched Pretraining Baseline Is Weakened by Non-Compute-Optimal Scaling

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales only model parameters (by ~14×) while holding training data fixed, following the LLaMA paradigm rather than compute-optimal pretraining where both parameters and data are scaled equally. The paper acknowledges this in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. A ~14× larger model trained with Chinchilla-optimal scaling (Hoffmann et al., 2022) — scaling both parameters and data — would likely outperform a parameter-only-scaled model. This means the pretraining baseline is artificially weak. The paper's key finding that test-time compute with a small model can outperform a ~14× larger model (e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions, Figure 9) may shrink or reverse against a properly compute-optimal larger model. Additionally, the larger model is evaluated with only greedy decoding — no majority voting, no best-of-N, no search. If the larger model were given even a modest test-time compute budget (e.g., best-of-8), the comparison would be far less favorable to test-time compute.

What evidence exists in the paper. Figure 9 and the bar charts in Figure 1 report the FLOPs-matched results. The paper transparently states the parameter-only scaling choice and notes it departs from Hoffmann et al. (2022). The ~14× model's exact training recipe is not detailed, making it impossible to assess how far it is from compute-optimal. The paper does not report what performance the larger model would achieve with its own test-time compute budget, which would be a fairer baseline.

Mitigation status. The paper acknowledges the caveat but does not address it experimentally. The leave-for-future-work framing is honest, but the headline claim that "a smaller model can outperform a 14× larger model" is stronger than the experimental setup justifies — it demonstrates superiority over a particular, non-compute-optimal larger model, not over pretraining in general.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Create Capability

The assumption or constraint. The paper's approach assumes that the base model already produces correct solutions at some non-trivial rate for the problem at hand. Test-time compute amplifies existing capability through better selection (search) or local refinement (revisions), but it cannot create capability where none exists. This is not an assumption the paper hides — it states it explicitly in the Section 7 takeaway box and in its characterization of the hardest difficulty bin.

The consequence. On the hardest questions (difficulty bin 5), all methods fail. Search accuracy hovers at 1–3% regardless of budget (Figure 3, right). Revisions achieve 2–3% regardless of sequential-to-parallel ratio (Figure 7, right). The compute-optimal policy provides zero benefit on these problems because there are no correct solutions in the proposal distribution to find or refine. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for all R values. For a practitioner facing a problem distribution that skews toward genuinely hard or out-of-distribution reasoning tasks, this approach offers no advantage — pretraining remains the only viable path to improve performance. This is a fundamental capability boundary: the method helps only with problems the model "almost" knows how to solve.

What evidence exists in the paper. The difficulty-bin analyses in Figures 3 (right), 7 (right), and 9 provide consistent, unambiguous evidence across all methods and conditions. Bin 5 shows near-flat scaling curves in every setting. The paper is transparent about this limitation.

Mitigation status. The paper does not attempt to address this limitation — it is treated as a fundamental property of test-time compute. The takeaway box in Section 7 states "test-time compute can amplify existing capability but does not create it from nothing." No mitigation is proposed because, within the paper's framework, there is none: the proposal distribution (base model) must contain correct solutions for search or revisions to help.

Verifier Over-Optimization Limits Scaling Even on Problems Within the Model's Reach

The assumption or constraint. The paper's search methods (beam search, lookahead search) rely on the PRM to score partial solutions and guide exploration. This assumes the PRM's scores remain well-calibrated under aggressive optimization — i.e., that solutions the PRM rates highly are genuinely more likely to be correct. When this assumption breaks, the paper calls it verifier over-optimization or reward hacking.

The consequence. Beam search — the strongest optimizer — degrades performance on easy problems at high budgets. In Figure 3 (right), beam search accuracy on difficulty bin 1 actually decreases as the budget increases from 4 to 256 generations, while best-of-N (a weaker optimizer) continues to improve. Lookahead search, the most powerful optimizer (adding k+1 generations per step), paradoxically performs worst overall (Figure 3, left) — it over-optimizes the PRM signal, finding solutions that score highly but are incorrect. Qualitative examples in Appendix M (Figure 29) show degenerate outputs: repetitive low-information steps and overly short solutions that exploit the PRM. The compute-optimal policy mitigates this by routing easy problems to best-of-N (weak optimization) instead of beam search, but over-optimization still limits the scaling ceiling on medium-difficulty problems. In Figure 3 (right, bin 3), beam search curves flatten well before the budget is exhausted. A practitioner cannot simply "throw more compute" at any problem and expect monotonic improvement — there is a verifier-quality-dependent ceiling beyond which additional compute is wasted or harmful.

What evidence exists in the paper. Figure 3 (right) shows beam search degradation on bin 1. Figure 3 (left) shows lookahead search underperforming simpler methods. Appendix M provides qualitative examples of degenerate outputs. The paper identifies this phenomenon as a central bottleneck in Section 5.3 and Section 8.

Mitigation status. The compute-optimal policy partially mitigates this by routing easy problems away from beam search, but it does not solve the underlying verifier robustness problem. On medium problems where beam search is deployed, over-optimization still caps performance. The paper suggests improving verifiers as the key research priority (Section 8) but does not develop or evaluate any methods for doing so. The limitation is well-characterized but unresolved.

Sequential Revisions Are Latency-Bound in Deployment

The assumption or constraint. The paper measures compute in "generations" — the number of complete solutions sampled — which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial: each revision conditions on the previous one, so a chain of K revisions requires K sequential forward passes through the LLM. Parallel best-of-N can run N independent generations simultaneously given sufficient hardware.

The consequence. The compute-optimal policy on easy-to-medium problems favors sequential revisions (Figure 7, right, bins 2–3: higher sequential-to-parallel ratios are optimal). On easy problems at moderate budgets, the optimal allocation might be 64 sequential × 2 parallel — requiring roughly 64× the wall-clock time of 128 parallel samples, even though both use comparable total FLOPs. For latency-sensitive applications (interactive assistants, real-time systems, live video analysis), the sequential-heavy strategies that the compute-optimal policy selects may be impractical regardless of their accuracy advantages. A practitioner deploying this method in a user-facing system may need to trade off accuracy against latency, and the paper's framework provides no guidance for this tradeoff — it optimizes only for total FLOPs, not wall-clock time.

What evidence exists in the paper. Figure 5 (right panel) and Figure 7 illustrate the sequential-to-parallel ratio sweep and the compute-optimal selections. The paper does not report latency measurements or discuss the wall-clock implications of sequential vs. parallel allocation. The revision model's architecture (Section 6.1) makes it clear that revisions are autoregressive and thus serial, but this practical consequence is not analyzed.

Mitigation status. The paper does not address the latency issue at all. The compute-optimal framework optimizes for generation count, not wall-clock time, creating an implicit assumption that latency is not a binding constraint. For practitioners in latency-sensitive regimes, a separate analysis — or a latency-aware extension of the compute-optimal framework — would be needed.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model is trained on trajectories where all in-context answers are incorrect, followed by a correct answer (Section 6.1). This training data construction reflects the intended use case — the model sees incorrect attempts and learns to produce a correct revision. However, it means the model never sees examples of what to do when the current answer is already correct. At inference time, the model may produce a correct answer early in the chain, then encounter it (or a slightly modified version) in context during a subsequent revision step.

The consequence. The paper reports that approximately 38% of correct answers get converted back to incorrect ones during the revision chain (Section 6.1). This is a direct consequence of the training data bias: the model has learned that in-context answers are always wrong and should be "fixed," so it incorrectly revises correct answers into wrong ones. This means that longer revision chains do not monotonically improve — performance can oscillate or degrade. The paper mitigates this by selecting the best answer from anywhere in the chain (via majority voting or verifier-based selection, Figure 5) rather than always taking the final revision, but this is a patch, not a solution. It means that much of the computation spent on later revisions in a chain may be wasted, and that the optimal chain length is limited by this reversion phenomenon.

What evidence exists in the paper. Section 6.1 explicitly reports the 38% figure and describes the mitigation strategy. Figure 6 (left) shows that pass@1 at each step gradually improves but with some fluctuation, consistent with occasional reversions. The ReSTEM experiment (Appendix K, Figure 16) further highlights the fragility of revision training — attempts to optimize the revision model with RL caused performance to degrade substantially, suggesting the training methodology is sensitive and not fully robust.

Mitigation status. The paper mitigates through within-chain selection (majority or verifier) rather than structural fixes to the training procedure. A more principled solution — such as training the model to recognize when no revision is needed, or including correct-in-context examples during training — is not explored. The reversion problem is a direct consequence of the training data construction (edit-distance-based pairing of incorrect and correct solutions post-hoc, Section 6.1) and may be fundamental to this particular revision training recipe.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around 4D-capable MLLMs from a reasoning-centric paradigm to a perception-centric one. The dominant approaches for spatial-temporal understanding — SFT on text data, RL with spatial reasoning rewards, and architectural integration of external 3D modules — all treat the problem as one of teaching the model to think about depth, motion, and time. The paper's central diagnosis, validated by the gap between 4D-SFT (+2.2% on R4D-Bench) and P4D (+4.3%) in Table 4, is that these approaches are solving the wrong problem. Models fail at region-level 4D VQA not because they cannot reason about 4D information, but because they never extract it from visual input in the first place. Text-only supervision is underdetermined — there are too many ways to produce the correct answer without genuine 4D perception — and the SFT loss provides no signal to distinguish a model that computed the car's 3D displacement from one that memorized a dataset-specific heuristic.

This is not merely an architectural contribution. It is a diagnostic reframing: the bottleneck is perceptual, not cognitive, and the research priority should shift from better reasoning strategies to better perceptual supervision. The practical significance is that this reframing redirects effort away from curating ever-larger 4D VQA datasets for SFT (which the 4D-SFT ablation shows yields diminishing returns) and toward developing and distilling better 4D perception teachers. The paper's use of L4P — a frozen, off-the-shelf 4D perception model — as a teacher, and the monotonic gains from adding 4D modalities to the distillation loss (Table 5: LD-Only at +2.3%, LD+D+F+M at +4.1%, LD+ED at +4.3%), suggests that improvements in 4D perception models directly translate to improvements in 4D understanding MLLMs, with minimal architectural changes. This creates a pipeline where progress in 4D computer vision (better depth estimators, better flow estimators, better joint 4D encoders) feeds directly into progress in MLLM capabilities — a virtuous cycle that the field had not previously articulated.

The paper also resolves a contradiction in the existing evidence that was hiding in aggregate benchmarks. Prior work (SpatialReasoner, ViLaSR, SpaceR) showed competitive performance on non-region 4D VQA benchmarks (Table 2) but collapsed on region-level 4D VQA (Table 3). SpaceR drops from 37.0% on STI-Bench to 37.0% on R4D-Bench — essentially zero gain over the NVILA-Lite zero-shot baseline for region-specific questions. The paper's diagnosis explains this inversion: RL-trained spatial reasoning produces "global" strategies that work for scene-level descriptions but are not grounded in per-object 4D perception. When the question targets a specific region — requiring the model to isolate that object's depth and motion from the rest of the scene — the lack of perceptual grounding becomes catastrophic. The False Positive category (Table 3) is the clearest demonstration: ViLaSR achieves only 3.7% (vs. 40.2% for 4D-RGPT), suggesting it has learned to confidently hallucinate motion that perceptual distillation prevents. This reconciliation — that different training paradigms produce different capability profiles that aggregate benchmarks obscure — is a finding that should change how the field evaluates spatial-temporal MLLMs. Aggregate accuracy numbers are insufficient; category-level analysis that separates metric perception from semantic understanding is necessary.

Perhaps the most landscape-shifting methodological contribution is the concept of training-only capability injection. The idea that an MLLM can be taught to internally represent expert-level perceptual information during training, and that this capability persists after the training-specific modules are removed, is qualitatively different from the dominant paradigm of architectural integration. Prior work (VGLLM, SDVLM, SR3D, SAT) attaches external 3D modules to MLLMs and feeds their outputs as additional features at inference time — the external model is a permanent crutch that adds cost to every query. The paper inverts this: the external model is a teacher during training only, and the MLLM absorbs its capability through distillation. Table 4 confirms that this approach is not only more efficient (no inference cost) but also more effective: 4D-Concat, which concatenates teacher features at inference time, achieves only +1.6% on R4D-Bench, compared to P4D's +4.3%. The distillation approach forces the model to internalize 4D knowledge in a way that feature concatenation does not — a finding with implications beyond 4D perception, applicable to any domain where expert models provide signals that MLLMs should absorb rather than consult.

However, the paper also establishes a clear boundary condition on this approach: distillation can only transfer what the teacher knows. The category-level results on R4D-Bench (Table 3) show that 4D-RGPT's gains are concentrated in metric 4D perception categories (Speed & Acceleration: +5.0%, Displacement & Path Length: +6.8%, Dimension Measurement: +6.1%) and weak in semantic motion understanding categories (Rotational: +1.6%, False Positive: -0.5%). This is exactly what one would expect if L4P provides strong depth and flow signals but weaker supervision on motion type classification and motion presence detection. The teacher's capabilities define the ceiling of the student's improvement, and the paper's results suggest this ceiling is both real and task-dependent. This is not a failure — it is a precise characterization of the distillation approach's scope, which future work can use to decide when to apply it.

Finally, the introduction of R4D-Bench changes the evaluation landscape by operationalizing "region-level 4D understanding" as a measurable capability with nine distinct sub-dimensions. Prior to this benchmark, the field had no way to distinguish a model that genuinely tracks objects through 3D space over time from one that uses global scene heuristics to answer spatial questions. R4D-Bench's nine-category taxonomy — spanning static (3D Video Grounding, Dimension Measurement, Spatial Relationship) and dynamic (Translational, Rotational, Counting, False Positive, Speed & Acceleration, Displacement & Path Length) capabilities — provides a diagnostic instrument that the paper uses to reveal capability profiles that aggregate benchmarks hide. The benchmark's hybrid automated-and-human-verified curation pipeline (Figure 3) also provides a template for retrofitting existing non-region benchmarks with region prompting, addressing a systematic gap in the evaluation ecosystem.

Follow-Up Research This Work Enables

Direct quantification of distilled perceptual quality against ground-truth annotations. The paper provides only qualitative visualizations (Figure 5, Figure A20) as evidence that P4D transfers 4D perception, and evaluates the transfer only indirectly through downstream VQA accuracy. A direct experiment would: (1) identify a subset of R4D-Bench or STI-Bench videos that have ground-truth depth or optical flow annotations (e.g., from the original ScanNet/DAVIS sources); (2) run the trained 4D-RGPT's $\hat{\bm{P}}_{\tt depth}$ and $\hat{\bm{P}}_{\tt flow}$ through the frozen teacher decoders (which still exist and can be used for evaluation even if discarded at deployment); (3) compute standard metrics (RMSE for depth, endpoint error for flow) against the ground truth; and (4) correlate perceptual accuracy with downstream VQA accuracy per category. This would answer the paper's central unanswered question: does 4D-RGPT actually learn to perceive depth and motion, or does it learn something correlated with depth and motion that improves VQA without genuine perception? A negative result — showing that VQA improvement is not correlated with perceptual accuracy — would fundamentally challenge the paper's diagnostic claim that the bottleneck is perceptual.

Cross-teacher and cross-architecture stress tests of P4D's generality. The paper uses a single teacher (L4P) and a single MLLM architecture (NVILA-Lite, 8B and 15B). Two experiments would test whether P4D is a general framework or a specific recipe: (1) Replace L4P with separate, state-of-the-art single-task teachers — e.g., DepthAnything V2 for depth, RAFT for optical flow, a separate motion segmentation model — and run P4D with the same latent+explicit distillation. The paper's architecture relies on L4P having a unified latent space that jointly encodes depth, flow, motion, and camera geometry (so latent distillation transfers joint 4D structure). Single-task teachers would produce fragmented latent spaces; if P4D still works, it suggests latent distillation is less critical than the paper claims. (2) Apply P4D to a non-NVILA architecture — e.g., LLaVA-Video-7B or InternVL2.5-8B — with the same teacher and same training data, and measure whether the gains replicate. NVILA-Lite uses Qwen2 as the LLM and SigLIP as the vision encoder; whether other LLM/encoder combinations can similarly absorb 4D perceptual knowledge is an open question. A negative result would correctly scope P4D as an NVILA-specific technique rather than a general framework.

Compute-matched SFT baseline to isolate the distillation signal. The paper's 4D-SFT baseline (Table 4) uses the same training data as P4D but does not control for total training compute. P4D requires the teacher L4P to process every training video — this teacher forward pass consumes FLOPs that the SFT baseline does not incur. A proper comparison would: (1) measure the total training FLOPs for P4D (including teacher forward passes and distillation loss backpropagation); (2) give the SFT baseline additional training epochs or additional data so that its total training FLOPs match; and (3) compare final accuracy. This isolates whether the distillation signal itself provides value beyond additional compute. If the compute-matched SFT baseline closes the gap, it would mean the paper's claimed benefit of distillation is actually a benefit of more training compute — an important negative result that would reframe P4D as a compute-efficient way to incorporate perceptual supervision, rather than a qualitatively different supervision signal.

Dynamic difficulty estimation and adaptive strategy allocation. The paper uses static difficulty bins computed from 2048 samples per question, which is prohibitively expensive. A dynamic approach would: (1) generate a small number of initial samples (e.g., 4–8) per question; (2) use the PRM's score distribution on those samples as a real-time difficulty signal; (3) allocate the remaining budget adaptively — switching to beam search if the difficulty suggests medium-hard, staying with best-of-N if easy, or escalating to a larger model if the difficulty is extreme. This connects directly to the paper's suggestion of "exploration-exploitation tradeoff" in Section 3.2. The experiment would compare this adaptive policy against the static compute-optimal policy (which requires pre-computed difficulty) and best-of-N, measuring both accuracy and total generation cost. A positive result — adaptive matching or approaching the compute-optimal policy without the 2048-sample upfront cost — would make the framework practically deployable. A negative result — adaptive significantly underperforming the oracle policy — would clarify the irreducible cost of difficulty estimation.

Training a revision model that avoids the 38% correct-to-incorrect reversion rate. The paper's revision model suffers from a fundamental training data bias: it only sees incorrect-to-correct trajectories, causing it to incorrectly "fix" already-correct answers 38% of the time. A targeted follow-up would: (1) augment the revision training data with correct-in-context examples, where the target is to recognize that no revision is needed (e.g., the sequence is [correct, correct] with the second occurrence being a copy of the first, or [incorrect, correct, correct] where the model learns to stop); (2) measure the reversion rate and end-of-chain accuracy; and (3) test whether the augmented training reduces the dependence on within-chain selection (majority voting / verifier-based selection), allowing longer revision chains to reliably improve. This experiment would address one of the paper's clearest documented failure modes. A null result — showing that correct-in-context training degrades the model's ability to fix actual errors — would reveal a deeper tension in revision training that the paper's simple incorrect-to-correct recipe avoids.

Category-level failure analysis linking perceptual errors to reasoning errors. The paper claims the bottleneck is perceptual, but never directly demonstrates that 4D-RGPT's errors are caused by failed depth/flow/motion perception rather than failed reasoning about correctly-perceived signals. A systematic analysis would: (1) take 4D-RGPT's incorrect answers on R4D-Bench, categorized by question type; (2) for Speed & Acceleration and Displacement & Path Length questions (where metric perception is directly testable), extract the model's implicit speed/displacement calculation from its chain-of-thought or from probing the LLM's hidden states; (3) determine whether the error originates from incorrect depth → incorrect displacement, incorrect temporal perception → incorrect time interval, or correct perception but wrong formula / arithmetic. This would directly validate or challenge the paper's core diagnostic claim. If most errors are reasoning errors despite correct perception, the paper's thesis — that the bottleneck is perceptual — would be substantially weakened, and the research priority would shift back toward reasoning-focused approaches.

Practical Applications and Downstream Use Cases

On-device 4D video understanding for augmented reality and robotics. The paper's training-only distillation approach means the deployed 4D-RGPT model is architecturally identical to the base NVILA-Lite-8B — no external perception modules, no additional inference cost. This makes it suitable for latency-sensitive, compute-constrained deployments where running a separate 4D perception model alongside the MLLM would be infeasible. In augmented reality, a user could point their device at a dynamic scene and ask region-specific 4D questions: "How fast is the car in the red bounding box moving toward me?" or "Has the box on the conveyor belt rotated since it entered the frame?" The model's +5.0% gain on Speed & Acceleration (Table 3) and +6.8% on Displacement & Path Length means measurably more reliable metric 4D answers without requiring cloud offloading. In robotic manipulation, a system watching an assembly video could answer "Did the gripper in region R1 fully close around the component in region R2?" — requiring depth-based spatial relationship tracking over time. The +5.9% gain on Spatial Relationship questions (Table 3) directly supports this use case.

Automated video inspection and monitoring with precise region queries. Industrial inspection and security monitoring often require answering specific, region-targeted questions about recorded video: "Did the package on the conveyor belt move more than 2 meters between timestamps 10s and 15s?" or "Is the person in the marked region moving toward the restricted area?" Existing MLLMs can process video but struggle with metric 4D answers tied to specific regions. 4D-RGPT's gains in Dimension Measurement (+6.1%, Table 3) and Translational motion (+6.8%) directly improve reliability on these tasks. The training-only design means the system can be deployed on existing hardware without additional GPUs for 4D perception — the distillation cost is paid once at training time. The benchmark's False Positive category (where 4D-RGPT achieves 40.2% vs. ViLaSR's 3.7%) is particularly relevant for monitoring applications where false alarms — confidently asserting motion where none exists — are costly, and 4D-RGPT's perceptual grounding reduces this failure mode compared to RL-trained alternatives.

Self-improving data generation pipelines for spatial-temporal reasoning. The paper's finding that P4D improves 4D understanding without architectural changes creates an opportunity for iterative self-improvement: use 4D-RGPT to generate high-quality 4D VQA answers on new, unlabeled dynamic videos, then use those answers (possibly filtered by the teacher's 4D signals as a quality check) as additional SFT data to further fine-tune the model. The teacher L4P provides depth, flow, and motion estimates that can serve as weak supervision for answer correctness — e.g., if the question asks about speed and the model's answer implies a speed that contradicts the teacher's depth and temporal estimates, the answer can be flagged for filtering or correction. This creates a scalable pipeline for expanding 4D VQA training data that doesn't require human annotation. The paper's Table A4 (incremental data mixture) already shows that adding diverse 4D datasets (VSTI-Bench, Wolf, RoboFAC, SAT) each provides marginal gains, suggesting that data diversity matters and that automatically generated data from 4D-RGPT could further expand the training distribution. The 15B model's consistent gains (Tables A1, A2) show the approach scales with model size, so iteratively improving both the model and the training data is a viable path.

When to Prefer This Method

The paper does not explicitly position P4D against a named set of alternatives with clearly articulated tradeoff conditions. The closest it comes is the comparison in Table 4, where P4D is compared against 4D-SFT, 4D-Concat, and 4D-PE. From these comparisons and the broader experimental results, practitioners can derive guidance:

Prefer P4D distillation over SFT-only training when (a) you have access to a pre-trained 4D perception model (or can train one) that produces depth, flow, and motion signals on your target video domain, and (b) your downstream 4D VQA tasks require metric spatial-temporal reasoning (speeds, distances, dimensions) rather than purely semantic motion understanding. The evidence is Table 4: P4D achieves +4.3% on R4D-Bench over Zero-shot vs. +2.2% for 4D-SFT on the same training data, with the gains concentrated in metric perception categories (Table 3). If your task is primarily semantic — classifying motion types, describing events — the gap narrows, and SFT may suffice with less engineering complexity.
Prefer P4D distillation over architectural integration (4D-Concat, 4D-PE) when inference latency or compute is constrained. Table 4 shows 4D-Concat achieves only +1.6% on R4D-Bench (vs. P4D's +4.3%) while requiring the teacher model to run at inference time. P4D provides better accuracy with zero inference overhead. The tradeoff is training cost: P4D requires the teacher's forward passes during training (to generate distillation targets), which increases training time compared to SFT. If training cost is the primary constraint and inference cost is not, 4D-Concat may be a simpler alternative, though it underperforms P4D.
Prefer scaling the base model over applying P4D when your problem distribution includes a substantial fraction of tasks that the base model fundamentally cannot solve (parallel to the paper's difficulty bin 5 concept in the test-time compute example paper — though 4D-RGPT's paper doesn't use difficulty bins, the False Positive and Rotational categories in Table 3 show minimal improvement from distillation). The distillation transfers what the teacher knows; if the teacher's 4D perception is weak on your domain (e.g., unusual camera geometries, transparent objects, extreme motion), the gains will be limited. The paper does not quantify this boundary, but the category-level heterogeneity in Table 3 serves as a warning: gains vary from -0.5% (False Positive) to +6.8% (Displacement & Path Length), and a practitioner should validate on their specific task distribution before committing to the distillation pipeline.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Apollo: An Exploration of Video Understanding in Large Multimodal Models2024· multimodal← LLM-judged similar
- InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions2024· multimodal← in-domain predecessor
- Beyond Language Modeling: An Exploration of Multimodal Pretraining2026· multimodal← similar topic
- Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks2025· multimodal← similar topic
EasyV2V: A High-quality Instruction-based Video Editing Framework2025· multimodal← in-domain predecessor
- Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model2025· multimodal← in-domain predecessor
Few-Step Distillation for Text-to-Image Generation: A Practical Guide2025· diffusion← topic thread
- EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing2025· diffusion← in-domain predecessor
- Black-Box On-Policy Distillation of Large Language Models2025· training-methods← topic thread
- ArcFlow: Unleashing 2-Step Text-to-Image Generation via High-Precision Non-Linear Flow Distillation2026· diffusion← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

SpatialTree: How Spatial Abilities Branch Out in MLLMs2025· multimodal→ in-domain predecessor
- Bridging Your Imagination with Audio-Video Generation via a Unified Director2025· multimodal→ in-domain predecessor
Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision2025· training-methods→ topic thread
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward2025· training-methods→ in-domain predecessor
- Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts2026· architecture→ topic thread
- The Lessons of Developing Process Reward Models in Mathematical Reasoning2025· training-methods→ similar topic
- Black-Box On-Policy Distillation of Large Language Models2025· training-methods→ similar topic
- QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management2025· training-methods→ similar topic
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks2025· multimodal→ similar topic
- Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding2026· vision→ LLM-judged similar
- Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models2025· multimodal→ in-domain predecessor
- Apollo: An Exploration of Video Understanding in Large Multimodal Models2024· multimodal→ similar topic
- VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models2025· multimodal→ similar topic
- UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving2025· multimodal→ similar topic