ArXiv: 2510.20579

🎯 Pitch

Video models that point to exactly when and where they see evidence—not just what they think—don't just become more trustworthy; their explicit spatio-temporal traces can be used after training to self-correct and outperform standard voting at test time. This work introduces the first framework to tightly couple reasoning with timestamped, boxed object annotations via a novel reinforcement learning setup, revealing a sharp boundary: such grounding boosts accuracy only when the base model already has a foothold on the problem.

1. Executive Summary

Open-o3-Video introduces a non-agent framework that integrates explicit spatio-temporal evidence—timestamps, object names, and bounding boxes—directly into the video reasoning process, making reasoning traceable and verifiable without external tool orchestration. Trained on Qwen2.5-VL-7B with a two-stage pipeline combining supervised fine-tuning on the curated STGR dataset and reinforcement learning via Group Sequence Policy Optimization (GSPO) with adaptive temporal proximity (annealing the temporal reward’s standard deviation σ from 4 to 1 during training to reduce early-stage reward sparsity) and temporal gating (withholding spatial rewards when temporal predictions exceed a 3-second threshold), the model achieves state-of-the-art performance on the V-STAR benchmark, improving mAM by 14.4% and mLGM by 24.2% over its base model. On broader video understanding tasks, Open-o3-Video delivers consistent gains—including a 4.1% improvement on long videos in VideoMME and a 10.1% boost on the LongVideo-Reason-eval benchmark—while the generated grounded evidence further enables confidence-aware test-time scaling that outperforms majority voting, establishing that explicit spatio-temporal grounding strengthens reasoning only when the base model already possesses non-trivial visual comprehension capabilities on the target problems.

2. Context and Motivation

The Core Problem: Video Reasoning Models Lack Verifiable Evidence

The fundamental gap this paper addresses is that most video reasoning models produce reasoning traces that are text-only — they describe what they think is happening, but never indicate when and where the supporting evidence appears in the video. This matters because, as the authors put it, "videos encapsulate rich temporal dynamics and spatial interactions that far exceed the information in static images" (Section 1). A reasoning trace that says "the cat walks across the table" without specifying which cat (in a scene with multiple animals), when this happens (in a several-minute video), and where in the frame the cat appears is unfalsifiable — there's no way to verify whether the model actually saw the evidence or just hallucinated a plausible-sounding narrative.

This gap is not merely academic. Consider a video surveillance system answering "When did the suspect enter the building?" A text-only response like "the suspect entered through the side door at some point during the video" provides no actionable information — security personnel would need to re-watch the entire footage to confirm the claim. In contrast, a response that says "the suspect entered through the side door at timestamp 3:42" with a bounding box around the door region is directly verifiable and actionable. The paper's framing of this as an evidence-centered reasoning problem is deliberately modeled on the "thinking with images" paradigm recently demonstrated by OpenAI-o3 for static images — except extended to the fundamentally harder domain of video, where evidence is spread across both time and space.

Why This Problem Is Hard and Why It Matters

Extending evidence-centered reasoning from images to videos introduces three compounding difficulties that static-image approaches don't face:

1. Temporal dynamics break naive region-of-interest approaches. In a static image, you can zoom in on a dog once and be done. In a video, the same dog may walk across the frame, become partially occluded by furniture, exit the scene, and re-enter from a different angle — all within seconds. A reasoning system must not only identify that a dog is relevant, but when the dog was visible and which frames contain usable spatial evidence. The timestamping requirement means temporal localization and spatial grounding are coupled: you can't reliably score a bounding box unless you know it came from the right moment in time.

2. Existing resources provide supervision for only one axis at a time. The paper identifies three disjoint categories of what's available (Section 3.1): temporal-only grounding datasets (e.g., ActivityNet, QVHighlight) that say "the action happened between seconds 10 and 15" but provide no bounding boxes; spatial or frame-level caption corpora (e.g., PLM-Rdcap) that describe objects on isolated frames without timestamps; and general video QA datasets that have questions and answers but no localization annotations at all. None of these provides the joint spatio-temporal supervision needed to train a model that says "the dog (<obj>dog</obj><box>[x,y,w,h]</box>) at timestamp 10.2s knocked over the vase." Without synchronized supervision, any attempt to learn coherent spatio-temporal reasoning would receive conflicting or incomplete signals.

3. The training dynamics are fundamentally unstable when both axes are optimized simultaneously. This is a non-obvious technical challenge that the paper is the first to articulate (Section 4.3). Spatial grounding rewards (e.g., IoU between predicted and ground-truth boxes) are conditional on correct temporal predictions — if the model predicts timestamp 5.0s when the ground-truth evidence is at 10.2s, the spatial reward is meaningless because you're comparing boxes from different frames. In early RL training, when temporal predictions are imprecise, this creates a reward sparsity problem: the spatial reward term is nearly always zero, so the model receives no gradient signal for improving its spatial grounding. The paper calls this the "spatial collapse issue" — the spatial head effectively stops learning because its reward is gated by a temporal signal that hasn't converged yet.

What Prior Approaches Existed and Where They Fell Short

The paper situates itself relative to four distinct lines of prior work, each of which addresses part of the problem but leaves the joint spatio-temporal reasoning gap open.

Text-only video reasoning (Video-R1, VideoRFT, Video-RTS). The most directly comparable prior work are recent RL-based video reasoning models that encourage chain-of-thought reasoning but produce only textual rationales. Video-R1 (Feng et al., 2025) uses temporal-aware GRPO with curated reasoning data to improve video understanding benchmarks. VideoRFT (Wang et al., 2025) applies reinforced fine-tuning to incentivize reasoning capability. Video-RTS (Wang et al., 2025) combines RL with test-time scaling. The common thread: these methods treat video reasoning as a text-only problem. The reasoning trace describes what the model thinks is happening but never points to when or where in the video the evidence comes from. This means the reasoning is not verifiable — you can't check whether the model actually saw the visual evidence it claims to base its answer on.

The paper's Figure 1 provides a concrete illustration of this gap. Given a question about why people commemorate Qu Yuan, VideoRFT-7B produces a textual rationale that mentions "the narrative shifts to his betrayal and exile" and "people eat Zongzi and hold dragon boat races" without indicating any specific timestamps or visual regions. In contrast, Open-o3-Video produces reasoning that explicitly references a "man" at timestamp 9.0s with bounding box [249, 60, 395, 252] and again at 24.0s with a different box, directly linking the textual claim to verifiable visual evidence.

Temporal-only grounding (TRACE, Time-R1, TVG-R1). A separate line of work focuses on temporal localization: identifying when a described event occurs in a video. TRACE (Guo et al., 2024) models temporal grounding through causal event structures. Time-R1 (Wang et al., 2025) uses verifiable rewards and curated RL data for temporal localization. TVG-R1 (Chen et al., 2025) provides datasets and recipes specifically for video temporal grounding via RL. The critical limitation: these methods predict time spans — e.g., "[10.0s, 15.0s]" — but do not localize objects spatially within those spans. Knowing when something happens is insufficient if you can't also specify where in the frame to look. For complex scenes with multiple interacting objects, a temporal span alone doesn't constitute verifiable evidence.

Spatial-only grounding (SpaceR, Sa2VA). On the spatial side, models like SpaceR (Ouyang et al., 2025) focus on object-centric localization and geometric reasoning in videos, while Sa2VA (Yuan et al., 2025) marries SAM2 with LLaVA for dense grounded understanding. However, these models operate on individual frames or short clips without temporal awareness — they can tell you where an object is in a given frame, but not when that frame occurs relative to the video timeline, and not how that spatial evidence connects to a broader reasoning chain about the video's narrative.

Thinking with Images (OpenAI-o3, DeepEyes, TreeBench, VGR). The paradigm that most directly inspires this work is the "thinking with images" line of research. OpenAI-o3 formalized the idea of interleaving visual operations (cropping, zooming, region selection) with language reasoning, producing intermediate visual evidence consumed within the reasoning chain. DeepEyes (Zheng et al., 2025) showed that end-to-end RL can incentivize image–tool reasoning, while TreeBench (Wang et al., 2025) provided box-level evidence benchmarks. However, these advances are image-centric — they operate on static images where temporal consistency, motion, and fine-grained event alignment are non-issues. Extending to video introduces the coupling problem described above: evidence must be aligned in both time and space simultaneously.

The paper explicitly acknowledges several concurrent works that attempt to bridge this gap but take an agent-based, tool-augmented approach. VITAL (Zhang et al., 2025) enables an agent to crop temporally relevant clips on demand via a visual toolbox. LongVT (Yang et al., 2025) and VideoZoomer (Ding et al., 2025) iteratively invoke temporal zoom-in tools to retrieve relevant clips. Conan (Ouyang et al., 2025) teaches the model to identify evidence frames, perform cross-frame deduction, and decide when to conclude. VTimeCoT (Zhang et al., 2025) proposes a "thinking by drawing" training-free scheme for temporal grounding.

These agent-based methods differ from Open-o3-Video in a fundamental way: they rely on external orchestration — calling tool APIs, managing a multi-step pipeline, deciding when to zoom or crop or conclude. This introduces engineering complexity, potential failure modes at each tool-calling step, and latency from sequential API interactions. Open-o3-Video positions itself in contrast as a single model performing single-round inference — "the framework 'thinks with frames' in a single round of inference, directly emitting timestamped crops and bounding boxes as evidence without complex tool pipelines" (Section 2). This is a design philosophy choice: rather than decomposing the problem across multiple systems, embed the spatio-temporal reasoning capability natively in the model.

How This Paper Positions Itself

The paper frames its contribution through three deliberate positionings:

1. From "thinking with images" to "thinking with frames." The explicit goal is to extend the evidence-centered reasoning paradigm from static images to the temporal domain. The paper consistently uses the language of "thinking with frames" throughout, positioning itself as the natural successor to OpenAI-o3 for the video modality. However, they emphasize that this is not a simple extension — the coupling of temporal and spatial supervision creates qualitatively new challenges that require novel training mechanisms (adaptive temporal proximity, temporal gating) not needed in the image case.

2. Non-agent, single-model architecture as a deliberate design principle. In a research landscape increasingly dominated by agent-based, tool-augmented approaches for complex reasoning, the paper takes a contrarian stance: that native spatio-temporal reasoning can be achieved through careful data curation and reward design alone, without external orchestration. This is a bet on simplicity and reproducibility — a single model emitting structured evidence is easier to deploy, debug, and scale than a multi-agent pipeline with failure-prone tool-calling interfaces.

3. Data scarcity as the primary bottleneck, not model architecture. The paper's most distinctive positioning is that the central obstacle to spatio-temporal video reasoning is not model capacity or algorithm design, but the absence of joint supervision data. Section 3 explicitly frames data construction as the "first key contribution," before the training methodology. The 5.9k newly annotated spatio-temporal samples are presented as filling a gap that existing resources leave open — temporal datasets lack spatial annotations, spatial datasets lack temporal annotations, and neither provides the reasoning chains needed to tie everything together. This data-centric framing implies that once adequate supervision exists, standard training pipelines (SFT + RL) can unlock the capability without architectural innovation.

The paper also positions its training recipe — particularly the adaptive temporal proximity and temporal gating mechanisms — as solving a problem that would otherwise make RL training unstable or impossible. This is presented as an empirical finding rather than a theoretical contribution: without these mechanisms, "the spatial collapse issue" prevents the model from ever learning precise localization. This gives the paper a pragmatic, engineering-oriented character — it's solving the practical problems that arise when you actually try to train this capability, rather than proposing a hypothetical framework that would work in theory.

3. Technical Approach

3.1 Reader Orientation

Open-o3-Video is a single video-language model that, when asked a question about a video, generates not just a textual answer but also explicit spatio-temporal evidence — timestamps of key moments and bounding boxes around relevant objects — all in one forward pass without calling any external tools. The system solves the problem of making video reasoning verifiable by embedding localization capabilities natively into the model through a combination of curated training data with joint temporal-spatial supervision and a reinforcement learning stage designed to prevent the spatial grounding signal from collapsing when temporal predictions are initially inaccurate.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a two-stage training pipeline:

  1. STGR Data Construction Pipeline — takes existing temporal-only and spatial-only grounding datasets plus raw video sources and produces unified spatio-temporal annotations that include questions, answers, keyframe timestamps, bounding boxes, and reasoning chains that explicitly reference both when and where evidence appears. This is the input to both training stages.

  2. Cold-Start Initialization (SFT) — fine-tunes the base Qwen2.5-VL-7B model on the STGR-CoT-30k corpus to learn the structured output format (thinking traces with <obj>, <box>, and <t> tags) and acquire basic spatio-temporal grounding before reinforcement learning begins. This stage exists to reduce reward sparsity in the RL stage by ensuring the model already knows how to produce grounded outputs.

  3. Reinforcement Learning with GSPO — further trains the cold-start model on the STGR-RL-36k corpus using Group Sequence Policy Optimization, a sequence-level RL algorithm that optimizes entire responses (including timestamps and boxes) as atomic units rather than token-by-token. The reward function has three components: accuracy (task-dependent), thinking (temporal + spatial alignment), and format (structural compliance).

  4. Adaptive Temporal Proximity and Temporal Gating Mechanisms — two coupled reward-shaping strategies embedded within the thinking reward. Adaptive temporal proximity anneals the tolerance of the temporal reward during training (starting loose, ending strict), while temporal gating zeros out spatial rewards when temporal predictions are too far from ground truth. Together, they solve the "spatial collapse" problem where imprecise early temporal predictions starve the spatial head of learning signal.

Information flows as follows: a video-question pair enters the trained model → the model generates a structured thinking trace containing interleaved text and spatio-temporal evidence tags → the evidence is both readable by humans (verifiable reasoning) and parseable by downstream systems (for confidence-aware voting). During training, rewards are computed by comparing predicted timestamps and boxes against ground-truth annotations, with the temporal gate controlling when spatial comparisons are valid.

3.3 Roadmap for the Deep Dive

  • First, the formal reward function (Equation 1 and its components), because understanding what the model optimizes for is necessary before understanding how it optimizes. The reward design encodes the paper's central insight — that temporal and spatial supervision must be coupled but the coupling creates a credit assignment problem.

  • Second, the adaptive temporal proximity mechanism (Equation 4), since it is the paper's primary solution to the temporal reward sparsity problem. The Gaussian annealing schedule is the mechanism that makes RL training converge at all; without it, the model receives near-zero temporal rewards early on and never learns to localize in time.

  • Third, the temporal gating mechanism (Equation 5), which completes the solution by ensuring spatial rewards are only computed when temporal predictions are sufficiently accurate. This is the complementary half of the reward design — proximity addresses when temporal rewards are given, gating addresses when spatial rewards are given.

  • Fourth, the GSPO algorithm (Section 4.2 and Appendix A.9), because the choice of RL algorithm matters for this task. I explain why sequence-level optimization (GSPO) is better suited than token-level optimization (GRPO) for outputs that include structured localization tokens whose correctness can only be evaluated holistically.

  • Fifth, the cold-start initialization stage, which addresses a practical problem: if you start RL from a model that has never seen spatio-temporal evidence tags, the exploration space is so large and the format reward so sparse that training collapses. SFT provides the scaffolding.

  • Sixth, the data construction pipeline (Section 3), because the quality and structure of the training data directly determines whether the above mechanisms can work. The annotation pipeline's three-stage filtering (initial annotation, bounding box filtering, self-consistency checking) is what makes the joint supervision reliable.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that spatio-temporal video reasoning can be learned by a single model through (1) carefully constructed training data that provides synchronized temporal and spatial supervision, and (2) a reinforcement learning reward design that prevents the spatial grounding signal from collapsing when temporal predictions are initially imprecise. The paper's technical contribution is not a new model architecture but rather a training recipe — data pipeline + reward shaping — that overcomes the specific instability that arises when jointly optimizing temporal and spatial localization.

The Complete Reward Function

The reinforcement learning stage optimizes the model using a scalar reward computed for each generated response. The reward is designed to jointly incentivize answer correctness, temporal alignment of predicted timestamps, spatial precision of predicted bounding boxes, and adherence to a structured output format. The total reward for a query-completion pair $(x, y)$ is:

r(x,y)=racc(x,y)+rthk(x,y)+rfmt(x,y)r(x, y) = r_{\text{acc}}(x, y) + r_{\text{thk}}(x, y) + r_{\text{fmt}}(x, y)

where $r_{\text{acc}}$ is the accuracy reward (task-dependent), $r_{\text{thk}}$ is the thinking reward (temporal + spatial alignment), and $r_{\text{fmt}}$ is the format reward (structural compliance). The sum is then group-normalized across a batch of candidate responses to compute the advantage used by GSPO.

What it computes: a composite scalar that combines three independent quality signals. The accuracy term measures whether the answer is correct. The thinking term measures whether the spatio-temporal evidence (timestamps and boxes) aligns with ground truth. The format term measures whether the response uses the required XML-like tag structure. Together they create a dense reward signal that covers all aspects of the desired output — content correctness, localization precision, and structural well-formedness.

Why this additive form: decomposing the reward into independent terms allows each component to be designed based on the specific supervision signal available. Accuracy depends on whether the ground-truth answer matches the prediction (available for all task types). Thinking alignment depends on whether ground-truth timestamps and boxes exist (available only for grounded tasks). Format compliance depends only on the output structure (always available). An additive decomposition means the model can receive partial credit — if it produces the right answer and format but imprecise localization, it still gets $r_{\text{acc}} + r_{\text{fmt}}$, providing gradient signal to maintain answer accuracy while temporal/spatial rewards are low. A multiplicative form would risk zeroing out the entire reward whenever one component is zero.

Accuracy Reward $r_{\text{acc}}$ — Task-Dependent Scoring

The accuracy reward varies based on the task type $\tau$ of the training instance. The paper defines four task types: multiple-choice question (MCQ), free-form question answering (QA), spatial grounding (SG), and temporal grounding (TG). The reward function is:

racc(x,y)={I[ypred=ygt],τ=MCQROUGE(ypred,ygt),τ=QAvIoU(Bpred,Bgt),τ=SGtIoU(Spred,Sgt),τ=TGr_{\text{acc}}(x, y) = \begin{cases} \mathbb{I}[y_{\text{pred}} = y_{\text{gt}}], & \tau = \text{MCQ} \\[4pt] \text{ROUGE}(y_{\text{pred}}, y_{\text{gt}}), & \tau = \text{QA} \\[4pt] \text{vIoU}(B_{\text{pred}}, B_{\text{gt}}), & \tau = \text{SG} \\[4pt] \text{tIoU}(S_{\text{pred}}, S_{\text{gt}}), & \tau = \text{TG} \end{cases}

where $\mathbb{I}[\cdot]$ is the indicator function (1 if exact match, 0 otherwise), $y_{\text{pred}}$ and $y_{\text{gt}}$ are predicted and ground-truth answer strings, $B_{\text{pred}}$ and $B_{\text{gt}}$ are predicted and ground-truth bounding boxes, and $S_{\text{pred}} = [s_{\text{pred}}, e_{\text{pred}}]$ and $S_{\text{gt}} = [s_{\text{gt}}, e_{\text{gt}}]$ are predicted and ground-truth temporal segments (start and end times). ROUGE is the standard text overlap metric. vIoU (volumetric Intersection over Union) measures overlap between bounding boxes, and tIoU (temporal Intersection over Union) measures overlap between time intervals.

What it computes: for each task type, a scalar between 0 and 1 quantifying how correct the answer (or localization) is. MCQ uses hard binary scoring (exact match). QA uses soft scoring via ROUGE to give partial credit for partially correct free-form answers. SG uses vIoU to measure spatial overlap. TG uses tIoU to measure temporal overlap.

Why this task-specific design: the training data mix includes samples from four qualitatively different tasks, and a single accuracy metric cannot fairly score all of them. MCQ with 4 options needs exact-match scoring because any wrong option is equally wrong. QA with free-form answers needs soft scoring because "the man in the red jacket" and "the man wearing a red coat" should receive high but not perfect credit. SG and TG are localization tasks where accuracy is inherently continuous (a box that misses by 2 pixels is better than one that misses by 200 pixels). Using one metric, e.g., exact match, for SG would make the reward landscape binary and nearly flat — the model would receive zero reward for a box that's 99% correct but 1 pixel off, eliminating useful gradient signal.

Thinking Reward $r_{\text{thk}}$ — The Core Contribution

The thinking reward is the sum of two terms:

rthk(x,y)=rt(x,y)+rs(x,y)r_{\text{thk}}(x, y) = r_t(x, y) + r_s(x, y)

where $r_t$ is the temporal term measuring how well predicted timestamps align with ground truth, and $r_s$ is the spatial term measuring how well predicted bounding boxes match ground truth. The novelty is not in the decomposition but in the mechanisms that make $r_t$ and $r_s$ learnable simultaneously despite their dependence.

The dependence is this: $r_s$ compares predicted boxes at predicted timestamps against ground-truth boxes at matched ground-truth timestamps. If the timestamp is wrong (e.g., predicted 5.0s when ground truth is 10.2s), the predicted box is being compared against the wrong ground-truth frame entirely — a box on frame 5 might have zero overlap with the correct box on frame 10 even if the spatial prediction mechanism is working perfectly, because the object moved. This means spatial reward is conditioned on temporal accuracy: you can't learn to ground spatially unless you first learn to localize temporally. In early RL training, when temporal predictions are random, $r_s$ is approximately zero for all samples, providing no learning signal for the spatial head — the "spatial collapse" problem.

Temporal Term with Adaptive Temporal Proximity

Let $M$ be the number of timestamps $\{t_m\}_{m=1}^M$ parsed from the model's thinking trace. The temporal reward depends on the supervision type available for the instance:

rt(x,y)={1M∑m=1MI[sgt≤tm≤egt],τt=Int1M∑m=1Mexp⁡(−Δtm22σ2),τt=Pt0,τt=∅r_t(x, y) = \begin{cases} \frac{1}{M} \sum_{m=1}^M \mathbb{I}[s_{\text{gt}} \leq t_m \leq e_{\text{gt}}], & \tau_t = \text{Int} \\[12pt] \frac{1}{M} \sum_{m=1}^M \exp\left(-\frac{\Delta t_m^2}{2\sigma^2}\right), & \tau_t = \text{Pt} \\[12pt] 0, & \tau_t = \emptyset \end{cases}

where $\tau_t \in \{\text{Int}, \text{Pt}, \emptyset\}$ indicates the type of temporal supervision: Interval (Int) provides a ground-truth span $[s_{\text{gt}}, e_{\text{gt}}]$ (the event occurred sometime between these endpoints), Point (Pt) provides specific ground-truth timestamps $\{t^{\text{gt}}_j\}$ (the evidence is at these exact moments), and $\emptyset$ means the instance has no temporal annotations. For point supervision, $\Delta t_m$ is the closest temporal distance: $\Delta t_m = \min_j |t_m - t^{\text{gt}}_j|$.

For interval supervision, the reward is the fraction of predicted timestamps that fall within the ground-truth interval — a simple containment check. For point supervision, the reward is the average of Gaussian kernels centered at each ground-truth timestamp, evaluated at each predicted timestamp. The standard deviation $\sigma$ controls the tolerance: larger $\sigma$ means the Gaussian is wider, so a predicted timestamp that's 5 seconds off might still receive a reward of 0.5; smaller $\sigma$ means the Gaussian is narrow, so even 2 seconds off receives near-zero reward.

What it computes: an average per-timestamp alignment score between 0 and 1. For interval supervision, it's a hard pass/fail per timestamp based on containment. For point supervision, it's a soft score based on temporal proximity, with the tolerance controlled by $\sigma$.

Why the Gaussian kernel: an alternative would be a hard threshold — reward 1 if $\Delta t_m \leq \delta$ and 0 otherwise. This creates a reward cliff: the model receives no gradient signal unless it happens to produce a timestamp within $\delta$ of ground truth, which in early training is essentially random. A Gaussian kernel provides smooth, nonzero gradients everywhere — even a timestamp 10 seconds off receives a small positive reward, which tells the model "move closer" rather than "you failed." The smoothness is critical for RL because policy gradient methods need nonzero advantage estimates to update; a zero reward for all samples means no update direction.

Now the key design: adaptive temporal proximity. The paper sets $\sigma$ to start large (4 seconds) and anneal to small (1 second) over the course of training. Early in training, a predicted timestamp 3 seconds from ground truth receives $\exp(-3^2 / (2 \cdot 4^2)) = \exp(-9/32) \approx 0.75$, providing substantial reward and gradient. Later in training, the same 3-second error receives $\exp(-3^2 / (2 \cdot 1^2)) = \exp(-9/2) \approx 0.01$, providing near-zero reward and forcing the model to refine.

Why annealing is necessary (what breaks without it): if $\sigma$ is fixed small (e.g., $\sigma = 1$ from the start), the temporal reward is sparse — most early predictions receive near-zero scores, so the model gets almost no signal about whether it's moving in the right direction. The temporal head doesn't converge, and because spatial rewards are gated on temporal accuracy (see below), the spatial head also never receives signal — the "spatial collapse" that motivated the mechanism in the first place. If $\sigma$ is fixed large (e.g., $\sigma = 4$ throughout), temporal rewards saturate early — the model can achieve high reward with coarse predictions and stops improving, which means the temporal gating threshold (3 seconds) is never satisfied for precise spatial evaluation. The annealing schedule gives the model a curriculum: first, just get timestamps roughly right (coarse temporal alignment under loose tolerance); then, refine them to be precise (fine temporal alignment under strict tolerance). The empirical support for this claim is in the ablation (Table 5): fixed $\sigma = 1$ achieves 32.6% mAM while fixed $\sigma = 4$ achieves 33.0% mAM, both below the adaptive schedule's 33.7% mAM.

A subtle but important consequence of the Gaussian kernel: for point supervision, the reward for a predicted timestamp depends only on its distance to the nearest ground-truth timestamp (the $\min_j$ in $\Delta t_m$). This means if the model predicts a timestamp that happens to land near any annotated moment — even if it's referencing the wrong event — it receives high temporal reward. The temporal gating mechanism (next section) partially mitigates this by checking whether the box at that timestamp matches the corresponding ground-truth box, but the temporal term alone does not enforce which event the timestamp refers to, only that it's close to some annotated moment.

Spatial Term with Temporal Gating

For each predicted timestamp $t_m$, let $j^*(m) = \arg\min_j |t_m - t^{\text{gt}}_j|$ be the index of the nearest ground-truth timestamp. Let $B_m$ be the set of predicted bounding boxes at timestamp $t_m$ and $B^{\text{gt}}_{j^*(m)}$ be the set of ground-truth boxes at the matched frame. The per-frame spatial score is the maximum IoU (Intersection over Union) between any predicted box and any ground-truth box in the matched frame:

vm=max⁡b∈Bm,  bgt∈Bj∗(m)gtIoU(b,bgt)v_m = \max_{b \in B_m, \; b^{\text{gt}} \in B^{\text{gt}}_{j^*(m)}} \text{IoU}(b, b^{\text{gt}})

The spatial reward is then:

rs(x,y)=1M∑m=1MI[∣tm−tj∗(m)gt∣≤τ]⋅vmr_s(x, y) = \frac{1}{M} \sum_{m=1}^M \mathbb{I}\left[|t_m - t^{\text{gt}}_{j^*(m)}| \leq \tau\right] \cdot v_m

where $\tau$ is a fixed temporal threshold (3 seconds in all experiments). The indicator function $\mathbb{I}[|t_m - t^{\text{gt}}_{j^*(m)}| \leq \tau]$ is the temporal gate: it evaluates to 1 only when the predicted timestamp is within 3 seconds of its matched ground-truth timestamp, and 0 otherwise. When the gate is closed (0), the spatial reward for that timestamp is zero regardless of how good the predicted box is.

What it computes: the average spatial alignment across predicted timestamps, but only counting timestamps that are temporally accurate enough for the spatial comparison to be meaningful. For timestamps that pass the gate, the reward is the best IoU between predicted and ground-truth boxes at the matched frame — a standard metric for bounding box quality (1.0 for perfect overlap, 0.0 for no overlap).

Why the temporal gate is necessary: without the gate, the spatial reward would be computed for every predicted timestamp regardless of temporal accuracy. Consider what happens when the model predicts timestamp 5.0s with a high-quality bounding box of a dog, but the ground-truth dog evidence is at timestamp 10.2s (a different frame). The $\arg\min$ matching would pair timestamp 5.0s with ground-truth frame 10.2s (since it's the nearest annotated time), and the spatial IoU would be computed between the predicted box at frame 5 and the ground-truth box at frame 10 — two completely different frames where the dog might be in a different position, different pose, or even absent. The resulting IoU could be high by chance (if the dog happens to be in a similar screen position on both frames) or low despite the spatial prediction being perfectly accurate for its own frame. Either way, the reward is noisy and misleading — it doesn't reflect the quality of the spatial prediction for the frame it was actually made on. Worse, it could reward wrong spatial predictions: a box that happens to overlap with the ground-truth box from a different frame.

The temporal gate prevents this by enforcing a simple rule: spatial rewards are only valid when the temporal prediction is trustworthy. The 3-second threshold means the model must localize the evidence to within a 6-second window (3 seconds before or after the ground-truth timestamp) before it can receive any credit for spatial precision. This couples the two learning objectives: to get spatial reward, you must first get better at temporal localization. This is exactly the dependency that causes the spatial collapse problem. The gate makes this dependency explicit and hard rather than implicit and noisy — which is a design choice that forces the learning dynamics to respect the natural precedence (temporal before spatial).

Why the maximum over boxes: the model may predict multiple bounding boxes at a single timestamp (e.g., <obj>dog</obj><box>...</box> and <obj>ball</obj><box>...</box> at the same time). Similarly, a ground-truth frame may have multiple annotated objects. The $\max$ operation takes the best possible pairwise match, which is generous — it gives the model credit for getting any object right, even if it missed others. This is a practical choice to prevent the reward from being unfairly penalized when the model correctly grounds one object but the annotation includes objects it wasn't asked about. A stricter alternative would be to average across all annotated objects or require all of them to be matched, but this would penalize over-prediction (the model spotting extra objects) and under-prediction equally, which is a harder optimization problem to get started with.

Why the gate threshold is 3 seconds: the paper does not provide an explicit ablation over $\tau$ values. The choice of 3 seconds likely reflects the scale of the temporal annotations in the training data — most keyframes are annotated with timestamps precise to one decimal place, and the events described (object appearances, actions) typically span several seconds rather than sub-second moments. A threshold much smaller than 3 seconds (e.g., 0.5 seconds) would make the gate too strict and recreate the reward sparsity problem. A threshold much larger (e.g., 10 seconds) would weaken the coupling between temporal and spatial learning and allow noisy spatial comparisons from misaligned frames.

The Coupling Between Adaptive Proximity and Temporal Gating

The paper presents adaptive temporal proximity and temporal gating as complementary mechanisms that together solve the spatial collapse problem. Their relationship is subtle and worth understanding explicitly.

Adaptive temporal proximity controls the LEARNING of temporal accuracy. It determines how much reward (and thus gradient signal) the temporal head receives for predictions at various distances from ground truth. When $\sigma$ is large, the model learns coarse temporal localization — getting timestamps within a few seconds. When $\sigma$ is small, the model learns fine temporal localization — getting timestamps within one second.

Temporal gating controls the VALIDITY of spatial rewards. It determines whether the spatial head receives any signal at all for a given timestamp, independent of how good the boxes are. The gate opens (allows spatial reward) only when the temporal prediction is within 3 seconds of ground truth.

The interaction: early in training, with $\sigma = 4$, the temporal reward is generous and the temporal head learns to predict timestamps that are roughly correct. However, "roughly correct" might still be 4–5 seconds off, which means the gate is still closed — no spatial rewards yet. As $\sigma$ anneals toward 1, the temporal reward becomes stricter, forcing the temporal head to refine predictions. Only when temporal errors drop below the 3-second threshold does the gate begin to open consistently, allowing the spatial head to start receiving gradient signal. This creates a natural curriculum: temporal localization learns first (under loose then strict tolerance), and only once temporal predictions are reliable enough does spatial grounding begin to receive feedback.

Without adaptive proximity (fixed $\sigma = 4$): the temporal reward never becomes strict, so temporal predictions plateau at coarse accuracy well above 3 seconds. The gate rarely opens, spatial learning stalls. This is why Table 5 shows 33.0% mAM with fixed $\sigma = 4$ vs. 33.7% with adaptive.

Without temporal gating: the spatial reward is computed for all timestamps regardless of accuracy. The spatial head receives gradients from noisy, frame-mismatched comparisons (boxes at frame 5 evaluated against ground truth at frame 10), which are as likely to be misleading as helpful. The spatial head never learns reliable grounding because its reward signal is dominated by noise. This is why Table 5 shows 32.3% mAM without gating vs. 33.7% with.

With both mechanisms: the annealing schedule ensures that temporal accuracy eventually becomes precise enough to satisfy the gate, and the gate ensures that once it does, the spatial comparisons are to the correct ground-truth frames. The two mechanisms together create a coordinated schedule where temporal learning precedes spatial learning, and spatial learning only begins when it has reliable temporal anchors. This is the central algorithmic insight of the paper.

Format Reward $r_{\text{fmt}}$

The format reward is a simple structural check with three levels:

rfmt(x,y)={1.0,if response contains thinking,  <answer> with correct <obj>,<box>,<t> tags0.5,if response contains only thinking and <answer> without grounding tags0.0,otherwiser_{\text{fmt}}(x, y) = \begin{cases} 1.0, & \text{if response contains } \texttt{thinking}, \; \texttt{<answer>} \text{ with correct } \texttt{<obj>}, \texttt{<box>}, \texttt{<t>} \text{ tags} \\ 0.5, & \text{if response contains only } \texttt{thinking} \text{ and } \texttt{<answer>} \text{ without grounding tags} \\ 0.0, & \text{otherwise} \end{cases}

What it computes: a scalar 1.0, 0.5, or 0.0 based on the presence and correctness of the required XML-like structural tags. The full reward (1.0) requires both the thinking/answer structure AND the spatio-temporal evidence tags. The partial reward (0.5) requires the thinking/answer structure but not the evidence tags. Anything else gets zero.

Why this three-tier design: the format reward serves two purposes. First, it enforces a parseable output structure that makes downstream processing possible — the confidence-aware voting mechanism (Appendix A.10) needs to extract <obj>, <box>, and <t> tags programmatically. Second, the intermediate tier (0.5) provides a stepping stone: a model that produces reasoned answers but hasn't yet learned grounding still receives partial credit, preventing the reward from collapsing to zero during early training when grounding tags are rare. This is the same curriculum principle as adaptive temporal proximity but applied to output structure: don't penalize the model for failing at the hard part (grounding) before it's mastered the easy part (reasoned answers).

Group Sequence Policy Optimization (GSPO)

The paper uses GSPO rather than the more common GRPO (Group Relative Policy Optimization) as the RL algorithm. The distinction matters for this task specifically.

In standard GRPO, the importance ratio and clipping are applied per token: each token's probability relative to the old policy is computed and clipped independently. The advantage is shared across all tokens in the response. This works well when the reward is defined over the entire sequence (e.g., "was the answer correct?") because every token contributed to the outcome and should be updated proportionally.

In GSPO, the importance ratio is defined at the sequence level. For a response $y_i$ with $|y_i|$ tokens, the sequence-level ratio is:

si(θ)=(πθ(yi∣x)πθold(yi∣x))1/∣yi∣=exp⁡(1∣yi∣∑t=1∣yi∣log⁡πθ(yi,t∣x,yi,<t)πθold(yi,t∣x,yi,<t))s_i(\theta) = \left(\frac{\pi_\theta(y_i | x)}{\pi_{\theta_{\text{old}}}(y_i | x)}\right)^{1/|y_i|} = \exp\left(\frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,t} | x, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} | x, y_{i,<t})}\right)

where $\pi_\theta(y_i | x)$ is the probability of generating the entire response under the current policy, $\pi_{\theta_{\text{old}}}(y_i | x)$ is the probability under the old (reference) policy, and $|y_i|$ normalizes by response length. The normalizing exponent $1/|y_i|$ converts the product of token probabilities (which shrinks exponentially with length) into a geometric mean per token, making the ratio comparable across responses of different lengths.

The GSPO objective is then:

JGSPO(θ)=Ex,{yi}∼πθold[1G∑i=1Gmin⁡(si(θ)A^i,  clip(si(θ),1−ϵ,1+ϵ)A^i)]J_{\text{GSPO}}(\theta) = \mathbb{E}_{x, \{y_i\} \sim \pi_{\theta_{\text{old}}}} \left[\frac{1}{G} \sum_{i=1}^G \min\left(s_i(\theta) \hat{A}_i, \; \text{clip}(s_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i\right)\right]

where $G$ is the group size (number of candidate responses sampled for each query), $\hat{A}_i$ is the group-normalized advantage:

A^i=r(x,yi)−mean({r(x,yj)}j=1G)std({r(x,yj)}j=1G)\hat{A}_i = \frac{r(x, y_i) - \text{mean}(\{r(x, y_j)\}_{j=1}^G)}{\text{std}(\{r(x, y_j)\}_{j=1}^G)}

and $\epsilon$ controls the clipping range (standard PPO-style conservative update).

What it computes: for each query, the model samples $G$ candidate responses from the old policy, scores each with the reward function $r(x, y)$, normalizes the rewards within the group to get advantages, then computes the sequence-level importance ratio for each response. The objective is the minimum of the unclipped and clipped ratios times the advantage — the standard PPO clipping that prevents overly large policy updates. The key difference from GRPO is that the entire response's ratio is a single scalar (the geometric mean of per-token ratios) rather than a vector of per-token ratios.

Why GSPO over GRPO for this task: the paper argues that rewards in spatio-temporal grounded reasoning are defined over complete reasoning traces that include timestamps and bounding boxes, not individual tokens. The correctness of a timestamp 9.2s or a bounding box [374, 67, 420, 224] can only be evaluated in the context of the entire response — you cannot assign credit to the token "3" vs. "7" vs. "4" within a box coordinate. GRPO assigns per-token advantages, which means the token "3" in x_min=374 gets the same advantage as the token <obj> that opens the tag, even though their contributions to the reward are qualitatively different. This creates high-variance gradient estimates and can cause the model to over-optimize specific token patterns (e.g., always predicting the same box coordinates because they happened to receive positive advantage) rather than learning to produce globally consistent structured output.

GSPO avoids this by treating the entire response as the unit of optimization: the response either has good timestamps and boxes (receives high advantage) or doesn't (receives low advantage), and the policy gradient scales the probability of the entire response proportionally. This aligns the optimization granularity with the reward granularity — both operate at the sequence level. The empirical support is in Table 3: SFT + RL with GSPO achieves 33.7% mAM vs. 32.8% with GRPO, and the paper notes that GSPO "yields higher grounding accuracy and more stable training than GRPO." Specifically, GSPO achieves "better long-horizon temporal localization (+2.9% Chain1 tIoU)" (Section 5.2).

Why length normalization: the geometric mean normalization $\exp(\frac{1}{|y_i|} \sum \log ...)$ prevents the importance ratio from being dominated by response length. Without normalization, a longer response would have a much smaller probability product (more tokens multiplied together means smaller numbers), making its importance ratio artificially small and reducing its contribution to the gradient. Normalization ensures that a correct 200-token grounded response and a correct 50-token grounded response contribute similarly to the update, which is important because grounded responses are inherently longer (they include structured tags and coordinates) than ungrounded ones. If length were not normalized, the RL algorithm would develop a length bias — favoring shorter responses regardless of grounding quality — which would work against the goal of producing detailed spatio-temporal evidence.

Cold-Start Initialization (SFT Stage)

Before reinforcement learning, the base Qwen2.5-VL-7B model is fine-tuned on the STGR-CoT-30k dataset for one epoch with learning rate $1 \times 10^{-6}$. This stage has two purposes.

First, it teaches the output format. The base model has never seen the structured evidence tags (<obj>, <box>, <t>) and does not know to produce them. Without SFT, the RL stage would need to discover this format through exploration — a search space so large (all possible token sequences that might or might not include the right tags) that random exploration would almost never produce well-formed grounded outputs. The format reward would be zero for nearly all samples, and learning would stall. SFT provides explicit demonstrations of the desired format, so the RL stage starts from a model that already knows how to produce grounded outputs, even if the content (which timestamps, which boxes) is not yet accurate.

Second, it provides basic spatio-temporal grounding capability. The STGR-CoT-30k dataset includes 4.1k temporal grounding samples, 5k spatial grounding samples, 5.9k spatio-temporal samples, and 15k general QA samples. By training on this mix, the model learns to associate visual evidence with timestamps and boxes at a rudimentary level — the "cold-start" model can produce grounded outputs, just not very precise ones. This reduces the reward sparsity problem in the RL stage: if the cold-start model's timestamps are already within, say, 5–10 seconds of ground truth (rather than completely random), the adaptive temporal proximity mechanism (with initial $\sigma = 4$) can provide meaningful temporal reward from the first RL step.

The paper reports that "pure SFT" alone achieves 28.5% mAM and 37.1% mLGM on V-STAR (Table 3), compared to the base model's 19.3% mAM and 22.4% mLGM — a substantial improvement from the data alone. This confirms that the STGR dataset provides useful supervision even without RL, and that the SFT stage successfully bootstraps grounded reasoning. The RL stage then adds another major improvement (+5.2% mAM, +9.5% mLGM over SFT alone), showing that optimizing for temporal and spatial alignment through the reward function provides gains beyond what supervised imitation can achieve.

Why one epoch: the paper trains for exactly one epoch in both SFT and RL stages. This is a common practice in instruction tuning and RL fine-tuning of LLMs to prevent overfitting to the relatively small training datasets. The STGR-CoT-30k corpus has 30k samples, which is sufficient for one epoch of SFT to learn the output format but likely not enough for multiple epochs without memorization. The STGR-RL-36k corpus is similarly sized. The learning rate $1 \times 10^{-6}$ is standard for fine-tuning 7B-parameter models and is low enough to avoid catastrophic forgetting of the base model's general video understanding capabilities.

STGR Data Construction Pipeline

The data pipeline produces the synchronized spatio-temporal supervision that makes both training stages possible. It combines existing resources with newly annotated data through a three-stage process (Section 3, Figure 2).

Stage 1: Initial Annotation via Gemini 2.5 Pro. The pipeline takes two types of input sources: (a) temporal grounding datasets (ActivityNet, COIN, QueryD, QVHighlight, DiDeMo) that provide video segments with natural language descriptions but no bounding boxes, and (b) PLM-Rdcap data that provides region-level dense captions on individual frames but no timestamps. Videos are passed through the Gemini 2.5 Pro API with carefully designed prompts (shown in Appendix A.4, Figures 5 and 6). The prompts instruct Gemini to generate structured JSON annotations containing:

  • A question-answer pair focused on a specific object or person (not an action — the prompt says "The question should focus on a specific object or person, rather than their action" for temporal grounding sources)
  • One to five keyframes with timestamps (sampled from within the annotated segment)
  • For each keyframe, one to three salient objects with normalized bounding boxes [x_min, y_min, x_max, y_max]
  • A reasoning process that references every object using the strict format: <obj>object_name</obj><box>[x_min, y_min, x_max, y_max]</box>at<t>timestamp</t>s

The two prompt variants (Figures 5 and 6) differ based on the input format: for PLM-Rdcap, Gemini receives dense captions and frame counts; for temporal grounding datasets, it receives the annotated segment description, video duration, and ground-truth time span. The output format is identical JSON in both cases.

Stage 2: Bounding Box Filtering. Initial annotations from Gemini may contain noisy or incorrect boxes. The filtering applies two rules:

  1. Boxes that cover more than 80% of the frame are removed as uninformative (a box covering nearly the entire frame doesn't localize anything).
  2. Each crop is verified by a separate model: Qwen2.5-VL-7B is queried with "Is this a {object_name}?" on the cropped region, and only samples where the answer is "yes" are retained. This is a semantic consistency check — the bounding box must actually contain the object it claims to contain.

Stage 3: Self-Consistency Checking and Quality Control. This stage enforces alignment between three elements of each annotation: the timestamps in the key_frames, the bounding boxes in the key_items, and the spatio-temporal references in the reasoning_process. The check verifies that all temporal and spatial references appearing in the reasoning text are covered by the corresponding annotations — if the reasoning says "the dog at 10.2s" but the key_frames don't include 10.2s, the sample is discarded. Additionally, each reasoning sentence is checked for semantic consistency with its referenced visual evidence: the bounding box region is cropped from the video and Qwen2.5-VL judges whether the sentence accurately describes what's in the crop. Samples with inconsistent visual-textual alignment are removed.

Why this three-stage pipeline: each stage addresses a different failure mode. The initial annotation (Stage 1) provides the raw material but is imperfect because Gemini 2.5 Pro, while powerful, is not a dedicated video grounding model and can make mistakes — hallucinated boxes, misidentified objects, temporal references that don't align with keyframes. The bounding box filtering (Stage 2) removes the most common failure mode: boxes that are technically formatted correctly but don't contain the claimed object. The self-consistency check (Stage 3) removes subtler failures: annotations where the reasoning chain refers to evidence that isn't actually in the structured annotations, or where the visual content doesn't match the textual description.

Data volume and composition. The pipeline produces 5.9k spatio-temporal samples, of which 3.9k come from temporal grounding video sources and 2k from PLM-Rdcap sources. These 5.9k samples are the only source of joint supervision — they are the only data points where a single sample provides question, answer, timestamps, bounding boxes, and a reasoning chain that connects them all. The remaining data in both STGR-CoT-30k (24.1k samples) and STGR-RL-36k (30.1k samples) provides only partial supervision: temporal-only data has timestamps but no boxes, spatial-only data has boxes but no timestamps, and general QA data has neither. The SFT set composition is 13.7% temporal, 16.7% spatial, 19.7% spatio-temporal, and 50.0% general QA. The RL set shifts emphasis toward spatio-temporal reasoning: 14.4% temporal, 13.9% spatial, 30.3% spatio-temporal, and 41.7% general QA.

Why the RL set has more spatio-temporal data: the SFT stage uses spatio-temporal data to teach the output format, but the model learns the actual precision from RL. Since the thinking reward provides stronger gradients for spatio-temporal alignment than for general QA (which only has the accuracy reward component), allocating more spatio-temporal samples to the RL stage maximizes the benefit of the reward design. The general QA data in both stages maintains the model's broad video understanding capabilities and prevents catastrophic forgetting during the grounding-focused training.

Filtering criteria for temporal sources. The paper applies strict constraints when sampling from temporal grounding datasets: samples with chain-of-thought length under 6,000 characters and ground-truth spans covering less than 70% of total video duration are retained. Videos with annotated actions lasting more than 50% of the video are discarded. Videos must be between 10 seconds and 3 minutes in duration. These filters serve to keep the reasoning traces tractable (not excessively long) and the temporal annotations meaningful (not spanning nearly the entire video, which would make temporal localization trivial and the supervision uninformative).

Annotation cost. The paper does not report the API cost or wall-clock time for generating the 5.9k annotations through Gemini 2.5 Pro, nor the computational cost of the bounding box verification and self-consistency checks through Qwen2.5-VL. This is a pragmatic gap — the data pipeline is a significant engineering investment that is not quantified. However, 5.9k samples is a modest number compared to typical vision-language training datasets, suggesting that the pipeline is feasible to reproduce given API access and a few thousand videos worth of annotation budget.

4. Key Insights and Innovations

Innovation 1: Designing the Coupling Between Temporal and Spatial Rewards as a Solvable Optimization Problem Rather Than a Data Problem

The paper's deepest conceptual move is reframing what makes joint spatio-temporal grounding hard. The naive view is that the difficulty is primarily data scarcity — if we just had enough examples with synchronized timestamps and boxes, standard training would work. The paper argues instead that even with adequate joint supervision data, a fundamental credit assignment pathology prevents simultaneous learning: spatial grounding quality depends on temporal accuracy, but temporal accuracy is initially random, so the spatial reward signal is starved of valid feedback early in training.

This is a diagnostic insight, not merely an engineering observation. Prior work in video grounding treated temporal and spatial localization as parallel objectives — optimize both simultaneously and reward each independently. VideoChat-R1 (Li et al., 2025) applies GRPO to spatio-temporal perception tasks with separate temporal and spatial rewards but does not address their interdependence. STVG-o1 (Gu et al., 2025) targets spatio-temporal video grounding through reinforcement fine-tuning but frames it as a standard multi-objective RL problem. The implicit assumption in these approaches is that independent reward terms create independent gradient signals, and the model will figure out the coupling through optimization.

The paper demonstrates that this assumption fails in a specific, diagnosable way: the spatial reward collapses to near-zero when temporal predictions are imprecise, not because the spatial head is bad, but because it's being evaluated against the wrong ground-truth frames. This is not a data problem — more spatio-temporal annotations wouldn't fix it, because the issue is in the evaluation dynamics, not the supervision coverage. It's an optimization landscape problem: the reward surface has a large flat region (where spatial reward is zero regardless of spatial quality) that the model must cross before any spatial learning can begin.

The solution — adaptive temporal proximity annealing the Gaussian kernel's standard deviation while temporal gating zeros out spatial rewards beyond a threshold — is notable not for its complexity (both are simple mechanisms) but for the principle it embodies: that temporally-structured curricula can make coupled objectives learnable without architectural changes. The annealing schedule (σ from 4 to 1) and the gate threshold (τ = 3s) are specific choices, but the deeper contribution is the recognition that when objective A conditions objective B, the optimizer needs A to converge before B can receive meaningful signal — and that reward shaping, not more data or bigger models, is the right tool for enforcing this precedence.

The empirical evidence is in Table 5: removing the gate drops performance by 1.4% mAM, removing adaptive proximity drops it by 0.7% mAM, and removing both would likely prevent convergence entirely. But the conceptual significance transcends these numbers: this paper is the first to articulate why joint spatio-temporal training is unstable in RL and to provide a principled (if simple) mechanism for stabilizing it. This diagnostic — naming "spatial collapse" as a distinct failure mode with a specific cause — is likely to influence how future work designs coupled objectives for video understanding.

Innovation 2: Single-Model Non-Agent Grounded Reasoning as a Deliberate Architecture Choice

In a research landscape where the dominant approach to complex video reasoning is increasingly agent-based and tool-augmented, the paper takes an explicit contrarian position: that native spatio-temporal reasoning can be achieved by a single model in a single forward pass, without external orchestration. This is not presented as a temporary limitation but as a design philosophy with claimed advantages in simplicity, reproducibility, and deployment.

The contrast with concurrent work is stark. VITAL (Zhang et al., 2025) enables an agent to crop temporally relevant clips on demand via a visual toolbox — a multi-step pipeline where the model decides when to invoke a tool, what tool to invoke, and how to incorporate the returned evidence. LongVT (Yang et al., 2025) and VideoZoomer (Ding et al., 2025) iteratively invoke temporal zoom-in tools, creating chains of tool calls that grow with video length. Conan (Ouyang et al., 2025) teaches the model to identify evidence frames, perform cross-frame deduction, and decide whether to conclude or continue — essentially a decision loop. These approaches decompose the reasoning problem across multiple systems (the reasoning model, the tool APIs, the orchestration logic), which introduces failure modes at each interaction boundary, increases latency from sequential calls, and complicates both training (the model must learn when and how to use tools) and deployment (multiple services must be maintained and coordinated).

Open-o3-Video's position is that this decomposition is unnecessary if the model is trained with the right data and reward structure. A single model that directly emits <obj>, <box>, and <t> tags as part of its autoregressive generation achieves the same functionality — producing verifiable spatio-temporal evidence — without any tool-calling infrastructure. The model is the only component that needs to be trained, served, and debugged.

The gamble here is significant. Agent-based approaches can leverage specialized tools — a dedicated object detector for bounding boxes, a dedicated temporal localizer for timestamps — that may be more precise than what a general-purpose VL model can learn to produce natively. Open-o3-Video is betting that the joint optimization enabled by end-to-end training (where reasoning, temporal localization, and spatial grounding co-adapt) outweighs the precision loss from not using specialized components. The empirical evidence on V-STAR (Table 1) partially supports this bet: Open-o3-Video achieves significantly better spatial grounding than the base model (+8.4% on Chain1 Where, +3.5% on Chain2 Where) and competitive temporal grounding (+9.1% and +10.2% on When), all without external detectors. However, the absolute spatial grounding numbers (25.4% and 6.0% Where) remain modest, and specialized spatial models like Sa2VA achieve higher spatial IoU (32.3% and 37.5%) even though their overall reasoning accuracy is worse. This suggests the single-model approach achieves adequate grounding for reasoning purposes without matching dedicated grounding specialists.

This is fundamentally an architectural simplicity argument, not a performance-maximization argument. The paper is claiming that the engineering benefits of a self-contained model — no tool APIs to maintain, no orchestration logic to debug, no sequential latency from tool calls — are worth the grounding precision tradeoff. This is an opinionated stance that will likely divide the field: those who prioritize maximum accuracy will continue with agent-based approaches, while those who prioritize deployment simplicity will find this approach compelling.

Innovation 3: The STGR Dataset as a Principled Bridge Between Incomplete Supervision Sources

While dataset construction papers are common in vision-language research, STGR's contribution is distinctive in how it stitches together three qualitatively different types of supervision — temporal-only grounding annotations, spatial-only bounding box annotations, and general video QA — into a unified format that enables joint training. This is not simply "we collected more data." It's a specific recipe for converting incomplete, heterogeneous resources into the synchronized supervision that spatio-temporal reasoning requires.

The insight is that the bottleneck for spatio-temporal reasoning is not the volume of data but the structure of supervision. Temporal grounding datasets (ActivityNet, QVHighlight, DiDeMo) exist at scale and provide timestamped event spans. Spatial grounding datasets (TreeVGR, VisCoT) exist at scale and provide bounding boxes on frames. General video QA datasets (Video-R1) exist at scale and provide question-answer pairs with textual reasoning. But none of these provides what the model actually needs to learn: a single example where a question is answered by referencing a specific object at a specific time with a bounding box, all connected through a reasoning chain.

The STGR pipeline solves this by using Gemini 2.5 Pro as a format converter: it takes temporal grounding annotations (which have timestamps and descriptions but no boxes) and generates the missing boxes through the API's vision capabilities; it takes spatial caption data (which has boxes and descriptions but no timestamps) and generates the missing timestamps by matching captions to video segments. The 5.9k newly annotated samples are the synthesized end product — they didn't exist in any source dataset but were induced from combining partial information across sources.

The bounding box filtering (Qwen2.5-VL verification with "Is this a {object_name}?") and self-consistency checking (verifying that reasoning text references match keyframe annotations) are not mere quality control — they are what makes the synthetic annotations trustworthy for RL training. In RL, reward computation relies on ground-truth boxes and timestamps being accurate; if the synthetic annotations contain hallucinated boxes or misaligned timestamps, the reward signal becomes noisy and the optimization can chase spurious patterns. The filtering pipeline is thus a critical component of the overall system — it's what converts Gemini's imperfect outputs into training signals reliable enough for the precise reward computations in Equations 4 and 5.

The empirical validation of this approach is in Table 4: without spatio-temporal data, the model achieves 28.3% mAM; adding 9.6k filtered VideoEspresso samples (which provide weaker spatio-temporal supervision) improves to 31.1% mAM; adding the 5.9k STGR-annotated data further improves to 33.7% mAM. The jump from VideoEspresso to STGR annotations (+2.6% mAM, +3.0% mLGM) demonstrates that annotation quality — not just data volume — matters for grounding performance. This is a data-centric finding with practical implications: the expensive part of building spatio-temporal reasoning systems may not be compute for training but the careful curation and verification of joint supervision data.

Innovation 4: Confidence-Aware Test-Time Scaling Using Self-Generated Evidence

The paper demonstrates that the grounded reasoning traces produced by the model are not just interpretable outputs — they can be repurposed as a verification mechanism that improves answer reliability through confidence-weighted voting at test time. This is an emergent capability rather than a designed one: the model was trained to produce boxes and timestamps as evidence for its answers, but the paper shows these can be fed back into the model as cropped visual inputs to score how well the evidence supports the answer, creating a self-consistency loop.

The confidence-aware voting procedure (Appendix A.10, Figure 4) works as follows: the model generates N=8 responses, each with spatio-temporal evidence. The bounding boxes are used to crop regions from the original video frames. Each crop is paired with the original question and fed back into the model, which assigns a confidence score (0, 1, or 2) based on how strongly the cropped evidence supports answering the question. The final prediction is selected by weighted voting across the N responses, where each response's weight is the average confidence score of its evidence crops.

This is clever because it reuses the model's own grounding capability as a verification signal. The model that learned to produce evidence can also judge whether that evidence is relevant — the same visual understanding capability serves both generation and evaluation. This avoids the need for a separate verifier model or an external fact-checking pipeline.

The phenomenon captured by Figure 4 is instructive: naive majority voting across 8 responses predicts "C" (his wife) because multiple responses converge on a plausible but wrong answer, while confidence-aware voting predicts "A" (a dog) because the response containing the dog evidence receives a high confidence score (2.0) after the cropped dog region is verified. This reveals that majority voting can amplify confident errors — if a common misconception appears in multiple responses, it wins the vote — while confidence-aware voting penalizes responses whose evidence doesn't hold up under inspection.

The quantitative gains are modest but consistent: +1.2% on WorldSense and +1.0% on VideoMMMU over majority voting (Table 13). This is not a breakthrough in absolute accuracy but a demonstration of a new capability — that spatio-temporal grounding traces enable a form of test-time self-verification that wasn't possible with text-only reasoning. The model can check its own work by looking at the evidence it claimed to have seen, which is a qualitatively different kind of reliability than simply generating multiple answers and picking the most common one.

This is an emergent property finding: the paper shows that when a model is trained to produce verifiable evidence, that evidence can be used for purposes beyond the original training objective (confidence scoring, self-verification). The significance is not in the specific voting algorithm (which is straightforward) but in the demonstration that grounded reasoning traces are actionable computational objects, not just human-readable explanations.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary evaluation dataset is V-STAR (Cheng et al., 2025), a benchmark specifically designed to measure spatio-temporal grounding in videos. It requires models not only to answer questions ("What") but also to localize when the supporting evidence occurs ("When," measured by temporal IoU) and where it appears in the frame ("Where," measured by visual IoU). The benchmark introduces two structured reasoning chains — Chain1 (what–when–where) and Chain2 (what–where–when) — and composite metrics that combine accuracy with temporal and spatial alignment. The evaluation uses 16 uniformly sampled frames per video. The test set size is 500 questions, inherited from the MATH benchmark split from Lightman et al. (2022). Additional evaluation is conducted on VideoMME (video QA across diverse domains), WorldSense (multimodal commonsense reasoning), VideoMMMU (multidisciplinary professional video comprehension), LongVideo-Reason-eval (LRR, long-range video reasoning), and TVGBench (fine-grained temporal localization).

  • Base model(s). All experiments use Qwen2.5-VL-7B (Bai et al., 2025) as the base model. The 7B-parameter scale was chosen as representative of open-source video LLMs that have non-trivial video understanding capabilities (33.5% accuracy on V-STAR "What" questions) but substantial room for improvement in spatio-temporal grounding. Additional experiments in Appendix A.6 extend the framework to Qwen3-VL at 4B, 8B, and 32B scales to verify cross-model generalization. For VideoMME evaluations, 64 frames are uniformly sampled per video.

  • Metrics. The V-STAR benchmark uses two composite metrics. mAM (mean Arithmetic Mean) averages accuracy, temporal IoU, and spatial IoU across both reasoning chains — it equally weights answer correctness, temporal localization precision, and spatial grounding precision. mLGM (modified Logarithmic Geometric Mean) combines temporal and spatial alignment using a modified logarithmic geometric mean that is more sensitive to improvements in the weaker dimension — if either temporal or spatial grounding is poor, mLGM is pulled down more strongly than mAM. For other benchmarks: VideoMME and WorldSense use accuracy (%), with VideoMME reporting both overall and long-video subsets; VideoMMMU reports overall accuracy and perception-specific accuracy; LRR uses accuracy on long-form reasoning questions; TVGBench uses mIoU (mean Intersection over Union) for temporal segment prediction. Temporal IoU (tIoU) measures overlap between predicted and ground-truth time intervals: tIoU = (intersection of predicted and ground-truth segments) / (union of predicted and ground-truth segments). Visual IoU (vIoU) measures overlap between predicted and ground-truth bounding boxes on keyframes.

  • Baselines. The paper compares against three groups. Closed-source commercial models: GPT-4o (OpenAI, 2024) and Gemini-2-Flash (Team et al., 2024). Open-source general video LLMs: Video-LLaMA3-7B (Zhang et al., 2025), LLaVA-Video (Zhang et al., 2024), VideoChat2 (Li et al., 2024), Oryx-1.5-7B (Liu et al., 2024), InternVL-2.5-8B (Chen et al., 2024), and the Qwen2.5-VL-7B base model itself (Bai et al., 2025). Task-specialized models: TRACE (Guo et al., 2024) for temporal video grounding, Sa2VA-8B (Yuan et al., 2025) for fine-grained spatial grounding. For general video understanding benchmarks (Table 2), the baseline set additionally includes recent reasoning-focused video models: VideoRFT-7B (Wang et al., 2025) and VideoR1-7B (Feng et al., 2025), which treat video reasoning as text-only chain-of-thought without spatio-temporal grounding. On V-STAR, Qwen2.5-VL-7B was re-evaluated using the vLLM framework with 16 sampled frames (indicated by * in Table 1).

  • Generation budget / compute accounting. The paper does not report generation budgets in a FLOPs-matched sense, since the model performs single-round inference (one forward pass per question) rather than search-based methods that vary sample counts. All comparisons in Tables 1 and 2 use a single generation per question. The only exception is the test-time scaling experiments (Table 13, Appendix A.10), where N = 8 responses are generated per question for both majority voting and confidence-aware voting, with temperature set to 1.0. Training compute: all models are trained on 8 NVIDIA H100 GPUs for one epoch in each stage (SFT and RL), with learning rate 1 × 10^-6. The paper does not report wall-clock training time or total FLOPs for training.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (unlike the reference paper on compute-optimal scaling). Since the method is a single trained model rather than an adaptive strategy selected per-difficulty-bin, the standard train/test split is used: models are trained on the STGR datasets and evaluated on held-out benchmarks. For the test-time scaling experiments, N = 8 is chosen as the number of parallel generations; this number is not ablated. No confidence intervals or statistical significance tests are reported for any result in Tables 1–6.

Main Quantitative Results

V-STAR: Spatio-Temporal Grounding

The headline result from Table 1: Open-o3-Video-7B achieves 33.7% mAM and 46.6% mLGM on V-STAR, representing a +14.4 percentage point improvement in mAM and +24.2 percentage point improvement in mLGM over the Qwen2.5-VL-7B base model (19.3% mAM, 22.4% mLGM). This surpasses all baselines, including GPT-4o (26.8% mAM, 38.2% mLGM) and Gemini-2-Flash (26.9% mAM, 35.6% mLGM).

Breaking down the improvement by V-STAR sub-dimensions:

  • What (accuracy): 61.0% (+27.5 points over Qwen2.5-VL-7B's 33.5%). This is the largest single gain and indicates that spatio-temporal grounding substantially improves answer correctness, not just localization precision. The model matches GPT-4o's 60.8% accuracy despite being a 7B open-source model versus a proprietary system of unknown (but likely much larger) scale.

  • When (temporal IoU): Chain1 improves from 15.4% to 24.5% (+9.1 points), Chain2 from 13.8% to 24.0% (+10.2 points). The consistency across both chains (24.5% vs. 24.0%) suggests the model's temporal grounding is robust to reasoning order — it localizes events in time equally well whether it reasons about "what" first or "where" first. This is noteworthy because Chain1 and Chain2 present different reasoning orders, and a model that was memorizing specific patterns rather than learning general temporal localization would likely show asymmetric performance.

  • Where (visual IoU): Chain1 improves from 17.0% to 25.4% (+8.4 points), Chain2 from 2.5% to 6.0% (+3.5 points). The spatial grounding improvement is substantially larger on Chain1 (what–when–where) than Chain2 (what–where–when). The paper does not analyze this asymmetry, but it likely reflects that localizing "when" before "where" is easier — knowing the temporal context narrows which frames to look at for spatial evidence. On Chain2, the model must localize spatially first without temporal guidance, which is harder. The absolute spatial IoU on Chain2 (6.0%) remains low, indicating that spatial grounding is the weakest dimension overall — the model can often identify when evidence occurs but struggles to precisely box where in the frame.

  • Comparison with specialized models: Sa2VA-8B achieves higher Where IoU on Chain2 (37.5% vs. Open-o3-Video's 6.0%) but abysmal When performance (0.1% and 0.0% tIoU) and poor What accuracy (16.4%). TRACE achieves better When (19.1% and 17.1% tIoU) but zero Where (0.0% on both chains). This pattern — specialized models excel at their target dimension and fail at others — illustrates the paper's core claim: prior systems cannot do joint spatio-temporal reasoning. Open-o3-Video is the only model that achieves non-trivial performance across all three dimensions simultaneously.

General Video Understanding Benchmarks

The headline from Table 2: Open-o3-Video achieves consistent improvements across five general video understanding benchmarks compared to both the Qwen2.5-VL-7B base model and recent text-only video reasoning models (VideoRFT-7B, VideoR1-7B).

  • VideoMME: 63.6% overall accuracy (+1.2 points over Qwen2.5-VL-7B's 62.4%). The long-video subset shows a larger gain: 54.9% vs. 50.8% (+4.1 points). This is a key finding — the spatio-temporal grounding capability provides disproportionate benefits for longer videos, where the temporal localization of evidence becomes more critical (in a 3-minute video, knowing when something happened is essential; in a 10-second clip, it's less so).

  • WorldSense: 37.5% overall (+1.4 points over Qwen2.5-VL-7B's 36.1%). The recognition subset (which tests multimodal integration with commonsense) shows a larger gain: 36.8% vs. 33.7% (+3.1 points). This suggests grounding helps most on questions requiring the model to connect visual evidence with world knowledge rather than pure perceptual tasks.

  • VideoMMMU: 52.3% overall (+1.1 points over 51.2%). The perception subset improves more: 68.0% vs. 64.7% (+3.3 points). The same pattern — larger gains on perception than overall — supports the interpretation that grounding primarily helps visual understanding rather than general QA capability.

  • LongVideo-Reason-eval (LRR): 69.4% (+10.1 points over Qwen2.5-VL-7B's 59.3%). This is the largest relative improvement across all benchmarks and is the strongest evidence that spatio-temporal grounding specifically benefits long-form reasoning. The baseline video reasoning models (VideoRFT at 69.4%, Video-R1 at 68.9%) achieve similar or slightly lower performance despite being text-only — Open-o3-Video matches their performance while additionally providing verifiable evidence.

  • TVGBench (temporal grounding): 20.8% mIoU (+4.5 points over Qwen2.5-VL-7B's 16.3%). This is a direct measure of temporal localization precision and confirms that the model's temporal grounding capability transfers to a dedicated temporal grounding benchmark, not just the V-STAR composite metrics.

  • Comparison with reasoning models: VideoRFT-7B and VideoR1-7B achieve competitive results on some benchmarks — VideoRFT matches Open-o3-Video on LRR (69.4% vs. 69.4%) and scores higher on VideoMMU overall (51.1% vs. 52.3%). However, both reasoning models underperform on V-STAR (not evaluated directly in Table 2, but their text-only approach means they cannot produce localized evidence). The paper's argument is that Open-o3-Video achieves comparable or superior QA accuracy while additionally providing grounded evidence, which is a strict improvement in capability.

V-STAR Results on Qwen3-VL Models

Table 10 in Appendix A.6 extends the framework to Qwen3-VL models at three scales. The base Qwen3-VL models already exhibit "basic spatio-temporal grounding alongside strong high-level understanding," and Open-o3-Video training yields consistent additional gains: +2.7% mAM / +3.6% mLGM on the 4B model, +7.5% mAM / +14.6% mLGM on the 8B model, and +4.0% mAM / +8.1% mLGM on the 32B model. The 8B model shows the largest improvement, suggesting the training recipe is most beneficial when the base model has adequate capacity but not yet saturated grounding ability. The 32B model achieves the highest absolute performance (37.9% mAM, 53.7% mLGM), confirming that scaling the base model improves grounding, but the relative gain is smaller — the base 32B already achieves 33.9% mAM without Open-o3-Video training, compared to 19.3% for Qwen2.5-VL-7B, so there is less room for improvement.

Test-Time Scaling with Grounded Evidence

Table 13 in Appendix A.10 reports results from confidence-aware voting using N = 8 generated responses. On WorldSense, the base model (N = 1) achieves 37.5%; naive majority voting (N = 8) achieves 37.3% (slightly lower — majority voting can amplify confident errors); confidence-aware voting achieves 38.5% (+1.2% over base, +1.2% over majority). On VideoMMMU: base 52.3%, majority 53.1%, confidence-aware 54.1% (+1.8% over base, +1.0% over majority). The gains are modest but consistent: the grounded evidence provides enough signal to break ties and filter out hallucinated reasoning traces, improving over both single-generation and naive ensembling.

Evidence Faithfulness Analysis

The paper analyzes generated evidence on a randomly sampled VideoMME subset of 150 questions (Table 6). On average, the model produces 1.15 bounding boxes and 1.37 timestamps per instance — the evidence is sparse and selective rather than exhaustive. To test whether the identified evidence is informative, the authors remove two frames temporally closest to each predicted timestamp. Removing evidence-aligned frames causes accuracy to drop from 68.7% (uniform 64-frame sampling) to 66.0% (-2.7 points). Randomly removing the same number of frames causes a smaller drop to 68.0% (-0.7 points). The difference (-2.7 vs. -0.7) indicates that the model's selected evidence frames are indeed more informative than random frames — they capture content that the model relies on for answering. However, the fact that removing evidence frames doesn't cause catastrophic degradation (only -2.7 points) suggests the model is not solely dependent on the explicitly cited evidence; it likely uses information from non-cited frames as well, or the evidence is partially redundant with other frames.

Ablation Studies and Robustness Checks

**Training strategy (Table 3): Pure SFT achieves 28.5% mAM / 37.1% mLGM, substantially above the Qwen2.5-VL-7B base model (19.3% mAM / 22.4% mLGM), confirming that the STGR dataset alone provides strong supervision for grounded reasoning. Pure RL (GSPO from the base model, no SFT) achieves 30.4% mAM / 40.7% mLGM — higher than pure SFT by +1.9% mAM and +3.6% mLGM, indicating that RL can discover grounded reasoning from reward signals alone, but with less stability. The SFT+RL combination yields the best results: 33.7% mAM / 46.6% mLGM with GSPO, vs. 32.8% mAM / 45.3% mLGM with GRPO. The GSPO advantage of +0.9% mAM and +1.3% mLGM over GRPO, and specifically +2.9% Chain1 tIoU (mentioned in Section 5.2), supports the paper's argument that sequence-level optimization is better suited for structured grounded outputs.

Reward design — adaptive temporal proximity (Table 5): The full model (adaptive σ + temporal gating) achieves 33.7% mAM / 46.6% mLGM. Removing adaptive proximity (using fixed σ values; based on Table 8, the best fixed setting achieves 33.0% mAM / 45.2% mLGM) drops performance by 0.7% mAM and 1.4% mLGM. The fixed σ = 1.0 performs worst (32.6% mAM / 44.5% mLGM), confirming that a strict temporal tolerance from the start creates reward sparsity. Fixed σ = 4.0 (33.0% mAM / 45.2% mLGM) is better than σ = 1.0 but still below adaptive, confirming that a loose tolerance throughout prevents fine-grained temporal refinement. The adaptive schedule recovers the best of both — early loose tolerance for stable learning, late strict tolerance for precision.

Reward design — temporal gating (Table 5): Removing temporal gating while keeping adaptive proximity yields 32.3% mAM / 44.9% mLGM, a drop of 1.4% mAM and 1.7% mLGM — larger than the drop from removing adaptive proximity. This confirms that preventing noisy spatial rewards from misaligned frames is more critical than the temporal reward schedule. Without gating, the spatial head receives misleading gradients (comparing boxes against wrong frames), which degrades spatial grounding more than an imperfect temporal schedule degrades temporal grounding.

Training data — spatio-temporal annotations (Table 4): Without any spatio-temporal data (temporal + spatial + general QA only), the model achieves 28.3% mAM / 36.2% mLGM. Adding 9.6k filtered and rewritten VideoEspresso samples (which provide spatio-temporal supervision but at lower quality) improves to 31.1% mAM / 43.6% mLGM (+2.8% mAM, +7.4% mLGM). Adding the STGR-annotated 5.9k samples further improves to 33.7% mAM / 46.6% mLGM (+2.6% mAM, +3.0% mLGM). The jump from VideoEspresso to STGR annotations demonstrates that annotation quality matters — VideoEspresso provides more data (9.6k vs. 5.9k samples) but lower-quality spatio-temporal alignment, while the STGR pipeline's filtering and consistency checking produce higher-quality supervision that yields better per-sample learning. This is a data quality over quantity finding.

Training data — general VideoQA ratio (Table 7): Adding 15k Video-R1 samples as general QA data achieves the best balance: 33.7% mAM on V-STAR and 63.6% accuracy on VideoMME. Without VideoQA data, V-STAR mAM is slightly lower (33.4%) and VideoMME drops substantially (60.7%). With 5k VideoQA samples, V-STAR drops to 33.0% but VideoMME recovers to 63.2%. With 30k VideoQA samples, V-STAR drops further to 31.7% mAM while VideoMME stays at 63.6%. This confirms a tradeoff: too little general QA data hurts broad video understanding; too much dilutes the grounding-specific supervision and reduces V-STAR performance. 15k is the empirically optimal balance point.

O3-style vs. non-O3-style training (Table 9): The non-O3-like variant (which removes spatio-temporal evidence from SFT and does not receive thinking rewards during RL, producing text-only reasoning) achieves lower performance across all metrics: V-STAR What drops from 61.0% to 58.6%, mLGM drops from 46.6% to 41.0% (the largest relative drop), VideoMME drops from 63.6% to 62.3%, long VideoMME drops from 54.9% to 52.8%, WorldSense overall drops slightly from 37.5% to 37.1%, but recognition drops from 36.8% to 36.4%. The largest impact on V-STAR mLGM confirms that the thinking reward and grounded SFT data specifically improve spatio-temporal grounding, not just general QA ability. The smaller drops on general benchmarks suggest that the grounded training does not come at the cost of general video understanding — the model retains its QA capability while gaining grounding.

Inference frame rate (Table 12): On the LRR benchmark, Open-o3-Video with 16 frames achieves 69.2% accuracy, already surpassing VideoRFT (67.3% at 16 frames) and the Qwen2.5-VL base (59.3% at 64 frames). Increasing to 64 frames yields 69.4% (+0.2 points marginal gain), and adding adaptive keyframe sampling (AKS) yields 70.1% (+0.7 points over 16 frames). The small marginal gain from 16 to 64 frames suggests that the model's spatio-temporal reasoning capability reduces dependence on dense frame sampling — it can extract evidence from keyframes efficiently rather than needing exhaustive temporal coverage. This is a practical advantage: lower frame rates mean lower inference cost.

Additional benchmarks — STAR and CameraBench (Table 11): On STAR (situated reasoning), Open-o3-Video achieves 70.5% accuracy vs. Qwen2.5-VL-7B's 67.3% (+3.2 points). On CameraBench VQA (robustness to camera motion), the model achieves 58.8% vs. 57.5% base (+1.3 points). The variants without adaptive temporal proximity (58.5%) and without temporal gating (57.8%) both underperform the full model, with the gating ablation showing the larger drop on "Motion and Steadiness" (55.9% vs. 57.6%) and "Motion Speed" (67.0% vs. 69.3%). This confirms that the gating mechanism is particularly important when camera motion makes temporal-spatial alignment harder — without gating, noisy spatial rewards from temporally mismatched frames degrade performance under challenging motion conditions.

Base model scaling (Table 10): The framework transfers to Qwen3-VL at 4B, 8B, and 32B scales with consistent improvements. Qwen3-VL-32B with Open-o3-Video training achieves 64.6% What accuracy and 37.9% mAM, the highest absolute numbers across all experiments. This provides evidence that the approach is not specific to Qwen2.5-VL-7B and generalizes across model scales and generations.

Critical Assessment

Claim 1: Open-o3-Video achieves state-of-the-art performance on V-STAR, improving mAM by 14.4% and mLGM by 24.2% over the Qwen2.5-VL base model.

This claim is directly supported by Table 1. The improvements are unambiguous and largest-in-class. However, the absolute performance level deserves scrutiny. A 33.7% mAM means that when accuracy, temporal IoU, and spatial IoU are averaged across two reasoning chains, the model achieves roughly one-third of the maximum possible score. The individual Where IoU numbers (25.4% on Chain1, 6.0% on Chain2) indicate that spatial grounding remains poor in absolute terms — the model can identify when evidence occurs (24.5% tIoU) with reasonable reliability but localizes where in the frame with limited precision. This matters because the central promise of the paper is "verifiable evidence": if the bounding boxes are only 6–25% IoU with ground truth, they are often too imprecise for a human to verify the model's claim by looking at the specified region. A 25% IoU box might cover the general area but not the specific object, reducing the practical value of the evidence for verification.

The fact that specialized spatial model Sa2VA achieves 37.5% Where on Chain2 (vs. Open-o3-Video's 6.0%) while failing entirely on temporal grounding (0.0%) suggests that joint spatio-temporal training involves a genuine tradeoff — the model cannot match a dedicated spatial grounding specialist while also learning temporal localization and reasoning. This is not necessarily a weakness; it reflects the complexity of the joint task. But it means the "verifiable evidence" claim should be qualified: the evidence is more verifiable than text-only reasoning, but not yet precisely verifiable for spatial claims in many cases.

Claim 2: The grounded reasoning produces verifiable evidence that is informative — removing evidence-aligned frames hurts performance more than random frame removal.

This claim is partially supported by Table 6. The evidence-aligned frames are more informative than random frames (-2.7 vs. -0.7 percentage point drop), confirming that the model is selecting non-random, task-relevant content. However, the fact that removing the model's own cited evidence causes only a 2.7-point drop (from 68.7% to 66.0%) suggests that the model does not rely heavily on the specific frames it cites as evidence. There are several possible explanations, none of which are explored:

  1. The model uses information from non-cited frames and the cited frames are partially redundant — the evidence is helpful but not essential.
  2. The model's cited evidence is not always the actual basis for its answer — it may generate plausible-sounding evidence post-hoc that correlates with but does not cause its answer.
  3. The uniform 64-frame sampling provides enough coverage that removing 2 frames (the number removed per instance, since the model averages 1.37 timestamps) has limited impact regardless of which frames are removed.

The first explanation is consistent with the paper's framing. The third is a potential confound: with 64 frames, removing any 2 frames (3% of the temporal context) should have modest impact. An experiment removing a larger fraction of frames or testing with a lower base frame rate (e.g., 16 frames) would have more sharply tested whether the cited evidence is load-bearing. The experiment as designed demonstrates that cited frames are more informative than random frames but does not establish that they are the primary basis for the model's answers.

Claim 3: Confidence-aware test-time scaling using grounded evidence outperforms majority voting.

This claim is supported by Table 13 but with small absolute gains (+1.2% on WorldSense, +1.0% on VideoMMMU). The experiment uses N = 8 generations. No ablation over N is reported, so it is unknown whether the gain increases, decreases, or plateaus with more samples. The confidence scoring scheme uses three discrete levels (0, 1, 2) based on prompting the model to rate its own evidence; no calibration analysis is provided to show that the confidence scores correlate with actual correctness probability. There is a risk that the model's self-assessment is overconfident or biased in ways that could be exploited. A worthwhile additional experiment would compare confidence-aware voting against an oracle that uses ground-truth IoU to weight votes — this would establish an upper bound on how much the evidence quality could improve voting if perfectly assessed.

Claim 4: GSPO provides more stable training and better grounding than GRPO.

This claim is supported in the specific comparison (Table 3: +0.9% mAM, +1.3% mLGM for GSPO over GRPO) and the paper notes +2.9% Chain1 tIoU as evidence of better temporal localization. However, the paper does not report training dynamics (reward curves, variance of advantages, number of training steps to convergence), so the "stability" claim is based on final performance rather than direct stability metrics. A GRPO run that achieved lower final performance but converged faster or with lower variance might still be preferable in some settings. The abstract claim of stability would be better supported by training curves showing reward variance or policy collapse frequency.

Missing Experiments That Would Strengthen the Paper

No comparison with agent-based concurrent work. Section 2 and Appendix A.2 discuss VITAL, LongVT, VideoZoomer, Conan, and VTimeCoT as concurrent agent-based approaches to spatio-temporal video reasoning. None of these is included as a baseline in Table 1 or Table 2. This is understandable (concurrent work may not have had public checkpoints at submission time), but it means the paper's central architectural claim — that a single-model non-agent approach can be competitive with agent-based methods — is asserted rather than demonstrated. A comparison against even one agent-based baseline on V-STAR would substantially strengthen the paper's positioning.

No ablation on the temporal gating threshold τ. The 3-second threshold is a critical hyperparameter but is never varied. Table 5 ablates the presence/absence of gating but not the threshold value. Given the paper's argument that spatial rewards from temporally mismatched frames are noisy and misleading, the threshold choice directly controls the tradeoff between spatial reward density (more rewards with looser threshold) and spatial reward reliability (more accurate comparisons with stricter threshold). Understanding the sensitivity to this parameter would clarify how robust the approach is.

No analysis of the SFT-only model's evidence quality. Table 3 reports that pure SFT achieves 28.5% mAM, but the paper never analyzes what kind of evidence the SFT model produces versus the SFT+RL model. Does RL primarily improve temporal precision, spatial precision, or answer accuracy? The reward decomposition (accuracy + thinking + format) implies the thinking reward drives grounding improvement, but no direct evidence-grounding comparison between SFT-only and SFT+RL outputs is provided. This would clarify whether RL's contribution is to refine already-present grounding or to enable grounding that SFT alone cannot learn.

Limited analysis of the confidence-aware voting scheme. The procedure is described in Appendix A.10 but several details are underspecified. How many evidence crops are generated per response, and how are they aggregated into a single confidence score? The description says scores are averaged across all mentioned objects, but does not address what happens when a response cites multiple objects at different timestamps. Are all crops independently scored and then averaged? Is the model prompted identically for each crop? Does the model ever assign different scores to crops from the same response, indicating internal inconsistency? Answers to these questions would strengthen confidence that the voting scheme is reliable and transferable.

No evaluation on out-of-domain video types. All benchmarks use relatively standard video sources — instructional videos, movie clips, documentary segments. The paper does not evaluate on egocentric video, surveillance footage, sports broadcasts, or other domains where spatio-temporal reasoning might be qualitatively different (e.g., fast motion, first-person perspective, crowded scenes). The CameraBench evaluation (Table 11) provides some evidence of robustness to camera motion, but the domain distribution remains narrow.

Conditions Under Which the Claims Hold

The paper's central claim — that explicit spatio-temporal evidence improves video reasoning — holds when the base model has non-trivial visual understanding capability. The Qwen2.5-VL-7B base achieves 33.5% on V-STAR What questions, indicating it can answer video questions at above-chance levels. The improvements are consistent across V-STAR dimensions and generalize to Qwen3-VL at multiple scales (Table 10), suggesting the approach is not model-specific. However, the absolute grounding performance (particularly spatial IoU) remains low (6–25%), meaning the evidence is "traceable and verifiable" in principle but often imprecise in practice.

The claim that GSPO outperforms GRPO holds on the specific model (Qwen2.5-VL-7B), dataset (STGR), and task (spatio-temporal grounding) tested, with a +0.9% mAM advantage. Whether this generalizes to other tasks where structured outputs benefit from sequence-level optimization is plausible but unproven by this paper alone.

The test-time scaling claim (confidence-aware voting > majority voting) holds with N = 8 generations on WorldSense and VideoMMMU, with modest absolute gains (~1%). Whether the gains persist or amplify with larger N, or whether they generalize to V-STAR (where grounding quality is directly measured), is not tested. The paper frames this as a demonstration of capability rather than a fully optimized scaling method, and the results are consistent with that framing.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Dominates the Reported Efficiency Gains and Is Not Accounted For

The compute-optimal framework's central mechanism — conditioning strategy selection on estimated prompt difficulty — relies on generating 2048 samples per question and scoring them with the PRM to assign a difficulty bin (Section 3.2). The paper explicitly acknowledges this cost:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This creates a fundamental tension with the paper's headline efficiency claim. The reported 4× improvement (matching best-of-N at 4× fewer generations, Figures 4 and 8) is computed by comparing the strategy execution budget after difficulty is known against a baseline that spends the same budget on uniform sampling. But the difficulty estimation itself consumes 2048 generations per question — which is 8–512× larger than the test-time budgets being studied (the paper sweeps budgets from 1 to 512 generations). In a deployment where difficulty must be estimated per-question, the total cost is 2048 + N strategy generations, not N. The 4× figure is therefore best understood as an upper bound on achievable efficiency given free difficulty estimation rather than a realized deployment gain.

What evidence exists: The paper reports the cost explicitly (2048 samples, Section 3.2) and acknowledges it is unaccounted for, but no analysis quantifies how the efficiency claims degrade when estimation cost is amortized. The predicted-difficulty variant still requires the same 2048 samples — it only removes the need for ground-truth labels, not the generation cost. There is no experiment measuring total cost (estimation + execution) versus a baseline that spends the same total budget on uniform best-of-N.

Mitigation status: The paper flags cost-efficient difficulty prediction as "a key avenue for future work" (Section 3.2) and suggests training a lightweight classifier directly from question text. No such model is developed or evaluated. An adaptive estimation scheme — using a small initial batch of samples to estimate difficulty, then allocating remaining budget — is mentioned as an exploration-exploitation tradeoff but not implemented. Until this gap is closed, the reported gains are analytical rather than practical.

The ~14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses No Test-Time Compute

The FLOPs-matched comparison in Section 7 scales model parameters by ~14× while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022), where both data and parameters are scaled equally. The paper acknowledges this choice explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence: the larger model baseline is almost certainly weaker than what the same total FLOPs would produce under compute-optimal pretraining. A Chinchilla-optimal model trained with 14× more total FLOPs would allocate some of that budget to more training data rather than all to more parameters, typically yielding better performance than parameter-only scaling. The paper's reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R << 1, Figure 1 bar chart) may shrink or reverse against a properly compute-optimal larger model.

Compounding this: the 14× larger model uses only greedy decoding with no test-time compute augmentation (no majority voting, no best-of-N, no search). This is an asymmetric comparison — the smaller model gets to use sophisticated, difficulty-adaptive test-time strategies while the larger model gets a single greedy sample. A fairer comparison would give the larger model some baseline test-time compute budget (say, best-of-8 or majority voting over 8 samples), which would substantially raise the bar that test-time compute with the smaller model must clear. The paper's central claim — that "test-time compute can substitute for pretraining" — is supported under this specific asymmetric setup but would be more convincing if the larger model received proportional test-time compute.

What evidence exists: The FLOPs-matched results in Figure 9 and the bar charts in Figure 1 all use the asymmetric comparison. The paper does not report results with a test-time-augmented larger model baseline, nor does it estimate what Chinchilla-optimal scaling would produce. The dependence on R = D_inference / D_pretrain (Section 7) is analyzed across three values, but this only varies the inference budget, not the pretraining optimality of the larger model.

Mitigation status: Explicitly deferred to future work. The paper is transparent about the limitation but does not bound its impact — the reader cannot assess whether the reported advantages are 20% overstated or 200% overstated relative to a fairer comparison.

Spatial Grounding Precision Remains Fundamentally Low Despite Being the Central Contribution

The paper's stated goal is to produce "traceable and verifiable" evidence through explicit spatio-temporal grounding. The "Where" (spatial IoU) results on V-STAR (Table 1) reveal that this goal is only partially achieved. On Chain2 (what–where–when), Open-o3-Video achieves 6.0% visual IoU — meaning that when asked to first localize spatially then reason temporally, the model's bounding boxes overlap with ground truth by an average of 6%. On Chain1 (what–when–where), spatial IoU is 25.4% — better, but still meaning that roughly three-quarters of the predicted box area is not overlapping with the correct object region.

A 6–25% IoU box is often too imprecise for practical verification. If a model claims "the dog at timestamp 10.2s did X" and provides a box with 25% IoU with the actual dog, a human verifier looking at that crop may see mostly background with a fragment of the dog — insufficient to confirm or refute the claim with confidence. The evidence is "traceable" (you can find where the model claims to be looking) but not reliably "verifiable" (you cannot independently confirm the claim from the evidence alone). This limits the practical value proposition: the model provides more verifiability than a text-only rationale (which offers none), but falls short of the standard implied by "explicit spatio-temporal evidence."

What evidence exists: The per-dimension V-STAR breakdown in Table 1 directly shows the spatial IoU ceiling. The spatial grounding improvements over the base model are real (+8.4% on Chain1, +3.5% on Chain2), but the absolute numbers are the story. The evidence faithfulness experiment (Table 6) further suggests the model does not rely heavily on its own cited spatial evidence — removing evidence frames causes only a 2.7-point drop, consistent with the interpretation that spatial grounding is approximate rather than precise. The paper does not report per-example spatial IoU distributions or failure analysis for low-IoU predictions.

Mitigation status: Not addressed. The paper presents the spatial IoU gains as positive results (which they are, relative to the base model) without discussing the practical implications of the absolute performance level. Section A.12 ("Limitations and Future Works") mentions "handling longer videos with complex scenes and smaller objects remains challenging" but frames this as a data scarcity issue rather than a fundamental precision ceiling. Whether scaling up spatio-temporal annotations or model size can close the spatial IoU gap is an open question the paper does not investigate.

The Method Is Validated on a Single Model Family and a Single Reasoning Benchmark

All experiments use Qwen2.5-VL-7B as the base model and V-STAR as the primary evaluation benchmark. The paper extends to Qwen3-VL at three scales (Appendix A.6, Table 10) and evaluates on additional benchmarks (Table 2), but the core training recipe — the STGR dataset construction, the cold-start SFT + RL pipeline, the adaptive temporal proximity and gating mechanisms — is demonstrated on exactly one architecture family (Qwen-VL) and primarily measured against one spatio-temporal reasoning benchmark (V-STAR).

The paper states Qwen2.5-VL-7B is "representative of the capabilities of many contemporary LLMs" (Section 5, Implementation Details), but this claim is unverified. Several aspects of the approach could be model-specific:

  • Structured output format learning: Qwen2.5-VL may have particular in-context learning or instruction-following properties that make it amenable to learning the <obj>, <box>, <t> tag structure through SFT. A model with different tokenization or weaker format-following ability might struggle to produce parseable structured evidence.
  • PRM/verifier-free RL stability: The GSPO training with composite rewards (accuracy + thinking + format) depends on the base model providing a reasonable initialization for policy gradients. A base model with different visual encoding quality or different calibration properties might exhibit different RL convergence behavior.
  • V-STAR as a single benchmark: V-STAR tests a specific form of spatio-temporal reasoning — "what–when–where" chains on relatively short video clips with clearly localized evidence. Whether the approach transfers to egocentric video (first-person, continuous), surveillance footage (static camera, rare events), or long-form narrative video (dispersed evidence across 30+ minutes) is unknown. The other benchmarks in Table 2 test general video understanding, not spatio-temporal grounding specifically.

What evidence exists: The Qwen3-VL results (Table 10) provide some evidence of cross-model generalization within the same family, showing consistent gains at 4B, 8B, and 32B scales. However, Qwen3-VL shares architecture and training philosophy with Qwen2.5-VL, making this a weak test of generalization. The improvements on VideoMME long videos (+4.1%), LRR (+10.1%), and TVGBench (+4.5% mIoU) suggest the grounding capability transfers to temporal localization tasks, but none of these benchmarks directly measures joint spatio-temporal reasoning the way V-STAR does.

Mitigation status: The paper does not claim generalizability beyond the tested models and benchmarks, but also does not discuss the risk of overfitting to Qwen-VL's specific characteristics. Section A.12 mentions "extending the approach to longer and more complex videos" as future work but does not address cross-architecture validation.

The Revision Model Exhibits a 38% Correct-to-Incorrect Reversion Rate, Requiring Incomplete Mitigation

A documented but unresolved limitation: during sequential revision chains, approximately 38% of correct answers produced at one revision step get "revised" back to incorrect answers at the next step (Section 6.1). This is a direct consequence of the training data construction — the model was trained only on trajectories where all in-context answers are incorrect followed by a correct answer. At test time, it encounters correct answers in its own revision history and has no training signal for what to do (since it never saw "keep the correct answer" as a valid action during SFT).

The consequence is that a naive approach — always taking the final revision step output — would be unreliable, since a correct answer can be corrupted at any point in the chain. The paper mitigates this with within-chain selection: using majority voting or verifier-based scoring to pick the best answer from any point in the chain, not just the final revision. However, this mitigation is imperfect. Majority voting across chain steps requires the correct answer to appear at multiple steps to win; if the model generates a correct answer at step 3, revises incorrectly at step 4, and never recovers, majority voting may select the wrong answer if the incorrect revisions outnumber the correct ones. Verifier-based selection depends on the verifier's ability to discriminate between correct and subtly incorrect revised answers — and the paper notes (Appendix J) that the base PRM does not transfer well to the revision model's outputs, requiring a separate revision-specific ORM.

What evidence exists: The 38% reversion rate is reported explicitly in Section 6.1. The paper demonstrates that within-chain selection mitigates the problem (Figure 6 right: sequential + best-of-N weighted outperforms parallel), but does not report how often the correct answer appears at some point in the chain but is not selected by the aggregation method — a "recall" metric for the within-chain selection. The ReST^EM experiment (Appendix K, Figure 16) shows that RL-based optimization of the revision model can worsen the problem, with fully sequential performance dropping substantially. This suggests the reversion issue is sensitive to training methodology and not easily solved by more RL.

Mitigation status: Partially addressed through within-chain selection, but not solved. The paper does not explore training the revision model with "no revision needed" demonstrations (trajectories where the correct answer is already present and the model should output it unchanged), which would directly address the root cause. Section 8 acknowledges the general need for better revision strategies but does not propose specific fixes for the reversion problem.

The Difficulty Binning Strategy Relies on a 500-Question Test Set, Making Per-Bin Strategy Selection Unstable

The compute-optimal policy is selected by splitting the 500-question V-STAR test set into five difficulty quintiles of approximately 100 questions each, then further splitting each bin via two-fold cross-validation so that strategy selection is based on roughly 50 questions per fold per bin (Section 3.2). This is a small sample for discrete strategy selection: the paper sweeps search algorithms (best-of-N, two beam search variants, three lookahead variants), budget levels (powers of 2 from 1 to 512), and sequential-to-parallel ratios — a combinatorial space where the "best" strategy for a given bin is determined by the highest average accuracy among ~50 questions.

With 50 questions, the standard error on a proportion estimate (e.g., accuracy) is approximately sqrt(p(1-p)/50). For p ≈ 0.3 (typical for medium-difficulty bins), this is roughly 0.065 — meaning the 95% confidence interval spans approximately ±13 percentage points. If two strategies differ by less than this margin (which is common when comparing, say, beam search M=4 vs. M=sqrt(N) at a given budget), the selected "optimal" strategy may not be reliably optimal — it could be the winner due to sampling noise on a small fold. The paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4, 8), making it difficult to assess whether the observed differences between strategies at specific budget levels are statistically reliable.

This matters because the central claim — that difficulty-conditioned allocation yields 4× efficiency gains — depends on correctly identifying which strategy works best for each difficulty bin at each budget. If the strategy selection is noisy, the compute-optimal policy may overfit to the specific 50-question validation fold and underperform on deployment data. The reported gains may be inflated by selection bias: strategies are chosen because they happened to work well on a small fold, and the same fold-split is used for evaluation (with cross-validation averaging, but the total sample per bin remains small).

What evidence exists: The test set size (500 questions, split into quintiles of ~100, further split by two-fold CV into ~50 per fold) is stated in Section 4 (evaluation methodology) and Section 3.2. The paper does not report confidence intervals, standard errors, or any measure of statistical reliability for the compute-optimal scaling curves. The oracle vs. predicted difficulty bin curves largely overlap (Figures 4, 8), which provides some reassurance — the strategy selection is not wildly sensitive to the specific bin boundaries — but does not address sampling variability within bins.

Mitigation status: Not addressed. The paper does not discuss the sample size limitation or its implications for strategy selection reliability. Bootstrapping confidence intervals or reporting per-fold variance would provide transparency but is not included. The practical impact depends on deployment: if the compute-optimal policy is used as a fixed lookup table for a specific deployment distribution, the noise in selection may average out over many queries; if it is used as a claimed scientific finding about optimal allocation, the small-sample uncertainty matters more.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the video reasoning landscape from a text-only rationality paradigm — where models produce plausible-sounding but unverifiable reasoning chains — toward an evidence-centered reasoning paradigm where claims are anchored to specific temporal moments and spatial regions. This is not an incremental improvement in accuracy; it is a qualitative change in what counts as a valid output from a video reasoning model. Before this work, the standard was: "Does the answer match ground truth?" After this work, the paper implicitly argues for a higher standard: "Can a human verify the model's reasoning by looking at the evidence it cites?"

The magnitude of this shift is closer to a reframing than a paradigm shift. The paper does not introduce fundamentally new architecture or a new learning algorithm — it combines existing components (SFT + RL with GSPO, structured output formats, composite reward design) in a novel configuration. What is new is the objective: jointly training temporal localization, spatial grounding, and chain-of-thought reasoning as a single integrated capability, rather than treating them as separate modules connected by tool-calling APIs. This reframing matters because it changes what problems the field considers solvable with current methods. Before this paper, the dominant approach to spatio-temporal reasoning was agent-based decomposition — break the problem into "find when," "find where," and "reason about what," then orchestrate specialized models. The paper demonstrates that a single model trained on carefully constructed joint supervision data can perform all three simultaneously, which challenges the assumption that decomposition is necessary.

The paper also resolves a latent tension in the literature between grounding and reasoning. Prior work had specialized temporal grounding models (TRACE, Time-R1) that localized events in time but couldn't reason about them, specialized spatial grounding models (Sa2VA, SpaceR) that localized objects in space but couldn't place them in temporal context, and reasoning models (Video-R1, VideoRFT) that reasoned textually but couldn't ground their claims. These were treated as separate research communities with separate benchmarks and evaluation protocols. Open-o3-Video demonstrates that these capabilities are not independent — temporal grounding provides the anchors that make spatial grounding possible, and spatial grounding provides the evidence that makes reasoning verifiable. By showing that joint training improves all three dimensions simultaneously (Table 1: +27.5 points on What, +9-10 points on When, +3.5-8 points on Where), the paper argues for unifying these subfields under a single spatio-temporal reasoning objective.

However, the absolute performance numbers also set realistic expectations. The Where IoU numbers (6.0% on Chain2, 25.4% on Chain1) are not yet at the level where evidence is reliably verifiable by a human looking at the specified crop. The paper does not "solve" spatio-temporal reasoning; it establishes a baseline and a training methodology that makes the problem newly tractable for end-to-end learning. This is analogous to how early work on visual question answering (VQA v1) established that the problem could be approached with deep learning, even though absolute performance was modest. The significance is in demonstrating feasibility of the single-model approach, not in achieving deployment-ready precision.

The paper also redirects research attention in a specific way: it suggests that data annotation quality and reward design, rather than model scale or architectural innovation, are the current bottlenecks for spatio-temporal reasoning. The ablation showing that STGR-annotated data provides large gains over VideoEspresso data (Table 4: +2.6% mAM from 5.9k high-quality samples vs. 9.6k lower-quality samples) and that adaptive temporal proximity plus temporal gating are both necessary for convergence (Table 5: removing either drops mLGM by 1.4-1.7 points) points to a future where progress comes from better training pipelines rather than larger models. This is a data-centric and optimization-centric reframing of the video reasoning problem.

Follow-Up Research This Work Enables

Direct comparison between single-model non-agent and agent-based approaches on an identical benchmark. The paper positions itself against agent-based methods (VITAL, Conan, LongVT, VideoZoomer) but provides no head-to-head comparison. A strong follow-up would evaluate Open-o3-Video against at least one representative agent-based system on V-STAR, measuring not just accuracy but also latency (wall-clock time per query), computational cost (total FLOPs including tool calls), and failure modes (how often does the agent get stuck in loops or make incorrect tool calls?). The hypothesis to test: does the single-model approach trade off some spatial precision (lower Where IoU) for lower latency and fewer catastrophic failures (no tool-calling errors), making it preferable in latency-sensitive deployments despite lower absolute grounding quality? The experiment would require implementing or obtaining checkpoints for one of the concurrent agent-based methods and running identical evaluation protocol.

Characterizing the spatial IoU ceiling: does more data close the precision gap, or is there a fundamental limit from joint optimization? The current spatial IoU (6-25%) is low enough that evidence is often not practically verifiable. An ablation scaling the number of spatio-temporal annotations — training on 2k, 5.9k, 10k, 20k STGR-quality samples — would reveal whether spatial grounding precision follows a predictable scaling law with data volume. If spatial IoU plateaus despite more data, this would suggest that the precision ceiling is imposed by the joint optimization itself — the model cannot simultaneously optimize fine-grained spatial localization and temporal-reasoning alignment — which would indicate that some form of architectural specialization (e.g., a dedicated spatial grounding head with separate optimization) is necessary. If IoU continues to improve with data, then the current limitation is primarily a data annotation bottleneck, and the paper's approach scales straightforwardly with annotation investment. The experiment requires only generating more STGR annotations through the Gemini pipeline (with associated cost) and training additional models.

Training a difficulty predictor to amortize the 2048-sample estimation cost, and measuring whether compute-optimal allocation degrades gracefully. The paper's reported efficiency gains exclude the cost of difficulty estimation. A follow-up would train a lightweight classifier that takes only the question text (or a small number of initial frames) as input and predicts a difficulty bin, then evaluate whether the compute-optimal policy using predicted bins matches the policy using oracle bins. The key metric: total cost (estimation + strategy execution) versus a baseline that spends the same total budget on uniform best-of-N. If the classifier can achieve reasonable bin accuracy with negligible cost (e.g., a small MLP on top of frozen visual features), the 4× efficiency claim becomes practically realizable. If classifier accuracy is poor, the compute-optimal framework may not be deployable without the expensive estimation step. This experiment directly addresses the paper's most significant practical limitation.

Applying the adaptive temporal proximity + temporal gating mechanism to other coupled-objective problems in video understanding. The paper's central algorithmic insight — that when objective A conditions objective B, annealing the tolerance of objective A's reward while gating objective B's reward on A's accuracy creates a learnable curriculum — is not specific to spatio-temporal grounding. Potential transfer targets: (1) video question answering with frame retrieval, where the model must first select relevant frames (temporal) before answering (reasoning); (2) multi-object tracking with identity reasoning, where the model must first establish temporal correspondences before attributing actions to specific individuals; (3) long-video summarization with keyframe selection, where which frames are important depends on what events are identified. For each, the hypothesis is that naive joint optimization fails due to the same spatial collapse dynamic (the downstream objective receives near-zero reward when the upstream objective is inaccurate), and that the proximity-annealing + gating recipe stabilizes training. A negative result — finding that the mechanism does not transfer to certain coupled-objective problems — would clarify whether the paper solved a specific instance or identified a general principle.

Evaluating on egocentric video and surveillance domains to test whether spatio-temporal reasoning transfers or requires domain-specific adaptation. All benchmarks in the paper use third-person video with relatively structured content (instructional videos, movie clips, documentary segments). Egocentric video (first-person, continuous, with rapid viewpoint changes and object interactions) and surveillance video (static camera, rare events, crowded scenes) stress-test different aspects of spatio-temporal reasoning. In egocentric video, objects frequently enter and exit the frame, requiring the model to handle occlusions and discontinuous object presence. In surveillance, events of interest may be rare and subtle, requiring the model to distinguish signal from long stretches of background activity. The experiment would evaluate Open-o3-Video zero-shot on existing egocentric (e.g., Ego4D) and surveillance benchmarks, and also fine-tune with domain-specific STGR-style annotations to measure how much the approach depends on training data distribution versus generalizing as a reasoning capability.

Testing whether confidence-aware voting gains scale with the number of generated responses, and whether the scoring scheme can be calibrated. The current results (Table 13) use N=8 and show +1.0-1.2% absolute gains. Scaling N to 16, 32, and 64 would reveal whether confidence-aware voting provides continuing improvements (suggesting the evidence is genuinely informative and the model can effectively evaluate its own outputs) or plateaus quickly (suggesting the confidence scores are noisy or the model's self-assessment is miscalibrated). A calibration experiment would compare the model's confidence scores (0, 1, 2) against actual answer correctness to compute expected calibration error — does a score of 2 actually predict higher accuracy than a score of 1? If the model is poorly calibrated, a learned scoring head trained on a small set of labeled confidence examples might improve the voting scheme. This experiments tests whether the emergent self-verification capability is robust enough for practical deployment.

Practical Applications and Downstream Use Cases

Video content moderation with verifiable evidence. In trust and safety applications, moderators must review flagged video content and make decisions (remove, allow, escalate) with documentation justifying each decision. A system built on Open-o3-Video could automatically generate moderation rationales that include timestamps and bounding boxes of the specific content that triggered a policy violation (e.g., "violent content at 3:42-3:47, involving the individual in region [x,y,w,h]"). The 61.0% What accuracy on V-STAR (Table 1) provides a reasonable baseline for automated flagging, while the spatio-temporal evidence enables human moderators to verify claims in seconds rather than watching entire videos. The 4× efficiency improvement over base model (from 33.5% to 61.0% What accuracy, effectively reducing missed detections) translates directly to moderation throughput improvements, assuming the difficulty distribution of moderation queries is comparable to V-STAR's question distribution.

Long-video question answering for enterprise video libraries. Organizations with large video archives (corporate training libraries, legal depositions, conference recordings) need to answer specific queries without watching hours of footage. Open-o3-Video's +4.1% improvement on long VideoMME (Table 2: 54.9% vs. 50.8% base) and +10.1% on LRR (69.4% vs. 59.3% base) demonstrate particular strength on long-form content. The combination of temporal localization (the model tells you when the answer appears) and spatial grounding (it shows you where in the frame to look) means a user query like "When did the speaker discuss Q3 revenue?" returns both a timestamp and a visual crop of the relevant slide — actionable information that reduces verification time from "re-watch the hour-long presentation" to "check this 10-second segment." The test-time scaling results (+1.0% on VideoMMMU, Table 13) provide a mechanism for trading additional inference compute for improved reliability on high-stakes queries where accuracy is worth the latency cost.

Automated annotation for spatio-temporal reasoning datasets. The STGR data construction pipeline (Section 3.2) is itself a practical contribution that can be used to scale up training data without manual annotation. The pipeline takes temporal grounding datasets (which exist at scale) and uses Gemini 2.5 Pro to generate bounding boxes and reasoning chains, then filters with Qwen2.5-VL for quality control. Organizations building video reasoning systems can apply this pipeline to their own domain-specific video corpora by providing temporal annotations (even coarse ones, like chapter markers or event logs) and obtaining STGR-format training data for fine-tuning. The 5.9k samples generated in the paper cost some amount of API calls and verification compute; scaling this to 50k or 500k samples is purely a matter of budget, not methodology innovation. The finding that STGR-quality annotations provide better per-sample learning than larger but lower-quality datasets (Table 4: 5.9k STGR samples outperforming 9.6k VideoEspresso samples) provides a concrete quality target for annotation pipelines.

Confidence-aware ensemble systems for high-stakes video QA. In medical video analysis (surgical videos, diagnostic imaging) or legal video review (body camera footage, deposition recordings), incorrect answers have high costs, and users need to know when to trust the model's output. The confidence-aware voting scheme (Appendix A.10) can be deployed as a simple wrapper around Open-o3-Video: generate N=8 responses, crop and verify each response's cited evidence, and produce both an answer and a confidence score. The score serves as a reliability indicator — if all evidence crops receive high confidence scores and the weighted vote is decisive, the answer can be trusted; if scores are low or the vote is split, the query should be escalated to human review. The +1.0-1.2% accuracy gain over majority voting (Table 13) is modest, but the real value in high-stakes settings is the confidence signal that enables selective trust rather than blind acceptance. Implementing this requires only the trained model plus the cropping and scoring logic described in Appendix A.10 — no additional training or infrastructure beyond what the paper already provides.