SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning

ArXiv: 2512.13874

🎯 Pitch

A video reasoning agent trained with RL shows a surprising 8.2% accuracy jump on videos over 10 minutes—but only if it first learns basic tool use through supervised fine-tuning, otherwise RL collapses the agent into ignoring tools and answering in one step.

---

1. Executive Summary

This paper introduces SAGE (Smart Any-horizon aGEnt), an agent-based system for long-video reasoning that performs multi-turn, tool-assisted reasoning on complex problems while handling simple queries in a single turn—mirroring how humans adaptively skim or watch videos in full depending on the task. The system is trained and evaluated on SAGE-Bench, a curated benchmark of over 1700 manually verified open-ended and multiple-choice questions drawn from popular YouTube entertainment videos averaging 727 seconds in duration. At its core lies SAGE-MM, an orchestrator vision-language model fine-tuned with a multi-reward reinforcement learning recipe (GRPO with step-level format and tool-use rewards plus an LLM-as-judge accuracy reward), which learns to reason across any temporal horizon by deciding when to invoke tools—web search for external knowledge, speech transcription for verbal content, temporal grounding for event localization, and frame extraction for visual analysis—rather than relying solely on exhaustive frame processing. SAGE achieves up to 6.1% improvement over the base Qwen3-VL-8B-Instruct model on open-ended questions and a striking 8.2% gain on videos longer than 10 minutes, establishing that agentic any-horizon reasoning substantially outperforms both direct single-turn video models and prior agent systems, but only when the orchestrator is first bootstrapped through a cold-start supervised fine-tuning stage—without which reinforcement learning collapses the model into single-turn behavior.

2. Context and Motivation

The Core Problem: Video Understanding Models Are Trained for a Single Mode of Reasoning

The fundamental question this paper tackles is: how should video reasoning models handle videos of widely varying lengths and complexity? This matters because humans exhibit a natural any-horizon reasoning ability — we skim long videos to find relevant segments, watch short clips in full, and seamlessly combine external knowledge with what we see on screen. A person watching a two-hour Formula 1 race doesn't process every frame equally; they iteratively locate key moments, draw on knowledge of team standings, and listen to commentary when visual information is insufficient.

In contrast, state-of-the-art video reasoning models operate in what the paper calls the Direct paradigm: given a set of sampled frames, the model produces an answer through a single sequence prediction process — one forward pass, one response. Whether the video is 30 seconds of a cooking clip or a two-hour documentary, the approach is uniform. This is computationally inefficient for long videos (processing many frames that may be irrelevant) and rigid for complex queries that require integrating information from multiple temporal locations or external sources.

The paper introduces a taxonomy (Section 1, Figure 1) that divides existing approaches into two camps:

• Direct paradigm: Single-turn reasoning from sampled frames. Models like Qwen3-VL, Gemini-2.5, InternVL3, and LongVILA fall here, even when trained with reinforcement learning. They process frames and output an answer in one shot.
• Agent paradigm: Multi-turn reasoning where an orchestrator model decides when to invoke tools (temporal grounders, search, transcription) across multiple steps before producing a final answer. Systems like VideoAgent, VideoMind, LVAgent, and VideoExplorer fall here.

The gap the paper identifies is that neither paradigm is sufficient alone for real-world video interaction. Direct models are wasteful on long videos (processing hundreds of frames for a question answerable from a 10-second segment) and cannot dynamically seek external information. Agent models, while more flexible, have been over-engineered toward multiple-choice benchmarks and rely almost exclusively on temporal grounding as their primary reasoning mechanism — they lack the ability to reach outside the video for knowledge, and their orchestrators are not trained to produce direct answers when multi-turn reasoning is unnecessary. What's missing is a system that decides per-query whether to answer directly or engage in multi-turn tool use — the any-horizon capability that humans exercise effortlessly.

Why This Problem Matters: Entertainment Video Understanding Is Underrepresented

The paper makes a deliberate, non-obvious choice to focus on entertainment videos rather than the conventional benchmarks of instructional, cooking, or egocentric video. The authors cite survey evidence (Hafuta et al., 2025; Dannenbaum et al., 2025) that entertainment is the primary purpose for which humans interact with videos in daily life — from watching sports highlights on YouTube to scrolling through comedy clips. If video reasoning models are to serve real users, they must handle the open-ended, temporally distributed queries that arise naturally when watching entertainment content: "How does the Ferrari livery look this year?" (Figure 1a), "What did Mr. Bean desire in the 'Pow Box' segment?" (Figure 15), or "What phrase did the speaker end with?" (Figure 16).

Existing benchmarks are poorly aligned with this use case along multiple dimensions:

• MCQ dominance: Popular long-video benchmarks like MLVU, Video-MME, LongVideoBench, and LVBench rely almost exclusively on multiple-choice questions with fixed answer sets. This enables string-matching evaluation but bears little resemblance to how users actually query video content — people ask open-ended questions and expect descriptive, free-form answers.
• Diagnostic rather than practical: Benchmarks like MLVU contain tasks specifically designed to probe model capabilities (temporal ordering, action counting, anomaly detection) rather than questions a real user would ask while watching. These are valuable for measuring technical proficiency but miss the practical utility dimension.
• Short average duration: Many existing benchmarks feature videos substantially shorter than the 727-second average of SAGE-Bench, failing to stress-test the long-horizon reasoning that real-world entertainment videos demand.

This benchmark gap has a direct consequence for model development: models optimized for MCQ benchmarks through string-matching rewards may fail on open-ended, real-world queries. The paper demonstrates this empirically — Video-R1, VideoRFT, and other RL-tuned Direct models actually underperform their untuned base models on open-ended questions in SAGE-Bench (Table 4), despite showing improvements on MCQ-heavy benchmarks during their own evaluations.

Where Prior Approaches Fall Short

The paper identifies specific, actionable limitations across both the Direct and Agent paradigms:

Limitation 1: Direct Models Cannot Adapt Their Compute to Video Length

Direct video understanding models — including the latest RL-tuned variants like Video-R1, VideoRFT, LongVILA-R1, and Video-Thinker — process a fixed number of sampled frames regardless of the question's complexity. If the question is "What color was the car at 2:15?", processing all 128 frames from a two-hour video is wasteful when only the frames around the 2:15 mark are relevant. Conversely, if the question requires integrating information from across the video (e.g., "How many times did the speaker change topic?"), a fixed frame budget may undersample critical moments.

More subtly, the RL training procedures used for these models are fundamentally tied to MCQ evaluation. Video-R1 (Section 2.2) and VideoRFT use option-matching and ROUGE-L metrics as their reward signal during GRPO training. This means the models are optimized to produce short, answer-letter responses (A, B, C, D) rather than descriptive free-form text. When deployed on open-ended problems — which constitute 54% of SAGE-Bench — these reward structures become actively harmful, as the model has been trained to output in a format that doesn't match the task. The paper's results bear this out: Video-R1 achieves 73.6% on MCQ but only 43.9% on open-ended questions (Table 4), a gap of nearly 30 percentage points.

Limitation 2: Agent Systems Over-Rely on Temporal Grounding

Existing long-video reasoning agents — VideoAgent, VideoMind, VideoExplorer, LVAgent, and VideoChat-A1 — share a common architectural pattern: an orchestrator model iteratively invokes a temporal grounding tool to locate events in the video, then reasons about the grounded segments. Temporal grounding is the sole mechanism for navigating the video's content.

The paper argues this is insufficient on two grounds:

First, temporal grounding models are unreliable on long entertainment videos. The authors found qualitatively (Section 3.1) that existing grounding models struggle when asked to locate events across entire videos longer than 10 minutes. SAGE addresses this by having the orchestrator predict coarse segment-level boundaries (maximum 10-minute windows) rather than grounding over the full video — essentially decomposing long-video grounding into sub-problems that existing models can handle more reliably.

Second, pure temporal grounding ignores non-visual reasoning pathways. Many real-world questions about videos require external knowledge or verbal content that temporal grounding alone cannot provide. For example, identifying "How does the Ferrari livery look this year?" (Figure 1a) benefits from knowing the 2024 season standings — information not present in the video frames themselves. SAGE equips its orchestrator with web search to retrieve external facts, speech transcription to access verbal content (commentary, dialogue), and frame extraction with analysis for targeted visual inspection — tools absent from prior agent systems.

The empirical consequence of this over-reliance is stark: existing agent systems underperform dramatically on open-ended questions in SAGE-Bench. VideoMind achieves 33.2% on open-ended vs. 69.7% on MCQ (Table 4); VideoExplorer achieves 35.1% vs. 69.6%. These are not models that gracefully degrade on harder question formats — they essentially break down, suggesting their tool-use strategies are optimized for the narrow distribution of MCQ-style temporal reasoning.

Limitation 3: RL for Video Agents Lacks Appropriate Reward Design

The paper identifies a specific, underexplored challenge: applying RL to train multi-turn video reasoning agents requires reward signals that work for open-ended answers and variable-length trajectories. Prior RL work on video reasoning (Video-R1, VideoRFT, LongVILA-R1) sidesteps this by restricting to MCQ problems where option-matching provides a clean verifiable reward. But this strategy cannot train models for the open-ended responses that real users expect.

The paper's solution — LLM-as-judge accuracy rewards combined with step-level tool-use rewards — addresses a gap that has no clear precedent in the video reasoning literature. The step-level rewards (format compliance, reasonable tool selection, argument validity, repetition penalties) are carefully designed so that their accumulated value over a maximum-length trajectory is comparable to the binary accuracy reward (Section 3.3). This prevents the model from being incentivized to take unnecessary tool calls just to accumulate positive step rewards — a subtle but important design choice that the naive alternative (only rewarding final correctness) wouldn't address.

Additionally, the paper observes a cold-start problem for RL in the agent setting. When RL was applied directly to the base Qwen3-VL-8B-Instruct model without supervised fine-tuning, the model collapsed to single-turn behavior — it stopped invoking tools entirely (Table 13). The authors hypothesize this is because the base model's training objective strongly biases it toward direct answer generation, and without initial exposure to multi-turn trajectories through SFT, the exploration space during RL is too vast for the model to discover effective tool-use strategies. The cold-start SFT stage — using tool-call trajectories synthetically generated by Gemini-2.5-Flash acting as the orchestrator — provides the necessary initialization. This finding contrasts with domains like math reasoning (DeepSeek-R1), where RL can discover effective strategies without SFT, and suggests that multi-turn tool use is a substantially harder behavior for RL to discover from scratch than chain-of-thought reasoning.

Limitation 4: Synthetic Data for Long Videos Is Expensive or Low-Quality

Training an orchestrator model to perform multi-turn reasoning requires question-answer pairs and corresponding tool-call trajectories. For long videos (1+ hours), collecting such data at scale is prohibitive: the paper estimates approximately $30 per video for human annotation on the Prolific platform (Section 1, A1).

Existing synthetic approaches avoid this cost through bottom-up pipelines: split the video into 10-30 second subclips, process each subclip with a short-video understanding model to generate captions, then feed the captions to an LLM to synthesize QnA pairs (as in LongVILA and Eagle). But this approach is slow and resource-intensive — for an hour-long video split into 120 subclips, even at 10 seconds per subclip, processing takes 20 minutes of wall-clock time. The paper's innovation is to leverage the long-context capabilities of Gemini-2.5-Flash to generate high-quality QnA pairs in a single pass over the full video, achieving approximately 100× cost savings compared to human annotation and 10× time savings compared to subclip processing while maintaining quality (fewer than 5% of synthetic samples required manual correction during verification).

How This Paper Positions Itself

The paper positions SAGE at a deliberate intersection of three research trajectories that have previously been pursued independently:

From the Agent literature: SAGE inherits the multi-turn, tool-using architecture but expands the tool set beyond temporal grounding and — crucially — trains the orchestrator to decide when not to use tools. This any-horizon design is the paper's central conceptual contribution: rather than being either a Direct model or an Agent, SAGE is trained to be both, selecting the appropriate mode per query.

From the RL for reasoning literature: SAGE borrows the GRPO optimization framework from DeepSeek-R1 and DeepSeek-Math but adapts the reward design for the open-ended, multi-turn setting through LLM-as-judge accuracy rewards and carefully calibrated step-level rewards. The paper explicitly contrasts with prior video RL work that used string-matching rewards and shows the resulting degradation on open-ended questions.

From the synthetic data literature: SAGE's data generation pipeline draws on the idea of using strong proprietary models as data generators but applies it to the specific challenge of long-video QnA generation, where the key bottleneck is temporal coverage — making sure questions span the full video duration. The percent_video_parsed field (Figure 3) is a simple but effective mechanism for enforcing this coverage that prior work does not employ.

The overarching thesis is that the combination of agentic multi-turn reasoning, any-horizon behavioral training through RL, and cost-effective synthetic data generation is what enables practical long-video reasoning systems — and that each component is individually necessary. The ablation of cold-start SFT (Table 13: RL without SFT collapses to single-turn, achieving only 23.6% on multi-turn) and the tool ablation (Table 9: every tool contributes non-trivially) provide empirical support for this integrated claim.

3. Technical Approach

3.1 Reader Orientation

SAGE is an agent-based video reasoning system built around a fine-tuned vision-language model (called SAGE-MM) that can either answer a question about a video directly in a single turn or iteratively invoke a set of tools—like a web search engine, a speech transcriber, an event localizer, and a visual analyzer—over multiple turns before producing a final answer. The system solves the problem of rigid, one-size-fits-all video understanding by learning to adapt its reasoning horizon to the complexity of each query: short, simple questions get immediate answers, while longer, harder questions involving external knowledge or targeted temporal search trigger a multi-step tool-use loop, much like how a person decides whether to skim or watch a video in full.

3.2 Big-Picture Architecture (Diagram in Words)

The SAGE system has four major components that interact in two sequential stages:

1. Input Encoder (pre-SAGE-MM): Samples 128 frames at 2 FPS from the input video and pools them temporally. Produces the frame representations F that the orchestrator sees.
2. Orchestrator VLM (SAGE-MM): A fine-tuned vision-language model (e.g., Qwen3-VL-8B-Instruct after SFT and RL) that serves as the central decision-maker. It receives the frames F, video metadata M (path and duration), tool definitions T, and the user query Q. In Stage 1 (Context VLM), it outputs a video context summary C and either a final answer or a recommended tool call. In Stage 2 (Iterative Reasoner), it receives the accumulated results of previous tool calls along with C and decides at each step whether to answer directly or invoke another tool.
3. Tool Set (Executors): A collection of six tools (Table 1) that SAGE-MM can call. These include web-search (Google Search via Serper API), parse-website (HTML content extractor), transcribe-speech (Whisper-large-v3 for ASR on a given temporal segment), ground-event (temporal localization using Qwen3-VL-30B-A3B-Instruct), extract-video-parts (frame or subclip extraction), and analyze (visual question-answering on extracted media using Qwen3-VL-30B-A3B-Instruct). The tools are executed by the system environment, not by SAGE-MM itself.
4. RL Training Loop: An outer optimization process that uses Group Relative Policy Optimization (GRPO) to update SAGE-MM. For each training prompt, the model rolls out $N=8$ action trajectories (each a sequence of state-action pairs up to length $N_{max}=6$ initially, then 11). A multi-component reward—combining step-level format, tool-use reasonableness, and argument validity bonuses with a final LLM-as-judge accuracy reward—is computed for each trajectory. The GRPO algorithm then computes advantages relative to the group of 8 rollouts and updates the model policy.

Information flows as follows: a video and question enter the system → 128 frames are sampled and metadata is extracted → SAGE-MM produces a Stage-1 JSON action string containing video context, query intent, and either a final answer or a tool recommendation → if a tool is recommended, the system environment executes it → SAGE-MM enters Stage-2, receiving the tool results and previous context, and iteratively decides to call another tool or output a final answer (up to 10 iterations) → the last final-answer field is extracted and evaluated.

3.3 Roadmap for the Deep Dive

The following breakdown builds understanding by moving from the static system definition to the dynamic training process that instills any-horizon behavior:

• First, the SAGE system workflow and action space (Section 3.1): We walk through the two-stage inference procedure, the JSON action schema SAGE-MM must produce at each step, and the complete set of six tools available to the agent. This establishes what the system does at inference time before we discuss how it is trained to do so.
• Second, the synthetic data generation pipeline (Section 3.2): We examine how the training and evaluation data—question-answer pairs and tool-call trajectories—is created using Gemini-2.5-Flash as a data generator. Understanding data provenance is critical because the quality and coverage of synthetic questions directly determine what behaviors the orchestrator can learn, and the cold-start SFT trajectories provide the initialization without which RL fails.
• Third, the RL post-training recipe (Section 3.3): We dissect the GRPO-based optimization, including the trajectory representation, the step-level reward components (format, reasonable-tool, args-repeat, args-valid), the accuracy reward with LLM-as-judge, the uniform reward assignment across trajectory steps, the staged $N_{max}$ scheduling, and the KL-divergence regularization. This is the core mechanism that converts a statically-trained orchestrator into an any-horizon reasoner.
• Fourth, the training hyperparameters and infrastructure: We catalog the specific learning rates, batch sizes, frame sampling parameters, GPU configurations, and optimizer settings used for both SFT and RL stages, since the reproducibility of the claimed improvements depends on these details.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that an agentic video reasoning system can be made practical by equipping it with diverse tools and training it through a carefully designed RL recipe that simultaneously optimizes for answer correctness, judicious tool use, and the ability to decide per-query whether multi-turn reasoning is necessary.

---

The SAGE Inference Workflow and Action Space

The inference procedure defines the environment within which SAGE-MM operates and the decision space it must learn to navigate. The system accepts four input modalities (Figure 2, top): 128 sampled video frames (F), video metadata (M), tool definitions (T), and a user query (Q). The metadata M contains the file path and total duration of the video—information necessary for SAGE-MM to specify valid temporal arguments when requesting tool executions (e.g., which 10-minute window to transcribe or ground within). The tool definitions T are presented as a structured list of available functions with their names, purposes, argument types, and return types (Table 1).

The orchestrator operates in two distinct stages with different output schemas. The paper refers to these schemas as JSON action strings—the model must output valid JSON with specific required fields, and producing malformed JSON triggers a regeneration loop (up to 4 attempts at temperature 0.7).

Stage 1 (Context VLM): This is a single-step stage that bootstraps the reasoning process. SAGE-MM outputs a JSON object with four required fields:

• video-context (C): A natural language description of the video's setting, characters, location, and general subject matter based on the 128 sampled frames. This context is carried forward into all subsequent reasoning steps—it provides the orchestrator with a persistent semantic summary of what the video is about, even as later steps focus on specific temporal segments.
• query-intent: A classification of what the user is asking for, which informs tool selection strategy.
• recommended-tool: If SAGE-MM determines it cannot answer directly, this field specifies which tool to invoke, along with all necessary arguments. If the query can be answered from the initial frames alone, this field may be omitted or set to null.
• final-answer: Either the predicted answer (if no tool call is needed) or null (if a tool call is being recommended). The any-horizon behavior is concretely realized here: SAGE-MM has the option to short-circuit the entire multi-turn process at the very first step.

Stage 2 (Iterative Reasoner): If Stage 1 produced a tool call rather than a final answer, the system environment executes the requested tool and appends the result to the conversation history. SAGE-MM now enters a multi-step loop (up to 10 iterations) where at each step $j$ it receives as input: the original query Q, the video metadata M, the tool definitions T, the video context C from Stage 1, and the concatenated results of all previous tool calls $A_1, A_2, \ldots, A_{j-1}$. At each step it outputs a JSON action string with three required fields:

• answerable: A boolean indicating whether the orchestrator believes it now has sufficient information to answer the user's question. This field is used internally by the reasoning process but is also observed during training as a diagnostic signal.
• recommended-tool: Information about the next tool to invoke, or null if answering directly.
• final-answer: The predicted answer, or null if recommending another tool call.

The loop terminates either when SAGE-MM sets final-answer to a non-null value or when the maximum step count (10 under Stage 2, for a total of 11 steps when counting Stage 1) is reached. The paper sets the total maximum $N_{max} = 11$ by default, with a reduced $N_{max} = 6$ during the first 100 steps of RL training for stability reasons (Section 3.3, discussed further in the RL section below).

The Tool Set: Table 1 enumerates the six tools available to SAGE-MM, each implemented by an external model or API rather than by SAGE-MM itself. The design philosophy is that the orchestrator decides what to do while the tools execute the specialized perception:

• web-search accepts a text query string and a number of results to return (integer). It interfaces with the Serper-hosted Google Search API and returns a list of URLs, titles, and text snippets. This tool enables SAGE to pull in external knowledge not present in the video frames—for instance, Formula 1 season standings or cultural references that a character makes.
• parse-website accepts a URL and returns the parsed HTML content of that webpage. It functions as a follow-up to web-search, allowing SAGE to drill into specific search results for detailed information.
• transcribe-speech accepts the video file path, a start timestamp, and an end timestamp. It runs the Whisper-large-v3 automatic speech recognition model on the specified temporal segment and returns the segment-level verbal transcript. A critical design choice is that transcription operates on segments rather than the full video—SAGE-MM must predict reasonable start and end times (maximum 10 minutes apart) based on its understanding of where relevant dialogue is likely to occur. This segment-level approach is more efficient than transcribing the entire video and forces the orchestrator to develop intelligent temporal reasoning about where speech content is located.
• ground-event accepts an event description string, the video path, and start/end timestamps defining the search window. It uses Qwen3-VL-30B-A3B-Instruct to temporally localize the described event within the specified window and returns the precise timestamps. Like transcription, grounding operates on segments rather than the full video. The paper qualitatively found (Section 3.1) that existing grounding models "struggle on longer entertainment videos" when asked to search the entire duration, motivating the segment-level constraint. SAGE-MM is responsible for proposing coarse boundaries (up to 10 minutes) within which the grounding model can operate reliably.
• extract-video-parts accepts a type parameter ("frames" or "subclip"), the video path, and start/end timestamps. It extracts either sampled frames or a video subclip from the specified window and returns the file paths to the saved media. This tool is the bridge between temporal reasoning and detailed visual analysis: SAGE-MM first identifies where to look using transcription or grounding, then uses this tool to actually retrieve the visual content for inspection.
• analyze accepts a query string and a list of media file paths (produced by extract-video-parts). It invokes Qwen3-VL-30B-A3B-Instruct to answer the query based on the provided visual media and returns the answer. This tool performs the actual fine-grained visual reasoning on temporally-localized content—SAGE-MM delegates the detailed "looking" to a more capable model while reserving its own computation for high-level orchestration.

A concrete execution example (Figure 2, bottom): For a query about a specific event in a long Formula 1 video, SAGE-MM might first use its Stage-1 context to predict that the event likely occurs in a particular 10-minute window (based on its understanding of the race structure from the sampled frames). It calls ground-event with these coarse boundaries, receives precise timestamps, then calls extract-video-parts to retrieve the relevant frames, and finally calls analyze on those frames to answer the specific visual question. At each step, the accumulated evidence grows, and SAGE-MM eventually decides the query is answerable and produces a final answer.

Comparison to prior agent designs: The key architectural distinction from prior work is twofold. First, the tool set is diverse—web-search, parse-website, and transcribe-speech have no equivalent in VideoAgent, VideoMind, or VideoExplorer, which rely purely on temporal grounding and caption retrieval. This enables external knowledge integration and verbal content access, which prove empirically important (Table 9 shows performance drops of 3-6% when individual tools are removed, and dropping transcribe-speech causes a catastrophic 36.5% drop on verbal questions). Second, SAGE-MM is trained to decide when not to use tools—the any-horizon capability means it can answer directly from the Stage-1 context when the question is simple, as demonstrated by the single-turn trajectory in Figure 17 where SAGE answers "Kraft" immediately for a short video query.

---

Synthetic Data Generation Pipeline

Training SAGE-MM requires two types of data: question-answer (QnA) pairs to define the task distribution, and tool-call trajectories to provide demonstrations of multi-turn reasoning behavior. Both are generated synthetically to avoid the prohibitive cost of human annotation for long videos. The pipeline (Figure 3) consists of two sequential stages, each leveraging Gemini-2.5-Flash in a different role.

Video Source Selection: The authors collect videos from 13 popular YouTube channels spanning diverse entertainment genres: sports (Formula1), food (ZachChoi), comedy (TheDailyShow, MrBean, TheOffice, Friends, fluffyguy, trevornoah), education (Vox, kurzgesagt, veritasium, QuantaScienceChannel), and travel (WalkingAlice). The total dataset comprises 6,668 videos with durations ranging from under 60 seconds to over 2,400 seconds. Table 2 breaks down the video count and resulting QnA/action counts by duration bucket, showing the distribution is intentionally broad: 1,493 videos under 60 seconds, 1,907 in the 60-180 second range, and decreasing counts in the longer buckets, with 576 videos exceeding 2,400 seconds (40 minutes). This diversity is essential for training an any-horizon agent, as the model must learn to handle everything from 30-second comedy clips to hour-long documentaries.

Stage 1: QnA Pair Generation: The authors leverage the long-context modeling capabilities of Gemini-2.5-Flash to process entire videos in a single pass. The key innovation is a carefully designed prompt that includes a percent_video_parsed field requirement (Figure 3, bottom inset). For each generated question, the model must output an estimate of what percentage of the video's duration has been covered by the questions generated so far, ranging from 0% to 100%. This simple mechanism enforces temporal coverage—the model is explicitly prompted to ensure that questions span the full temporal extent of the video rather than clustering around the beginning or visually salient moments.

The prompt instructs Gemini-2.5-Flash to generate 10-20 QnA pairs per video, producing questions of varying types (open-ended and multiple-choice) and difficulty levels. The generated questions span the full temporal range: some reference specific moments at various timestamps, others require integrating information across the entire video, and some ask about general themes or require external knowledge. The output of this stage is 99.1k QnA pairs across all videos (Table 2). Approximately 54% of questions in the final SAGE-Bench are open-ended (942 out of 1,744), with the remainder being multiple-choice (802), reflecting the paper's emphasis on real-world interaction patterns.

Verification and Quality: The authors manually verified over 1,700 generated samples and found that fewer than 5% required edits—a remarkably low error rate for fully synthetic data. At an estimated $30 per video for human annotation, generating 99.1k QnA pairs through human effort would cost approximately$3 million. The Gemini-2.5-Flash pipeline achieves approximately 100× cost savings relative to human annotation and 10× time savings relative to the subclip-processing approach used by prior work (LongVILA, Eagle), where each video is split into 10-30 second segments and processed sequentially.

Stage 2: Tool Call Trajectory Generation: The synthetic QnA pairs define the tasks, but SAGE-MM also needs demonstrations of how to solve them through multi-turn tool use. The authors construct a SAGE system instance where Gemini-2.5-Flash serves as the orchestrator (SAGE-MM), with the standard tool set available. For each question in the training set, they run this expert system four times, producing four distinct tool-call trajectories per question (with potential variation due to nondeterminism in the expert model's decisions). Each trajectory is a sequence of state-action pairs: at each step, the expert Gemini-2.5-Flash observes the current state (query, metadata, tool definitions, context, and previous results) and outputs a JSON action string specifying which tool to call or whether to answer. These trajectories are recorded as input-output pairs.

From these expert trajectories, the authors extract unique state-action pairs to construct a cold-start SFT dataset of 417.7k actions (Table 2). The SFT training objective is straightforward: for each state in the dataset, the model is trained via standard next-token prediction to produce the corresponding action string (the JSON output) that the expert Gemini-2.5-Flash produced. This is behavior cloning—the model learns to imitate the expert's tool-use decisions without any reward optimization.

Design Choice—Why SFT Before RL: The paper's experiments reveal a critical finding about the necessity of this cold-start stage (Table 13). When the authors attempted to apply RL directly to the base Qwen3-VL-8B-Instruct model without SFT, the model collapsed to single-turn behavior—it produced 1,727 single-turn responses and only 17 multi-turn trajectories out of 1,744 evaluation samples, with an overall accuracy of 56.6% (far below the base model's 64.9% and the SFT+RL model's 68.0%). The authors hypothesize this collapse occurs because the base model's pretraining objective strongly biases it toward direct answer generation, and without initial exposure to multi-turn trajectories, RL's exploration space is too vast for the model to discover effective tool-use strategies. The SFT stage provides a warm-start policy that already exhibits some multi-turn behavior (the SFT-only model in Table 8 generates 1,038 multi-turn trajectories vs. 942 for the base model), giving RL a reasonable initialization from which to optimize.

This finding is significant because it contrasts with domains like mathematical reasoning, where DeepSeek-R1 demonstrated that RL alone (without SFT) can discover effective chain-of-thought strategies. The paper argues that multi-turn tool use is a fundamentally harder behavior to discover through pure RL than chain-of-thought reasoning, likely because the action space is more constrained (valid JSON with specific fields) and the reward signal for intermediate tool-use decisions is sparser than for step-by-step reasoning traces.

RL Training Dataset Construction: From the full SFT dataset, the authors filter a subset of 7.68k samples specifically for RL training. The filtering criterion is based on the synthetic tool-call trajectories: half of the selected samples require tool calls (multi-turn problems) and half have single-turn responses. This balanced split is essential for training any-horizon behavior—if all training samples required multi-turn reasoning, the model would learn to always invoke tools regardless of question complexity. By including single-turn examples, the model learns that some questions can and should be answered directly.

---

RL Post-Training Recipe

The reinforcement learning stage is where SAGE-MM transitions from imitating expert trajectories to optimizing its own behavior for the dual objectives of answer correctness and appropriate tool use. The paper uses Group Relative Policy Optimization (GRPO), the same algorithm that underlies DeepSeek-R1 and DeepSeek-Math, with customizations specific to the multi-turn, open-ended video reasoning setting.

Trajectory Representation: During RL, for each input set $S_1 = \{T, F, M, Q\}$ (tool definitions, frames, metadata, query), the model rolls out $N=8$ action trajectories. The $i^{th}$ trajectory $\tau_i$ is a sequence of state-action pairs:

$\tau_i = \bigl[(S_1, A_1), (S_2, A_2), \ldots, (S_N, A_N)\bigr]$

where $A_j$ is the action (JSON output) produced by SAGE-MM at step $j$, and $S_{j+1}$ is the state constructed by appending the action $A_j$ and its tool execution result to the history: $S_{j+1} = \{T, Q, M, C, A_1 \ldots A_j\}$. The key structural property is that each state $S_{j+1}$ contains the full conversation history—all previous tool calls and their results—so SAGE-MM has access to accumulated evidence when making subsequent decisions.

What it represents: Each trajectory is a complete attempt to answer the query, potentially spanning 1 to $N_{max}$ steps. The 8 trajectories for a given input are generated independently (in parallel during training), producing diverse solution paths that may differ in which tools are called, in what order, and what final answer is produced.

Why this representation: By generating multiple trajectories per input, GRPO can compute relative advantages—how much better or worse each trajectory is compared to the average of its group—without needing a separate value function model. This is more memory-efficient than actor-critic methods (which require training a critic network alongside the policy) and has been shown effective for LLM fine-tuning in DeepSeek-R1.

Reward Structure (Equation 2): Each trajectory $\tau_i$ with $N$ steps receives a single scalar reward $R_i$ that is uniformly assigned to every action in the trajectory:

$R_i = (s_1 + s_2 + s_3 + \ldots + s_N) + a_N$

$r(A_1) = r(A_2) = \ldots = r(A_N) = R_i$

where $s_j$ is the step-level reward at step $j$ and $a_N$ is the accuracy reward computed at the final step.

What it computes: A total trajectory score combining intermediate behavioral incentives (did the model produce valid JSON? did it select reasonable tools? did it avoid repeating arguments?) with a final outcome judgment (was the answer correct?). This total is then broadcast to every action in the trajectory—meaning all steps share the same reward, regardless of which specific step contributed most to success or failure.

Why this form: Uniform reward assignment is a deliberate simplification enabled by synchronous rollout generation. Because all 8 trajectories for a given input are completed before advantages are computed, the algorithm can wait to see the final outcome before scoring individual actions. The alternative—assigning rewards only to the final action or using temporal credit assignment—would introduce additional complexity without clear benefit, since the GRPO advantage computation already handles relative comparisons across trajectories. The uniform assignment means that every action in a successful trajectory gets a positive reward signal, which encourages the model to reproduce all the decisions that led to success, not just the final answer prediction.

One subtle consequence: if a trajectory takes unnecessary tool calls but eventually produces the correct answer, those unnecessary steps still receive positive reward. The step-level rewards (particularly reasonable-tool and args-repeat) partially mitigate this by penalizing obviously wasteful actions regardless of final outcome, but they cannot fully eliminate the credit assignment ambiguity. The paper's choice to accept this ambiguity in exchange for algorithmic simplicity reflects a practical engineering tradeoff.

Step-Level Reward Components: The step-level reward $s_j$ at each step $j$ is the sum of four sub-rewards, each carefully calibrated so that the maximum accumulated step-level reward over a full-length trajectory (10 steps in Stage 2, plus 1 in Stage 1) is comparable to the accuracy reward magnitude:

1. Format Reward ($s_{\text{format}}$):

$s_{\text{format}} = \begin{cases} +0.05, & \text{if JSON contains only required fields} \\ -0.10, & \text{otherwise} \end{cases}$

What it computes: A small bonus for producing a valid JSON action string with exactly the required fields and no extraneous content, or a larger penalty for malformed or invalid JSON.

Why this form: The asymmetric penalty (-0.10 penalty vs. +0.05 bonus) reflects the paper's priority: it is more important to avoid invalid outputs that break the tool execution pipeline than to reward perfect formatting. A single malformed JSON at any step can derail the entire trajectory (the system must regenerate with temperature 0.7 up to 4 times, introducing nondeterminism and potential failure). Over a maximum 11-step trajectory, the format reward can contribute at most $11 \times 0.05 = 0.55$, which is substantial enough to matter in the overall $R_i$ computation but not so large as to dominate the accuracy signal.

2. Reasonable-Tool Reward ($s_{\text{reasonable-tool}}$):

$s_{\text{reasonable-tool}} = \begin{cases} +0.10, & \text{if current tool call is reasonable} \\ -0.10, & \text{otherwise} \end{cases}$

What it computes: A GPT-4o judge evaluates whether the tool call at the current step makes sense given the query and the history of previous tool calls. The judge receives the full conversation context and the proposed action, and outputs a binary verdict.

Why this form: This reward addresses a specific failure mode: the model might learn to call tools at random or in nonsensical orders (e.g., parse-website before web-search, or analyze on frames that haven't been extracted). Without this reward, the only signal about tool-use quality would come indirectly through whether the final answer is correct, which is a noisy and delayed signal. By providing immediate feedback on tool selection rationality, the model receives denser training signal. The ±0.10 magnitude is twice that of the format reward, reflecting the greater importance of strategic tool selection over formatting precision. Over 11 steps, this component can contribute $11 \times 0.10 = \pm 1.10$.

3. Argument-Repetition Penalty ($s_{\text{args-repeat}}$):

$s_{\text{args-repeat}} = -0.05 \cdot \sqrt{\text{num-repetitions}}$

What it computes: A penalty for calling the same tool with identical arguments multiple times. The num-repetitions is a count of how many times the exact same (tool, arguments) pair has appeared in the trajectory. The square root ensures diminishing marginal penalties—the first repetition hurts most, and successive identical calls hurt progressively less.

Why this form: Without this penalty, the model could enter an infinite loop of identical tool calls, never making progress toward an answer. The square root prevents the penalty from growing unboundedly while still strongly discouraging repetition early. The coefficient -0.05 is chosen to make the penalty meaningful but not dominant: a tool called with identical arguments 4 times would incur $-0.05 \times \sqrt{4} = -0.10$, comparable to a single unreasonable-tool penalty.

4. Argument-Validity Penalty ($s_{\text{args-valid}}$):

$s_{\text{args-valid}} = \begin{cases} -0.1, & \text{if arguments are invalid} \\ 0, & \text{otherwise} \end{cases}$

What it computes: A penalty when the tool call arguments are syntactically or semantically invalid (e.g., a start timestamp after an end timestamp, a non-existent file path, a missing required argument).

Why this form: This is a hard constraint—invalid arguments cause tool execution failures, which waste steps and prevent the trajectory from reaching a correct answer. The -0.1 penalty is designed to be painful enough to strongly discourage invalid arguments while being symmetric with the reasonable-tool penalty magnitude.

Calibration of Step Rewards: The paper explicitly states that the step reward values were set "such that the accumulated step-level reward for a trajectory with 10 steps would be comparable to the accuracy reward." The maximum possible step-level reward for a 10-step trajectory (assuming all format bonuses, all tools reasonable, no repetitions, no invalid arguments) would be $10 \times (0.05 + 0.10) = 1.50$, while the maximum accuracy reward (discussed below) is +1.25. The minimum possible step-level reward (all format penalties, all tools unreasonable, maximum repetitions and invalid arguments) could be substantially negative. This calibration ensures that a perfect tool-use trajectory without a correct answer scores similarly to a correct answer with no tool use, forcing the model to optimize both behaviors simultaneously rather than fixating on either tool use for its own sake or answer accuracy at the expense of process quality.

Accuracy Reward ($a_N$):

$a_N = \begin{cases} -2.0, & \text{if JSON action string is invalid} \\ -0.5, & \text{if wrong answer and $N \geq 1$} \\ +1.25, & \text{if correct answer and visual tools in $\tau_i$} \\ +1.0, & \text{otherwise} \end{cases}$

What it computes: A final outcome reward based on the predicted answer. The reward is computed by an LLM judge (GPT-4o) that compares the model's final-answer to the ground-truth answer and outputs a binary correct/incorrect verdict. The reward depends on both correctness and whether the trajectory used visual tools—ground-event or extract-video-parts.

Why this form: The four-tier structure encodes specific training objectives:

• -2.0 for invalid JSON final output: A harsh penalty for trajectories that fail to produce a parseable answer at all. This magnitude (twice the maximum possible accuracy reward) reflects that producing no answer is worse than producing a wrong answer—it represents a complete system failure rather than an incorrect prediction.
• -0.5 for wrong answer with tool calls ($N \geq 1$): A moderate penalty for engaging in multi-turn reasoning but arriving at the wrong conclusion. Importantly, if the model answers wrongly in a single turn ($N=0$, meaning the model answered directly in Stage 1), this penalty does not apply. This asymmetry encourages the model to attempt direct answers when uncertain rather than wasting computation on fruitless tool calls.
• +1.25 for correct answer with visual tools: A bonus for trajectories that successfully used ground-event or extract-video-parts and arrived at the correct answer. The higher reward relative to +1.0 reflects the greater difficulty and practical importance of getting visual tool calls right—the model must not only answer correctly but also demonstrate the ability to navigate the video content through targeted visual inspection.
• +1.0 for correct answer without visual tools: The baseline positive reward for correct answers achieved either directly or through non-visual tools (web search, speech transcription). This is lower than the visual-tool bonus to incentivize the model to engage with the video's visual content when relevant, rather than relying solely on transcripts or external search.

The choice of LLM-as-judge (GPT-4o) for accuracy evaluation, rather than string matching or ROUGE metrics used by Video-R1 and VideoRFT, is a critical design decision. String matching works for MCQ (where the answer is "A", "B", "C", or "D") but fails for open-ended responses where multiple phrasings can be correct (e.g., "the car was red" vs. "it was a red vehicle"). The paper maintains uniformity by using the same LLM judge during both training and evaluation, ensuring the optimization signal aligns with the deployment metric.

GRPO Optimization: With rewards computed for all 8 trajectories in a group, GRPO computes advantages and updates the policy. The GRPO algorithm (as introduced in DeepSeek-Math and used in DeepSeek-R1) works as follows for a group of $G$ trajectories: for each trajectory $\tau_i$, the advantage $A_i$ is computed as the normalized reward relative to the group mean and standard deviation:

$A_i = \frac{R_i - \text{mean}(\{R_1, \ldots, R_G\})}{\text{std}(\{R_1, \ldots, R_G\})}$

The policy is then updated to increase the probability of actions in trajectories with positive advantage and decrease the probability of actions in trajectories with negative advantage, subject to a KL-divergence constraint that prevents the policy from diverging too far from the reference (SFT) model. The KL-divergence coefficient is set to $0.005$.

What this computes: For each group of 8 rollouts, GRPO identifies which trajectories were better than average (positive advantage) and which were worse (negative advantage). The policy gradient pushes the model toward the better-performing behaviors and away from the worse-performing ones.

Why this form: GRPO's group-relative advantage computation eliminates the need for a separately trained value function (critic), which would require additional memory and training complexity. The tradeoff is that advantages are only relative within each group—if all 8 trajectories in a group happen to be poor, the best one still gets a positive advantage and is reinforced. In practice, the diversity induced by sampling 8 trajectories per prompt usually produces enough variation for meaningful relative comparisons. The KL-divergence constraint (coefficient = 0.005) serves as a regularizer, preventing the RL update from moving the policy into regions where it produces nonsensical outputs—a well-known failure mode in RL fine-tuning of language models.

Training Stability Measures: The paper implements two specific stability measures during RL training:

• Staged $N_{max}$ scheduling: For the first 100 RL steps, the maximum trajectory length is set to $N_{max} = 6$ rather than the default 11. The authors state this is "necessary for stable training, aligned with findings from a concurrent work for training long-horizon LLM agents" (Xi et al., 2025). The intuition is that early in training, the model's tool-use policy is unreliable, and allowing long trajectories risks the model wandering through many ineffective tool calls before eventually failing, which generates low-quality training signal. By restricting the horizon initially, the model is forced to learn efficient tool-use strategies that produce answers quickly. Once the policy has stabilized (after 100 steps), the horizon is expanded to 11 steps, allowing the model to tackle problems requiring more extensive reasoning. At evaluation time, $N_{max} = 11$ is used throughout.

• Synchronous rollout generation: The paper notes that "rollout generation is synchronous, i.e., advantages are computed only after all trajectories are completed in a batch" (Section 3.3). This means that for a given training prompt, all 8 trajectories are fully generated before any reward computation or policy update occurs. This is a standard design in GRPO implementations and ensures that the group statistics (mean and standard deviation of rewards) are computed over complete, comparable trajectories.

---

Training Infrastructure and Hyperparameters

The paper reports two distinct training phases with different configurations.

Supervised Fine-Tuning (SFT) Configuration:

• Base models: Qwen3-VL-8B-Instruct, Qwen3-VL-4B-Instruct, Qwen2.5-VL-7B-Instruct, and Molmo2-8B (all fine-tuned separately to produce SAGE-MM variants)
• Training duration: 1 epoch over the 417.7k action dataset
• Batch size: 64
• Learning rate: $1 \times 10^{-5}$ with linear decay scheduler
• Frame sampling: 128 frames at 2 FPS
• Temporal pooling factor: 2
• Maximum tokens per frame: 192
• Minimum tokens per frame: 128
• Frozen modules: Visual encoder and projector (only the language model backbone is updated)
• Hardware: 16× NVIDIA H100 GPUs

Why freeze the visual encoder and projector: By keeping the visual backbone frozen, the SFT stage teaches the language model to interpret visual features and produce appropriate JSON action strings without modifying the feature extraction pipeline. This is both computationally cheaper (fewer trainable parameters) and prevents catastrophic forgetting of visual understanding capabilities acquired during pretraining. The tradeoff is that the model cannot adapt its visual representations to the specific demands of tool-use reasoning—it must work with whatever features the frozen encoder produces.

Reinforcement Learning (RL) Configuration:

• Training dataset: 7.68k samples (filtered subset from SFT data, balanced between single-turn and multi-turn)
• Batch size: 16 prompts per update
• Rollouts per prompt: $N=8$ trajectories
• Initial learning rate: $1 \times 10^{-6}$ with cosine decay scheduler
• KL-divergence coefficient: $0.005$
• $N_{max}$: 6 for the first 100 steps, then 11 for the remaining 380 steps (total 480 steps reported)
• The paper reports results for the model trained for 480 steps total
• Hardware: 16× NVIDIA H100 GPUs

Why a lower learning rate for RL: The RL learning rate ($1 \times 10^{-6}$) is an order of magnitude lower than the SFT learning rate ($1 \times 10^{-5}$). This is standard practice in RL fine-tuning of language models: the SFT model already produces reasonable outputs, and the RL stage should make targeted improvements without destroying the base capabilities. The lower learning rate, combined with the KL-divergence constraint, provides a conservative update regime that preserves the SFT model's knowledge while optimizing the reward-sensitive aspects of behavior.

Evaluation Configuration:

• Sampling temperature: 0.0 for deterministic evaluation (with regeneration at temperature 0.7 for up to 4 attempts if JSON is malformed)
• Frames: 128 sampled frames for both Direct baselines and SAGE-MM Stage-1 input
• Transcript: Provided as extra context to Direct baselines for fair comparison (since SAGE can access transcripts through its transcribe-speech tool)
• Judge: GPT-4o LLM-as-judge for both open-ended and MCQ problems
• Serving: All models served using vLLM during evaluation

Why temperature 0.0 with regeneration at 0.7: Deterministic evaluation (temperature 0.0) ensures reproducibility and removes sampling variance as a confound when comparing models. However, deterministic models can still produce malformed JSON due to model error rather than sampling stochasticity. The regeneration loop at temperature 0.7 introduces controlled randomness on retry—if the first attempt fails structurally (invalid JSON), the model gets additional attempts with some diversity, increasing the chance of producing a valid output. The paper acknowledges this "may lead to non-deterministic behavior during inference," which is a minor caveat affecting the strict reproducibility of reported numbers.

Why the RL dataset is filtered to 7.68k balanced samples: The full SFT dataset of 99.1k QnA pairs and 417.7k actions is too large for efficient RL training (each RL step requires 8× rollout generation per prompt). The 7.68k subset represents approximately 7.7% of the full data. The deliberate balance between single-turn and multi-turn samples ensures the RL optimization sees both behaviors during training, preventing the policy from collapsing to either extreme. Without single-turn examples, the model would never receive positive reward for answering directly; without multi-turn examples, it would never learn to use tools effectively.

4. Key Insights and Innovations

Innovation 1: "Any-Horizon" Reasoning as a Learned Behavioral Meta-Strategy, Not an Architectural Choice

The paper's most fundamental conceptual contribution is the framing of any-horizon reasoning as a learned behavioral capability rather than a fixed architectural property. Prior to SAGE, the video reasoning landscape was implicitly partitioned: models were either Direct (single-turn) or Agent (multi-turn), and their operating mode was baked into the architecture at design time. Direct models like Qwen3-VL and Video-R1 always process frames and output answers in one shot; Agent systems like VideoMind and VideoExplorer always engage in multi-turn tool use, regardless of whether the query actually requires it. The choice was structural — you built one kind of system or the other.

SAGE reframes this as a policy decision to be optimized. The orchestrator SAGE-MM is not architecturally prevented from answering in a single turn, nor is it hardcoded to always invoke tools. Instead, through the RL training recipe's reward structure and balanced single-turn/multi-turn data mixture, the model learns to treat the decision to go multi-turn as a strategic choice — one that should be made per-query based on whether the expected information gain from tool use justifies the additional computation. This is visible in the action distribution shift from SFT to RL in Table 8: the SFT model (trained purely by imitating Gemini-2.5-Flash trajectories) overcalls tools, generating 1,038 multi-turn trajectories out of 1,744; after RL, the count drops to 796 multi-turn with a corresponding increase in single-turn responses (706 → 948), while accuracy improves in both modes. The RL process doesn't just improve tool-use quality — it actively prunes unnecessary tool calls, learning when not to use tools as much as how to use them.

This represents a fundamental shift in perspective, not an incremental improvement. The dominant assumption in the agent literature (VideoAgent, VideoExplorer, LVAgent) has been that multi-turn reasoning is always the right approach for complex video understanding — the architectural overhead of tool orchestration is justified by the flexibility it provides. SAGE's insight is that this overhead is sometimes wasted, and that a properly trained system can amortize the cost of tool-use decisions across queries by making them contingent on difficulty. The parallel to the "compute-optimal" framing in the PaLM 2-S* test-time compute paper is direct: just as that work showed that aggressive search is counterproductive on easy problems and necessary on hard ones, SAGE shows that multi-turn tool use is counterproductive on simple queries and essential on complex ones. The any-horizon concept is essentially a difficulty-conditioned reasoning mode selector operating at the level of agent behavior rather than inference FLOPs.

The significance extends beyond video reasoning. Any system that can operate in multiple modes (fast/direct vs. slow/deliberative) faces the challenge of mode selection. SAGE demonstrates that this selection can be learned via RL with appropriately shaped rewards rather than engineered through heuristics, and that the cold-start SFT stage is essential for providing the behavioral diversity from which RL can optimize — without it, the model collapses to its pretraining bias toward the fast mode (Table 13: RL without SFT produces only 17 multi-turn trajectories, accuracy 56.6%).

Innovation 2: The Cold-Start Problem as an Inherent Barrier to RL for Multi-Turn Agent Behavior

The paper's most diagnostically important finding is a negative result with general implications: reinforcement learning applied directly to a base vision-language model fails to induce multi-turn tool-use behavior, even when the model is architecturally capable of producing it. Table 13 shows that when GRPO is applied to Qwen3-VL-8B-Instruct without the cold-start SFT stage, the model collapses almost completely to single-turn reasoning — out of 1,744 evaluation samples, it produces only 17 multi-turn trajectories, with those few achieving an abysmal 23.6% accuracy. Overall accuracy drops from 64.9% (base model, Direct mode) to 56.6%. The model has essentially unlearned its ability to engage with the SAGE system's tool infrastructure.

This finding matters beyond the specific SAGE implementation because it identifies a fundamental asymmetry between chain-of-thought reasoning and tool-use reasoning in the context of RL discovery. DeepSeek-R1 demonstrated that LLMs can discover sophisticated chain-of-thought strategies through pure RL without supervised demonstrations — the model spontaneously learns to allocate more thinking tokens to harder problems. This finding energized the "RL-only" paradigm for reasoning. SAGE's result demonstrates that this paradigm does NOT straightforwardly transfer to agentic tool use. The authors hypothesize (Section 6, "Importance of SFT") that the base model's pretraining objective — which strongly biases it toward direct answer generation in a single turn — creates a local optimum that RL cannot escape. The exploration required to discover effective multi-turn strategies (valid JSON action strings with correct field values, appropriate tool selection, temporal argument prediction) is too vast relative to the sparse reward signal available during early RL training. The model never stumbles upon a successful multi-turn trajectory, so it never receives positive reinforcement for tool use, and the policy collapses to the only behavior that sometimes works: direct answering.

This diagnostic insight has direct practical implications for anyone building RL-trained agents. It suggests that:

1. Behavior cloning (SFT) on expert trajectories is not merely a helpful initialization — it is necessary to provide a policy that occasionally succeeds at multi-turn behavior, creating the positive reward signal that RL can then amplify.
2. The "RL-only" successes in reasoning (DeepSeek-R1, etc.) may be specific to domains where the base model's pretraining distribution already contains examples of extended reasoning traces, making discovery through exploration feasible. Tool use, by contrast, requires producing structured outputs in a format (JSON with specific schemas) that is almost entirely absent from pretraining data.
3. The staged $N_{max}$ scheduling (6 for first 100 RL steps, then 11) can be understood as addressing a related problem: early in training, long trajectories are likely to fail, and restricting the horizon forces the model to learn efficient strategies that produce outcomes quickly enough to generate usable reward signal.

This is arguably a more broadly significant contribution than any single accuracy number, because it identifies a boundary condition for a widely-adopted technique (RL fine-tuning for agent behavior) that prior work had not characterized. The field's enthusiasm for RL post-training, fueled by DeepSeek-R1's success, implicitly assumed general applicability to any sequential decision-making task. SAGE's negative result on RL-without-SFT provides an important corrective: when the base policy has near-zero probability of producing successful tool-use trajectories, pure RL cannot bootstrap itself.

Innovation 3: Multi-Reward RL Design That Balances Process and Outcome for Open-Ended Agent Tasks

The paper's third distinctive contribution is the specific reward engineering framework that makes RL training viable for open-ended, multi-turn video reasoning — a problem where prior work had retreated to MCQ-only training with string-matching rewards. The reward structure (Equation 2, Section 3.3) is not a collection of independently obvious components but rather a carefully calibrated incentive system where the relative magnitudes of different reward terms encode a theory of what good agent behavior looks like.

The key design principles embedded in the reward structure are:

First, step-level rewards are calibrated to be comparable to the accuracy reward over a full trajectory. The maximum step-level reward for a perfect 10-step trajectory (all format bonuses, all tools reasonable, no repetitions) is $10 \times (0.05 + 0.10) = 1.50$, which is in the same range as the accuracy reward values (-0.5 to +1.25). This prevents either component from dominating: the model cannot succeed by producing perfect tool calls that lead to wrong answers (step rewards alone can't compensate for -0.5 accuracy penalty), nor can it ignore process quality and succeed purely on answer correctness (a trajectory with terrible tool use would accumulate negative step rewards that offset even a +1.25 accuracy bonus). The calibration reflects a philosophy of balanced optimization — the system should care about both how it reaches answers and whether those answers are correct.

Second, the accuracy reward asymmetry between tool-using and non-tool-using trajectories (+1.25 vs. +1.0) encodes a deliberate preference. By giving higher reward for correct answers that used visual tools (ground-event or extract-video-parts), the RL objective incentivizes the model to engage with the video's visual content when it's relevant, rather than resorting to shortcuts (answering from transcripts alone, relying entirely on web search, or guessing). This addresses a subtle failure mode: a model could achieve reasonable accuracy by ignoring the video entirely and relying on external knowledge from web search, but such a model would fail on questions that genuinely require visual inspection. The +0.25 bonus is small enough that it doesn't incentivize spurious tool calls (the accumulated step-level penalties for calling unnecessary tools would outweigh it), but large enough to tilt the optimization toward visual engagement when it's genuinely helpful.

Third, the uniformly-assigned reward with the -0.5 penalty for wrong answers with tool calls ($N \geq 1$) creates an implicit cost for multi-turn reasoning that encourages any-horizon behavior. If the model engages in tool use and gets the wrong answer, it receives -0.5 rather than a possible 0 (for single-turn wrong answer, which avoids the penalty) or +1.0/+1.25 (for correct answers). This means the model only benefits from multi-turn reasoning when it increases the probability of being correct by enough to justify the risk. For questions where the model is already confident in a direct answer, the expected value of going multi-turn is negative. The model thus learns to reserve tool use for questions where the marginal information gain exceeds the penalty risk — precisely the any-horizon behavior the paper targets.

This reward design contrasts sharply with prior video RL work (Video-R1, VideoRFT) which used simple option-matching and ROUGE-based rewards restricted to MCQ problems. Those reward structures answer the question "did the model pick the right letter?" — a closed-form verification problem. SAGE's reward structure answers the question "did the model follow a reasonable process and arrive at a semantically correct answer?" — an open-ended evaluation problem that requires LLM-as-judge for the accuracy component and explicit process rewards for the behavioral components. The fact that this reward design produces consistent improvements across four different base models (Qwen2.5-VL-7B, Qwen3-VL-4B, Qwen3-VL-8B, Molmo2-8B; Table 4) suggests it captures something general about agent optimization rather than being tuned to a specific model's quirks.

Innovation 4: Reframing Long-Video Data Generation as a Single-Pass Long-Context Problem

The paper's synthetic data pipeline represents a methodological reframing of how training data for long-video understanding is produced, with implications for the economics of video AI research. The dominant approach in prior work (LongVILA, Eagle) was bottom-up: decompose the video into short clips, process each clip independently with a short-video understanding model, aggregate the resulting captions, and then synthesize questions. This pipeline is conceptually straightforward — it leverages strong short-video models that already work well — but it treats the long-video data generation problem as inherently requiring temporal decomposition.

SAGE's reframing is to treat it instead as a long-context modeling problem: with a model that can attend to an entire video in a single forward pass (Gemini-2.5-Flash), QnA generation can happen holistically. This is not merely an engineering optimization (faster, cheaper) — it changes the kind of questions that can be generated. A bottom-up pipeline that processes subclips independently may miss cross-temporal connections: questions that require integrating information from minutes 5, 30, and 55 of a video are difficult to generate when those minutes are processed as isolated segments. The single-pass approach, by giving the generator access to the full temporal context, can produce questions that span arbitrary temporal distances. The percent_video_parsed mechanism (Figure 3) is the enabler: by requiring the model to track its own coverage, the prompt converts temporal coverage from an implicit hope into an explicit constraint.

The economic implications are significant. At $30 per video for human annotation and 20 minutes per video for subclip processing, generating 99.1k QnA pairs at scale would be either financially prohibitive (human annotation) or logistically cumbersome (subclip processing). The ~100× cost reduction and ~10× time reduction reported by the authors are not just convenience metrics — they are what make the SAGE-Bench curation and SAGE-MM training feasible at the reported scale. If each video required 20 minutes of processing, generating data for 6,668 videos would take ~2,200 GPU-hours of processing before any model training begins. The single-pass approach reduces this to ~220 GPU-hours.

More importantly, the low error rate of the synthetic data (<5% requiring edits during manual verification of 1,700+ samples) challenges the assumption that synthetic data for complex tasks necessarily requires expensive quality control. The combination of a strong generator model (Gemini-2.5-Flash) with a carefully engineered prompt that enforces coverage and diversity produces data that is usable with minimal human intervention. This finding, if it generalizes to other video domains beyond entertainment, could substantially lower the barrier to entry for building video reasoning systems on custom content collections.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All primary experiments use SAGE-Bench, a curated evaluation set of 1,744 manually verified samples drawn from popular YouTube entertainment videos spanning sports, food, comedy, education, and travel channels (Table 3). The average video duration is 727 seconds (roughly 12 minutes), with a wide spread across seven duration buckets ranging from under 60 seconds to over 2,400 seconds. The benchmark contains 802 multiple-choice questions (46%) and 942 open-ended questions (54%), of which 1,216 require visual information, 134 require verbal information only, and 394 require both visual and verbal reasoning. All samples were manually verified for correctness, with fewer than 5% requiring edits during the verification process. A held-out subset of 1,473 samples from the training distribution is used for cross-benchmark evaluation on MINERVA (Table 5). The videos used in SAGE-Bench are sampled from the same source channels as the training videos but use a strictly disjoint set of QnA pairs; some video overlap exists between training and evaluation.

• Base model(s). The primary orchestrator SAGE-MM is built on Qwen3-VL-8B-Instruct for all ablations and main comparisons. To demonstrate cross-architecture generalizability, the paper also fine-tunes Qwen2.5-VL-7B-Instruct, Qwen3-VL-4B-Instruct, and Molmo2-8B as SAGE-MM variants (Table 4). The tool backend uses Qwen3-VL-30B-A3B-Instruct for temporal grounding (ground-event) and media analysis (analyze), and Whisper-large-v3 for speech transcription. The choice of Qwen3-VL family as primary is motivated by its strong native function-calling capabilities; the inclusion of Molmo2 demonstrates that the training recipe transfers to a different architecture family.

• Metrics. The primary metric throughout is accuracy — the fraction of evaluation samples for which the model's final answer is judged correct by an LLM-as-judge (GPT-4o). The judge receives the question, the model's predicted answer, and the ground-truth answer, and outputs a binary correct/incorrect verdict after semantic comparison (prompt provided in Figure 7). The same judge is used for both open-ended and multiple-choice questions, maintaining uniformity between training and evaluation. Accuracy is reported both overall and broken down by question type (MCQ vs. open-ended), answer modality (verbal-only, visual-only, both), and video duration bucket (Table 7). A secondary metric reported in ablation Table 15 is the no-answer rate — the percentage of samples where the model fails to produce a valid final-answer field within the maximum step budget.

• Baselines. The paper compares against an extensive set of 18 baseline configurations spanning three categories (Table 4). Direct baselines without RL: Gemini-2.5-Flash, GPT-4o, Qwen3-VL-8B-Instruct, Qwen3-VL-4B-Instruct, Qwen2.5-VL-7B-Instruct, Molmo2-8B, and Qwen3-VL-30B-A3B-Instruct — all evaluated by passing 128 sampled frames and the full video transcript as input, producing a single-turn answer without tool use. Direct baselines with RL: Video-Thinker-7B (Wang et al., 2025), LongVILA-R1-7B (Chen et al., 2025), VideoRFT-7B (VideoRFT), and Video-R1-7B (video-r1) — all trained with GRPO using option-matching or ROUGE-based rewards and evaluated with 128 frames plus transcript. Agent baselines: VideoAgent (Fan et al., 2025) using GPT-4o as orchestrator, LVAgent (Chen et al., 2025) using InternVL-8/72B + LLaVA-Video-72B, LongVT-7B-RFT (Yang et al., 2025), VideoMind-7B-Planner (Liu et al., 2025), VideoExplorer-7B-Planner (VideoExplorer), and VideoChat-R1.5-7B-M (Yan et al., 2025) — all evaluated following their recommended setups with their native tool sets and orchestrator configurations. Two oracle agent baselines are created by plugging Gemini-2.5-Flash and GPT-4o directly into the SAGE system as the orchestrator (SAGE-Flash with Gemini-2.5-Flash or GPT-4o as SAGE-MM), which tests the system design independent of orchestrator training quality.

• Generation budget / compute accounting. The paper does not use an explicit compute budget constraint in the style of FLOPs or token counts. Instead, evaluation efficiency is measured through inference runtime per sample (Table 10), reported in seconds. The maximum reasoning horizon is capped at $N_{max} = 11$ total steps (1 Stage-1 step + up to 10 Stage-2 iterations). For the Direct baselines, frame count is the primary compute lever, and the paper sweeps 16, 32, 64, 128, 256, 512, 1024, and 1536 frames to produce the accuracy-vs-runtime curve. For SAGE, the frame budget is fixed at 128 for Stage-1 context understanding, with additional frames accessed on-demand through extract-video-parts and analyze tool calls (whose cost is included in the runtime measurement). The runtime comparison in Table 10 thus reflects wall-clock end-to-end cost including any tool API calls and model invocations, measured under the same hardware conditions (vLLM serving on NVIDIA H100 GPUs).

• Cross-validation / statistical protocol. No cross-validation is reported for the main benchmark results. The paper evaluates at temperature 0.0 for determinism, with a regeneration protocol for malformed JSON: up to 4 retries at temperature 0.7. To assess variance, the paper reports five runs of the base Qwen3-VL-8B-Instruct model at temperature 1.0 on SAGE-Bench, finding a standard deviation of 0.22 (Table 16). For training data, the synthetic QnA pairs are generated from a set of videos that overlaps with the evaluation set videos, but the QnA pairs themselves are strictly disjoint — the paper manually verified this separation. The MINERVA evaluation (Table 5) uses a held-out subset of 1,473 training samples, not the SAGE-Bench test set, so it serves as a cross-dataset generalization check rather than a held-out evaluation within the primary benchmark.

---

Main Quantitative Results

Overall Performance on SAGE-Bench (Table 4)

The headline result from Table 4 is that SAGE with a trained Qwen3-VL-8B-Instruct orchestrator after SFT+RL achieves 68.0% overall accuracy, which represents a +3.1 percentage point improvement over the base Qwen3-VL-8B-Instruct model in Direct mode (64.9%). The gain is concentrated in specific evaluation dimensions: open-ended questions improve by only +1.6% (54.0% to 55.6%), while verbal-only questions improve dramatically by +14.1% (68.7% to 82.8%). Visual-only questions show a more modest +2.1% gain (61.9% to 64.0%), and questions requiring both modalities improve by +2.6% (72.8% to 75.4%).

The improvement distribution across base model families is notably uneven. Qwen3-VL-4B-Instruct shows the largest absolute gain: from 62.7% (base Direct) to 68.4% (SAGE SFT+RL), a +5.7 percentage point improvement. Molmo2-8B gains +4.3% (61.8% to 66.1%). Qwen2.5-VL-7B-Instruct gains +4.8% (58.6% to 63.4%). The pattern suggests that the SAGE system architecture and training recipe provide consistent benefits across different base model families and scales, with the magnitude of improvement inversely correlated with the base model's starting capability — weaker base models benefit more from the agentic reasoning infrastructure.

SAGE-Flash, which substitutes Gemini-2.5-Flash as the backend for ground-event and analyze tools (while keeping the same trained SAGE-MM orchestrator), pushes overall accuracy to 71.8% — a further +3.8% over standard SAGE and +6.9% over the base model. This configuration even outperforms the variant where Gemini-2.5-Flash itself serves as the SAGE-MM orchestrator (71.3%), which is a striking result: a fine-tuned 8B parameter orchestrator with access to Gemini-2.5-Flash as a tool outperforms Gemini-2.5-Flash orchestrating itself. This suggests that the tool execution quality matters independently of orchestration quality, and that the SFT+RL training produces orchestration policies that are more effective at utilizing strong tools than the off-the-shelf reasoning of the tool model itself.

Comparison to closed-source Direct baselines: The base GPT-4o achieves 71.6% overall, and Gemini-2.5-Flash achieves 68.1% — both evaluated in Direct mode with 128 frames and transcript as context. SAGE-Flash (71.8%) marginally exceeds both, while standard SAGE (68.0%) is competitive with Gemini-2.5-Flash and slightly below GPT-4o. The key comparison is not the absolute number but the fact that an 8B open-weight model with the SAGE system design can approximately match or exceed closed-source frontier models at a fraction of the model scale.

Comparison to RL-tuned Direct models: All four RL-tuned Direct baselines trained on MCQ-oriented rewards (Video-R1: 57.6%, VideoRFT: 55.3%, LongVILA-R1: 52.6%, Video-Thinker: 41.3%) substantially underperform their untuned base models on SAGE-Bench. Video-R1-7B, which is fine-tuned from Qwen2.5-VL-7B-Instruct, achieves 57.6% vs. the base model's 58.6% — the RL training with option-matching rewards has actually degraded performance on this benchmark. The degradation is most severe on open-ended questions: Video-R1 achieves only 43.9% on open-ended vs. 73.6% on MCQ, a gap of 29.7 percentage points, compared to the Qwen2.5-VL-7B base gap of 28.8 points (45.4% open-ended vs. 74.2% MCQ). The RL training has widened an already-large gap rather than closing it. This is a critical negative result that validates the paper's central claim about the inappropriateness of string-matching rewards for open-ended video reasoning.

Comparison to Agent baselines: All existing agent systems underperform SAGE substantially. VideoChat-R1.5-7B-M achieves the highest among them at 54.8% — 13.2 points below SAGE. VideoMind achieves 50.0%, VideoExplorer 50.1%, LVAgent 49.7%, LongVT 46.7%, and VideoAgent 42.0%. The performance gap is not uniform across question types: on MCQ questions, VideoMind (69.7%) and VideoExplorer (69.6%) are competitive with the base Qwen3-VL-8B-Instruct (77.7%) and substantially above the RL-tuned Direct models, but on open-ended questions they collapse — VideoMind achieves 33.2%, VideoExplorer 35.1%, both far below SAGE's 55.6%. This pattern reveals that existing agent systems have been optimized for the MCQ evaluation paradigm and their multi-turn reasoning strategies do not transfer to open-ended problems. The paper's diagnosis is that the over-reliance on temporal grounding as the sole reasoning mechanism, without web search or speech transcription, leaves these systems unable to gather the information needed for open-ended queries that require external knowledge or verbal content.

SFT-only vs. SFT+RL: The SFT-only training (behavior cloning on Gemini-2.5-Flash trajectories without RL) produces mixed results that vary by base model. For Qwen3-VL-8B-Instruct, SFT (63.9%) actually performs slightly below the base model (64.9%), while for Qwen2.5-VL-7B-Instruct, SFT improves from 58.6% to 61.1%, and for Qwen3-VL-4B-Instruct, SFT improves from 62.7% to 64.6%. The RL stage consistently adds further gains: +4.1% for Qwen3-VL-8B, +2.3% for Qwen2.5-VL-7B, +3.8% for Qwen3-VL-4B, and +2.8% for Molmo2-8B. The key insight is that SFT alone is insufficient for the strongest base models — the Qwen3-VL-8B's native Direct reasoning is already strong enough that naive behavior cloning actually degrades performance, likely because the cloned Gemini-2.5-Flash trajectories reflect a different reasoning style that the 8B model cannot perfectly execute. Only after RL optimization, which adapts the cloned behaviors to the model's own capabilities, does the agent paradigm overtake the Direct baseline.

Duration-Wise Performance Analysis (Table 7)

The paper's central claim about any-horizon reasoning predicts that SAGE's advantages should grow with video duration, since long videos are where adaptive tool use (segment-level grounding, targeted transcription) provides the most benefit over uniform frame processing. Table 7 confirms this prediction with a striking pattern: SAGE's improvements are concentrated almost entirely in videos longer than 600 seconds.

For the base Qwen3-VL-8B-Instruct, performance degrades steadily with duration: 73.9% accuracy on videos under 60 seconds, declining to 47.5% on videos exceeding 2,400 seconds. The SFT-only model shows a similar degradation curve (74.3% down to 48.1%), with slight improvements in some short-video buckets and slight degradations in others. The SFT+RL model, however, breaks this pattern: while short-video performance is roughly comparable to the base model (78.5% vs. 73.9% in the 0-60 second bucket, but 70.3% vs. 72.3% in 60-180 and 77.4% vs. 81.9% in 180-300), long-video performance shows substantial gains. In the 600-1200 second bucket, SAGE achieves 63.2% vs. the base model's 55.0% — an +8.2 percentage point improvement. In the 1200-2400 bucket: 61.9% vs. 59.2% (+2.7%). In the 2400+ bucket: 53.8% vs. 47.5% (+6.3%).

The SAGE-Flash configuration amplifies these long-video gains dramatically: +14.6% in the 600-1200 bucket (69.6% vs. 55.0%), +8.8% in 1200-2400 (68.0% vs. 59.2%), and +8.7% in 2400+ (56.2% vs. 47.5%). The fact that SAGE-Flash produces the largest relative improvements in exactly the buckets where the base model degrades most severely — videos longer than 10 minutes — directly supports the thesis that agentic, tool-assisted reasoning is most valuable when uniform frame processing becomes impractical due to temporal scale.

The short-video buckets show a more nuanced picture. In the 60-180 second bucket, SAGE SFT+RL actually underperforms the base model (70.3% vs. 72.3%, -2.0%). In the 180-300 bucket, the gap is even larger (77.4% vs. 81.9%, -4.5%). The SFT-only model shows similar short-video degradations (68.1% vs. 72.3%, 75.0% vs. 81.9%). This suggests that the multi-turn agent overhead — the JSON formatting, the tool invocation loop, the potential for malformed outputs requiring regeneration — imposes a fixed cost that hurts performance on problems the base model could already solve directly. The any-horizon training partially mitigates this (SFT+RL does better than SFT on short videos, indicating RL learns to answer directly more often), but does not fully eliminate it. The net benefit of the agent paradigm is positive only for videos long enough that the gains from targeted tool use outweigh the overhead costs of the agent infrastructure.

Any-Horizon Behavior Analysis (Table 8)

Table 8 provides direct evidence for the paper's core behavioral claim: that SAGE learns to adaptively switch between single-turn and multi-turn reasoning based on query difficulty. The table categorizes all evaluation samples by whether the model chose to answer in a single turn (in Stage 1) or engaged in multi-turn reasoning (Stage 2), and reports accuracy separately for each mode.

The expert Gemini-2.5-Flash system (SAGE-Flash with Gemini as orchestrator) produces a natural reference distribution: 859 single-turn responses (49.3% of samples) at 76.9% accuracy, and 885 multi-turn responses (50.7%) at 66.0% accuracy. The multi-turn accuracy being lower than single-turn is expected — these are the harder questions that the model correctly identifies as requiring tool use.

The SFT-only model (Qwen3-VL-8B-Instruct) shows a clear tool-overcalling pattern: 706 single-turn responses (40.5%) at 79.0% accuracy, and 1,038 multi-turn responses (59.5%) at only 53.7% accuracy. Compared to the expert, the SFT model engages in multi-turn reasoning on 8.8% more samples, but those additional multi-turn trajectories achieve substantially lower accuracy — the model is invoking tools on questions that don't need them, and its tool-use quality on those unnecessary calls is poor enough to drag down the multi-turn average.

The SFT+RL model corrects this imbalance: 948 single-turn responses (54.4%) at 79.6% accuracy, and 796 multi-turn responses (45.6%) at 54.3% accuracy. The proportion of multi-turn responses has shifted from 59.5% to 45.6%, moving closer to the expert's 50.7% distribution. Single-turn accuracy remains high and essentially unchanged (79.6% vs. 79.0%), while multi-turn accuracy improves from 53.7% to 54.3%. The RL process has pruned unnecessary tool calls without degrading tool-use quality — the model is making fewer but better multi-turn decisions. The overall accuracy gain from SFT (63.9%) to SFT+RL (68.0%) can be decomposed into two effects: the accuracy improvement within the multi-turn mode (+0.6%), and the shift in distribution toward the higher-accuracy single-turn mode (roughly +3.5% from the mode shift alone, since single-turn accuracy is ~25 points higher than multi-turn accuracy and 14.9% of samples shifted from multi-turn to single-turn).

The SAGE-Flash variant shows an even stronger version of this pattern: 940 single-turn (53.9%) at 78.8% accuracy and 804 multi-turn (46.1%) at a much-improved 63.4% accuracy. The multi-turn accuracy jump from 54.3% to 63.4% reflects the better tool execution quality when Gemini-2.5-Flash handles ground-event and analyze — the orchestrator's decisions about when to use tools haven't changed substantially (similar single-turn/multi-turn split), but the tools produce more accurate results when invoked, raising the multi-turn success rate.

MINERVA Cross-Benchmark Evaluation (Table 5)

To assess generalization beyond the SAGE-Bench distribution, the paper evaluates on MINERVA, a complex video reasoning benchmark covering sports, short films, and cooking videos. The results in Table 5 use a held-out subset of 1,473 training samples where SAGE-MM is based on Qwen2.5-VL-7B-Instruct (not the primary Qwen3-VL-8B-Instruct used in other tables, making this a cross-model-family evaluation as well).

The base Qwen2.5-VL-7B-Instruct achieves 32.7% overall on this subset. The SFT-only SAGE actually degrades performance to 28.3%, consistent with the earlier finding that SFT alone can hurt strong base models. The SFT+RL SAGE recovers to 32.0%, approximately matching the base model overall. However, the critical duration-split reveals the same pattern as SAGE-Bench: on videos ≤600 seconds, SAGE SFT+RL (34.7%) slightly underperforms the base model (37.8%), while on videos >600 seconds, SAGE (28.4%) outperforms the base model (25.8%) by +2.6 percentage points. SAGE-Flash pushes the long-video gain to +3.2% (29.0% vs. 25.8%).

The comparison to RL-tuned baselines on this benchmark is informative: VideoRFT-7B (30.4%), VideoMind-7B (30.7%), and Video-R1-7B (31.5%) all underperform the base Qwen2.5-VL-7B on the overall metric, with VideoChat-R1.5-7B achieving 33.8% — slightly above the base. The same pattern of RL-trained Direct models failing to improve over base model holds on MINERVA as it did on SAGE-Bench. However, VideoChat-R1.5 shows a notably different pattern: it achieves 35.3% on short videos and 31.8% on long videos — performing well on both, unlike SAGE which trades short-video performance for long-video gains. This suggests VideoChat-R1.5's agent design and training (which the paper describes as using a "mixed" approach) may be more robust across duration ranges, though its overall accuracy (33.8%) still trails VideoChat-R1.5's reported numbers on other benchmarks.

Eval Runtime vs. Accuracy Tradeoff (Table 10)

Table 10 presents an efficiency analysis comparing accuracy to inference runtime per sample across all methods. The key finding is that SAGE achieves 68.0% accuracy at 8.6 seconds per sample, which represents a favorable point on the accuracy-efficiency frontier.

For the Direct Qwen3-VL-8B-Instruct baseline sweeping frame counts: 16 frames achieves 55.7% at 0.8 seconds; 64 frames gives 62.3% at 2.3 seconds; 128 frames (the default SAGE input setting) gives 64.9% at 3.6 seconds; 256 frames gives 66.1% at 5.7 seconds; 512 frames gives 65.9% at 7.8 seconds. To match SAGE's 68.0% accuracy, the Direct baseline would require substantially more than 512 frames — and at 512 frames (7.8 seconds), it's already approaching SAGE's runtime while achieving lower accuracy (65.9% vs. 68.0%). The saturation at higher frame counts (1024 frames drops to 62.5%, 1536 drops to 60.8%) suggests that simply adding more frames to a Direct model hits a performance ceiling due to context-length limitations or attention dilution, a ceiling that SAGE's targeted frame extraction bypasses.

Compared to other RL-trained Direct models: Video-R1-7B achieves 57.6% at 7.3 seconds — slower and far less accurate than SAGE. VideoRFT-7B achieves 55.3% at 7.2 seconds. These models are slower than the base Direct model because their chain-of-thought reasoning traces add generation time, but the additional reasoning doesn't translate to accuracy improvements on SAGE-Bench.

Compared to Agent baselines, the runtime advantage is dramatic. VideoMind-7B, the fastest agent baseline, achieves only 50.0% accuracy at 24.7 seconds per sample — nearly 3× slower than SAGE. VideoChat-R1.5-7B achieves 54.8% at 132.1 seconds (15× slower). VideoExplorer-7B achieves 50.1% at 137.7 seconds (16× slower). LVAgent achieves 49.7% at 92.9 seconds (11× slower). VideoAgent achieves 42.0% at a staggering 1,445 seconds per sample (168× slower). The paper attributes these runtime differences to design choices: VideoAgent's mandatory preprocessing of every 2-second subclip, VideoExplorer's initial 30-second embedding computation and multi-step retrieval, and VideoMind's repeated verifier invocations for each of five candidate segments per grounding step.

The 8.6-second average runtime for SAGE breaks down as: a fixed Stage-1 cost for processing 128 frames and generating the initial context (~3-4 seconds, based on the Direct baseline at 128 frames taking 3.6 seconds), plus variable Stage-2 cost for tool calls and additional reasoning. The any-horizon nature means that simple questions incur only the Stage-1 cost (single-turn), while complex questions pay for additional tool invocations. The average of 8.6 seconds across 1,744 samples with an average of ~2-3 tool calls per multi-turn trajectory (implied by Table 14's #turns data) suggests that each tool call adds roughly 1-2 seconds of overhead, consistent with API call latency plus model inference time for the tool backend.

Training Mode Comparison: Agent vs. Direct (Table 6)

Table 6 tests whether the improvements from SAGE come from the agent architecture or simply from training on more data. The experiment fine-tunes Qwen3-VL-4B-Instruct in two parallel setups: (a) in Agent mode with the full SAGE recipe (SFT on tool-call trajectories, then RL with the full reward structure), and (b) in Direct mode, where the same synthetic QnA pairs are used but the model is trained to output answers directly without tool calls (SFT supervises only the final answer, RL uses only the accuracy reward).

The baseline Qwen3-VL-4B-Instruct (Direct) achieves 62.7% overall. Direct training (SFT → RL) on the same QnA data improves this to 66.3% (+3.6%) — demonstrating that simply training on the synthetic QnA pairs and applying RL with the LLM-judge accuracy reward provides a meaningful boost even without agent infrastructure. The Qwen3-VL-4B-Thinking variant (which uses the model's native chain-of-thought mode) actually underperforms at 60.1%, suggesting that native thinking modes are not well-calibrated for this task distribution.

The Agent training (SAGE with SFT+RL) achieves 68.4% (+5.7% over baseline, +2.1% over Direct-trained). The improvement over Direct training is concentrated in open-ended questions: Agent achieves 57.4% vs. Direct-trained's 52.0% (+5.4%), while MCQ performance is comparable (Agent: 81.3% vs. Direct-trained: 83.0%, a slight -1.7%). This pattern is telling: for multiple-choice questions where answer format is constrained, the Direct approach works well and the agent overhead may slightly hurt; for open-ended questions where the model needs to gather information from multiple sources, the Agent infrastructure provides genuine value.

The SFT-only ablation within Agent mode shows the same pattern seen with Qwen3-VL-8B: SFT alone (64.6%) improves over the base (62.7%) but the RL stage (+3.8% to 68.4%) provides the majority of the gain. Within Direct mode, SFT (65.8%) provides almost all of the improvement, with RL adding only +0.5% to 66.3%. This suggests that the RL reward structure is doing more useful work in the Agent setting (where it must optimize tool-use behavior in addition to answer quality) than in the Direct setting (where only answer quality matters and SFT already captures most of the available improvement).

---

Ablation Studies and Robustness Checks

Tool ablation (Table 9): Removing individual tools during inference reveals their relative importance. Dropping transcribe-speech causes the largest overall decline: accuracy drops from 68.0% to 62.5% (-5.5%), driven by a catastrophic 36.5% drop on verbal questions (82.8% → 46.3%). This confirms that speech transcription is the primary mechanism by which SAGE accesses dialogue and commentary, and without it, the system is essentially blind to verbal content. Dropping extract-video-parts causes a -5.0% overall decline (68.0% → 63.0%), with visual questions declining by -5.4% (64.0% → 58.6%). Dropping analyze causes -4.6% overall (68.0% → 63.4%), with similar visual question degradation. Dropping web-search and parse-website together causes -2.5% overall (68.0% → 65.5%), with both-modality questions dropping by -5.3% (75.4% → 70.1%). Notably, dropping ground-event causes only -0.7% overall (68.0% → 67.3%), with both-modality questions actually improving slightly (75.4% → 72.3%) while visual questions remain nearly unchanged (64.0% → 64.3%). The paper attributes this to the "tool's inherent inaccuracy" — temporal grounding on entertainment videos is unreliable enough that removing it doesn't significantly hurt, and may even help by avoiding misleading grounding results that could derail subsequent reasoning. This is a negative result with practical implications: it suggests that investment in better temporal grounding for entertainment videos would have higher marginal value than any other tool improvement, since the current grounding tool contributes close to zero net benefit.

Cold-start SFT ablation (Table 13): This is the paper's most diagnostically important ablation, already discussed in Section 4 as Innovation 2. The key numbers: base Qwen3-VL-8B-Instruct (no fine-tuning) already exhibits reasonable any-horizon behavior as an orchestrator, producing 802 single-turn and 942 multi-turn responses with 63.2% overall accuracy — only marginally below the SFT model's 63.9%. The base Qwen3-VL-4B-Instruct, by contrast, shows almost no multi-turn behavior (399 multi-turn vs. 1,345 single-turn) and achieves only 54.5% accuracy — it's not a viable orchestrator without fine-tuning. When RL is applied directly to Qwen3-VL-8B-Instruct without SFT, the model collapses to 1,727 single-turn responses and only 17 multi-turn trajectories, with overall accuracy dropping to 56.6% — well below both the base model (63.2%) and the SFT+RL model (68.0%). This demonstrates that SFT is essential for preserving and improving multi-turn behavior during RL, even for models that initially show some multi-turn capability.

Evaluation mode ablation (Table 12): This test asks: after training SAGE-MM under the Agent paradigm, what happens if you evaluate it in Direct mode (forcing single-turn answers)? For Qwen3-VL-8B-Instruct after SFT, Agent mode (63.9%) slightly outperforms Direct mode (63.6%), with the difference concentrated in open-ended questions (52.4% Agent vs. 50.9% Direct). After RL, the gap widens: Agent mode achieves 68.0% vs. Direct mode's 69.8% — unexpectedly, Direct evaluation mode outperforms Agent mode on overall accuracy, though open-ended questions still favor Agent mode (55.6% vs. 57.6%). The SAGE-Flash configuration shows the same crossover: overall accuracy is higher in Agent mode (71.8% vs. 70.5% for SFT, similar for RL), but the gap is small. For Molmo2-8B, the Agent mode advantage is much larger: SFT Agent (63.3%) substantially outperforms SFT Direct (55.7%), and RL Agent (66.1%) substantially outperforms RL Direct (61.0%). This suggests that Qwen3-VL-based models are strong enough at Direct reasoning that the agent overhead can sometimes outweigh the benefits, whereas Molmo2 benefits more consistently from the agent infrastructure.

Video input ablation (Table 11): Evaluating both the base Qwen3-VL-8B-Instruct and trained SAGE without access to the video frames (text-only from metadata, transcript, and tool calls) produces dramatic accuracy drops: the base model falls from 64.9% to 42.1% (-22.8%), and SAGE falls from 68.0% to 41.0% (-27.0%). The drops are concentrated in visual questions (base: 61.9% → 33.6%; SAGE: 64.0% → 39.3%) and both-modality questions (base: 72.8% → 58.6%; SAGE: 75.4% → 48.0%). This serves primarily as a memorization check: since some videos in SAGE-Bench overlap with training videos (though QnA pairs are disjoint), a high text-only accuracy would suggest the model had memorized answers from training. The near-identical degradation for both base and trained models (22.8 vs. 27.0 percentage points) indicates that SAGE is not exploiting training video overlap for inflated performance — its improvements come from better visual reasoning, not from memorization.

Maximum steps ( $N_{max}$ ) ablation (Table 15): Varying the maximum allowed reasoning steps from 1 to 16 reveals the optimal horizon and the cost of insufficient steps. With $N_{max}=1$ (essentially Direct mode), accuracy is only 43.4% with a 46.8% no-answer rate — nearly half of questions cannot be answered in a single step. Increasing to $N_{max}=3$ raises accuracy to 62.8% but still leaves 20.2% unanswered. $N_{max}=6$ brings accuracy to 66.6% with only 3.9% unanswered. The default $N_{max}=11$ achieves the best balance: 68.0% accuracy with 1.3% unanswered. Further increases to 13 or 16 provide minimal additional benefit (67.8-67.9% accuracy, 1.1% unanswered), confirming that 11 steps is sufficient to capture the vast majority of solvable questions while avoiding the diminishing returns of allowing excessively long trajectories. The SFT-only model (without RL) shows a different pattern: accuracy increases more slowly with $N_{max}$, from 33.8% at $N_{max}=1$ to 64.6% at $N_{max}=16$, with persistently high no-answer rates even at large $N_{max}$ (3.0% at 16). The SFT model is less capable of producing answers within its step budget, while the RL-trained model has learned to be more efficient.

Turns vs. duration (Table 14): The average number of reasoning turns (Stage-1 + Stage-2 steps) increases monotonically with video duration: from 2.00 turns for videos under 60 seconds to 3.54 turns for videos over 2,400 seconds in the SFT-only model, and from 1.74 to 2.77 in the SFT+RL model. The RL model consistently uses fewer turns than the SFT model at every duration bucket (approximately 0.3-0.8 fewer turns), consistent with RL learning to prune unnecessary tool calls. The gradual increase in turns with duration demonstrates that SAGE naturally adapts its reasoning depth to the temporal scale of the input — longer videos elicit longer reasoning chains, shorter videos elicit shorter ones — which is the operational definition of any-horizon behavior. The SFT model's uniform right-shift (more turns at every duration) suggests it detects the correlation between video length and problem difficulty but overestimates how many tool calls are needed, a bias that RL corrects.

Variance analysis (Table 16): Five runs of the base Qwen3-VL-8B-Instruct on SAGE-Bench at temperature 1.0 yield accuracy scores of 64.9, 64.6, 64.9, 65.2, and 65.1 — a mean of 64.9 with standard deviation 0.22. This extremely low variance (CV = 0.34%) indicates that SAGE-Bench scores are measured with high precision and that the reported improvements (+3.1% for SAGE SFT+RL, +6.9% for SAGE-Flash) are many standard deviations above noise. The paper does not report variance for SAGE itself, likely because the regeneration protocol for malformed JSON makes SAGE outputs technically nondeterministic even at temperature 0.0, and the practical difference across runs would be negligible given the benchmark's inherent stability.

---

Critical Assessment

Claim 1: SAGE achieves "notable improvements of up to 6.1% on open-ended video reasoning tasks" (from the abstract) and "8.2% improvement on videos longer than 10 minutes."

What the experiments demonstrate: The 6.1% figure appears to reference the Qwen2.5-VL-7B-Instruct comparison in Table 4, where SAGE achieves 50.1% open-ended accuracy vs. the base model's 45.4% — a +4.7 percentage point absolute improvement but +10.4% relative. The exact 6.1% number is somewhat elusive across tables. For Qwen3-VL-8B-Instruct, the open-ended gain is only +1.6% absolute (54.0% → 55.6%). For Qwen3-VL-4B-Instruct, it's +5.9% absolute (51.6% → 57.5%, taking base from Table 4's 51.6 — wait, the table shows 51.5% for Qwen3-VL-4B-Instruct open-ended and 57.4% for SAGE SFT+RL, which is +5.9%). The 6.1% appears closest to Molmo2-8B's open-ended improvement: 48.1% (base) → 55.2% (SAGE) = +7.1%, not 6.1%. The figure likely comes from a specific model variant or an average that isn't spelled out explicitly.

The 8.2% improvement on >10-minute videos is cleanly supported by Table 7: in the 600-1200 second bucket, SAGE SFT+RL achieves 63.2% vs. base 55.0% = +8.2 percentage points. This is the single bucket where gains exceed 8%. In the 1200-2400 bucket, the gain is +2.7%; in 2400+, it's +6.3%. So "8.2% improvement on videos longer than 10 minutes" is strictly true for the 10-20 minute range but overstates the consistency of gains across all long-video buckets.

What is not demonstrated: The 8.2% figure applies to a single duration bucket (600-1200s) for a single model variant (Qwen3-VL-8B SFT+RL). It's not an average across all long videos — if you average the 600-1200, 1200-2400, and 2400+ buckets (weighted by their sample counts: 484, 147, 180 respectively), the weighted average gain is approximately (484×8.2 + 147×2.7 + 180×6.3)/(484+147+180) = 5,512/811 ≈ +6.8%. Still substantial, but not the headline 8.2%. Moreover, the SFT-only model shows negligible or negative gains in these same buckets (600-1200: +1.8%; 1200-2400: -3.4%), meaning the long-video benefit is entirely attributable to RL, not the agent architecture alone.

Claim 2: "SAGE significantly outperforms both direct single-turn video models and prior agent systems."

What the experiments demonstrate: This claim is well-supported for agent system comparisons. Every prior agent system (VideoMind, VideoExplorer, LVAgent, VideoAgent, VideoChat-R1.5, LongVT) achieves substantially lower accuracy than SAGE on SAGE-Bench, with gaps ranging from 13.2 to 26.0 percentage points (Table 4). The margin is large enough that even accounting for potential benchmark-specific advantages (SAGE was trained on data from the same source channels) the qualitative conclusion is robust.

For Direct model comparisons, the claim requires more qualification. SAGE SFT+RL (68.0%) does outperform the base Qwen3-VL-8B-Instruct (64.9%) and other open-weight Direct models. But it does not clearly outperform GPT-4o (71.6%) in Direct mode — only SAGE-Flash (71.8%) edges past GPT-4o, and by a tiny margin (0.2%). The claim that SAGE "significantly outperforms" closed-source Direct models is not supported; the more accurate characterization is that an 8B open-weight model with the SAGE system design can approximately match closed-source frontier models. The significance of this result is in the model scale efficiency (matching GPT-4o with an 8B orchestrator), not in absolute accuracy superiority.

What is not demonstrated: The comparison to Direct models gives the Direct baselines 128 frames and the full transcript — generous conditions since most RL-tuned Direct models were trained with different frame budgets and without transcripts. More importantly, the comparison does not test whether simply scaling the Direct model's frame budget could match SAGE's accuracy. Table 10 shows the Qwen3-VL-8B Direct baseline saturating around 66.1% at 256 frames and then declining — suggesting that simply adding frames cannot close the gap. But this is a single data point for a single model; the paper does not systematically compare SAGE against Direct models with matched inference compute budgets (e.g., Direct model with 2× or 4× the frame count and correspondingly higher runtime).

Claim 3: "An effective RL post-training recipe essential for instilling any-horizon reasoning ability."

What the experiments demonstrate: Tables 8, 13, and 14 collectively support this claim. RL shifts the action distribution toward the expert (Table 8: multi-turn proportion shifts from 59.5% to 45.6%, approaching the expert's 50.7%), reduces average turns at every duration bucket (Table 14), and — critically — prevents the collapse to single-turn behavior that occurs when RL is applied without cold-start SFT (Table 13). The RL recipe demonstrably changes the model's behavior in the any-horizon direction.

The claim is further supported by the comparison between Agent and Direct training modes (Table 6): RL within the Agent mode provides +3.8% over SFT, whereas RL within Direct mode provides only +0.5% over SFT. The RL reward structure is doing substantially more useful work in the Agent setting, which is exactly what you'd expect if it's specifically incentivizing any-horizon tool-use decisions.

What is not demonstrated: The paper doesn't ablate individual components of the reward structure to determine which matter most. Would RL work with only the accuracy reward and no step-level rewards? Would the reasonable-tool reward alone suffice without format and args-repeat? The calibration claim — that step rewards are set so accumulated values are "comparable" to the accuracy reward — is asserted but not ablated. A reward ablation table (e.g., accuracy-only, accuracy+format, accuracy+format+reasonable-tool, full reward) would substantially strengthen the methodological contribution. The paper's reward structure is presented as a working configuration, not as the product of systematic design-space exploration.

Claim 4: "A cost-effective synthetic QnA pipeline using Gemini-2.5-Flash" that produces "high-quality data at low cost" with "<5% error rate."

What the experiments demonstrate: The manual verification of 1,700+ samples with <5% requiring edits is a credible quality check, though it's unclear whether the verification was done by the authors (potential for unconscious bias) or by independent annotators. The cost claims (~100× savings over human annotation, ~10× savings over subclip processing) are stated without detailed cost accounting — no specific dollar figures or GPU-hour calculations are provided that would allow independent verification. The paper says human annotation "can cost approximately $30 on the Prolific platform" for a single 1-hour video, translating to ~$200,000 for the full 6,668-video dataset. The Gemini-2.5-Flash API costs for processing these videos are not reported.

What is not demonstrated: The paper doesn't directly compare the quality of Gemini-2.5-Flash-generated QnA pairs against human-written or subclip-pipeline-generated questions. The <5% edit rate establishes that the synthetic data is not obviously wrong, but it doesn't establish that the questions are as diverse, natural, or challenging as human-authored alternatives. The paper doesn't train a SAGE-MM variant on human-annotated or subclip-generated data and compare performance — all training uses the Gemini-2.5-Flash pipeline. This leaves open the question of whether a different data generation approach would produce a better or worse orchestrator.

Other Observations

Single benchmark dependency: All primary results are on SAGE-Bench, which was curated by the same team using the same data generation pipeline as the training data (different QnA pairs, overlapping video sources from the same channels). This creates a potential for subtle distributional overlap: questions generated by Gemini-2.5-Flash about Formula 1 videos from a specific channel may share stylistic properties even when the specific QnA content is disjoint. The MINERVA evaluation (Table 5) provides some cross-benchmark evidence, but the MINERVA subset is also drawn from the training distribution and uses a different base model (Qwen2.5-VL-7B) than the primary results (Qwen3-VL-8B). The paper does not evaluate on a fully independent benchmark with no training data overlap.

The SFT model's surprising weakness: The finding that SFT on expert Gemini-2.5-Flash trajectories degrades performance for Qwen3-VL-8B (63.9% SFT vs. 64.9% base, Table 4) deserves more diagnostic attention than the paper provides. If behavior cloning on expert trajectories hurts, it suggests a capability mismatch — the expert (Gemini-2.5-Flash) makes tool-use decisions that the 8B student model cannot reliably execute, so imitating those decisions leads to failures the base model would have avoided by answering directly. This is a known problem in imitation learning (the "expert-student gap") but the paper doesn't analyze which expert behaviors the student fails to replicate or whether filtering the SFT trajectories by some confidence measure could improve SFT-only performance.

Missing confidence intervals on main results: The paper reports a variance analysis only for the base model (Table 16, σ=0.22 at temperature 1.0). No variance is reported for SAGE itself, and the main results (Table 4) use temperature 0.0, which should be deterministic except for the regeneration protocol on malformed JSON. The reported accuracy numbers are thus point estimates without uncertainty quantification. Given the 1,744-sample test set and the 3-7 percentage point gaps being claimed as significant, standard binomial confidence intervals (approximately ±2.3% for 95% CI on 1,744 samples at 65% accuracy) suggest that most of the headline comparisons are statistically meaningful but the smaller gaps (e.g., SAGE 68.0% vs. SAGE-Flash 71.8%, a 3.8% gap) are near the edge of what can be confidently distinguished.

The SAGE-Flash vs. SAGE comparison conflates tool quality with orchestrator quality: SAGE-Flash uses Gemini-2.5-Flash for ground-event and analyze, while standard SAGE uses Qwen3-VL-30B-A3B-Instruct. The 3.8% gain from SAGE-Flash reflects better tool execution, not better orchestration. The paper acknowledges this but the headline numbers (71.8% for SAGE-Flash) could be misinterpreted as reflecting training improvements. The fair comparison for the training recipe is the standard SAGE configuration (68.0%), while SAGE-Flash demonstrates the ceiling with near-perfect tools.

The any-horizon claim and short-video degradation: Table 7 reveals that SAGE SFT+RL underperforms the base model on videos in the 60-180 second (-2.0%) and 180-300 second (-4.5%) ranges. This means the any-horizon agent is worse than a Direct model on a significant fraction of the benchmark — the 60-300 second range contains 506 of 1,744 samples (29%). The paper's framing emphasizes the long-video gains while downplaying these short-video losses. A user deploying SAGE on a mixed distribution of video lengths would need to assess whether the long-video gains outweigh the short-video costs for their specific use case. The paper doesn't analyze why short-video performance degrades — is it JSON formatting failures? Unnecessary tool calls that introduce errors? Regeneration overhead? Understanding the failure mode would inform whether the short-video problem can be addressed through better training or is inherent to the agent architecture.

6. Limitations and Trade-offs

1. The Difficulty Estimation Overhead Is Not Accounted for in Reported Gains

The assumption or constraint. SAGE's any-horizon reasoning capability depends on the orchestrator SAGE-MM correctly deciding per-query whether to answer directly or engage in multi-turn tool use. This decision is made in Stage 1 — the model must determine from the initial 128 frames and video metadata whether the question is answerable in a single turn. The paper does not provide a mechanism for cheaply making this determination; SAGE-MM must run its full Stage-1 inference (processing 128 frames, generating the video context C, the query intent, and the tool recommendation) to decide whether to proceed to Stage 2. Crucially, the paper does not account for the cost of incorrect mode selection in the any-horizon framework — specifically, the cases where SAGE-MM incorrectly decides to go multi-turn on a question the base model could have answered directly, incurring unnecessary tool-call overhead and sometimes producing a worse answer than direct prediction would have.

The consequence. The practical cost of any-horizon reasoning is higher than the headline runtime of 8.6 seconds per sample (Table 10) would suggest, because that average includes both the correct and incorrect mode-selection decisions without parsing out the overhead attributable to unnecessary multi-turn reasoning. More critically, SAGE's short-video performance degradation (Table 7: -2.0% in the 60-180s bucket, -4.5% in the 180-300s bucket relative to the base model) demonstrates that mode-selection errors are not rare — on videos where direct answering would suffice, SAGE sometimes engages in multi-turn reasoning that reduces accuracy. The any-horizon training does not eliminate this cost; it only reduces it relative to the SFT model (which was even worse at over-calling tools, Table 8). A practitioner deploying SAGE on a distribution skewed toward shorter videos would pay an accuracy penalty for the agent architecture without receiving the long-video benefits that justify it.

What evidence exists in the paper. Table 7 provides direct evidence: the base Qwen3-VL-8B-Instruct in Direct mode outperforms SAGE SFT+RL on videos in the 60-180s (72.3% vs. 70.3%) and 180-300s (81.9% vs. 77.4%) buckets. These 506 samples represent 29% of SAGE-Bench. Table 8 further shows that even after RL, SAGE's multi-turn accuracy (54.3%) is dramatically lower than its single-turn accuracy (79.6%), meaning every unnecessary transition from single-turn to multi-turn mode carries an expected accuracy cost of ~25 percentage points. The SFT+RL model still engages in multi-turn reasoning on 45.6% of samples — higher than the expert Gemini-2.5-Flash's 50.7% if you account for SAGE-Bench's difficulty distribution, but the paper does not analyze whether the marginal multi-turn decisions (beyond the expert's choices) are net-positive or net-negative for accuracy.

Mitigation status. The authors do not explicitly acknowledge this as a limitation. The paper frames the any-horizon capability as purely beneficial — the ability to answer directly when appropriate is presented as an efficiency gain over always-multi-turn agent systems. But the short-video degradation numbers in Table 7 are not discussed in the main text; they appear only in a table that the paper interprets through the lens of long-video gains ("SAGE shows significant improvements on samples belonging to buckets with duration longer than 600 seconds"). The cost of incorrect mode selection — and the fact that the mode selector itself consumes compute (the Stage-1 inference) — is an unresolved tradeoff that affects deployability on mixed-duration video distributions.

---

2. The SAGE System's Performance Is Tightly Coupled to Tool Quality, Creating a Hidden Dependency

The assumption or constraint. SAGE's architecture delegates specialized perception tasks — temporal grounding, visual analysis, and speech transcription — to external tool models (Qwen3-VL-30B-A3B-Instruct for grounding and analysis, Whisper-large-v3 for transcription) that are not trained or fine-tuned as part of the SAGE pipeline. The orchestrator SAGE-MM learns to rely on these tools during RL training, developing strategies that assume a certain level of tool reliability. If the tool models produce errors (incorrect timestamps, hallucinated answers to analyze queries, transcription mistakes), SAGE-MM may propagate those errors into incorrect final answers or, worse, make downstream decisions based on faulty tool outputs.

The consequence. SAGE's performance is not solely a function of the orchestrator's quality — it is the product of an orchestrator-tool stack whose error characteristics are intertwined. This creates two practical problems. First, deployability is fragile: upgrading or replacing a tool model (e.g., switching to a better temporal grounder) may not improve system performance if SAGE-MM has learned tool-use strategies adapted to the specific error patterns of the training-time tools. A new tool might be more accurate overall but fail in different ways that the orchestrator is not prepared for. Second, the source of errors is difficult to attribute: when SAGE gets a question wrong, it's unclear whether the orchestrator made a poor tool-use decision, whether the tool produced bad output, or whether the orchestrator failed to recognize that the tool output was incorrect. The paper's tool ablation (Table 9) shows that dropping ground-event causes only a -0.7% overall accuracy decline and actually improves performance on questions requiring both visual and verbal reasoning (75.4% → 72.3% — wait, that's a decline; rechecking: both-modality drops from 75.4% to 72.3% when dropping ground-event, which is a -3.1% decline, not an improvement). The key point is that the ground-event tool contributes close to zero net benefit because its errors approximately cancel its successes, yet SAGE-MM has been trained to use it — meaning the training process has not learned to work around the tool's unreliability.

What evidence exists in the paper. The SAGE-Flash vs. SAGE comparison (Table 4, Table 7) is the clearest evidence of tool dependency. Replacing Qwen3-VL-30B-A3B-Instruct with Gemini-2.5-Flash as the ground-event and analyze backend produces a +3.8% overall accuracy gain (68.0% → 71.8%) and dramatically larger gains on long videos (+14.6% in the 600-1200s bucket, Table 7). This is with the same orchestrator weights — SAGE-MM's policy hasn't changed, only the tools it calls have improved. The fact that tool quality alone accounts for 3.8 percentage points of accuracy (more than the entire gain from SFT+RL over the base model, which is +3.1%) demonstrates that orchestrator performance is bounded by tool quality. The tool ablation (Table 9) further shows that removing analyze costs -4.6% overall, removing extract-video-parts costs -5.0%, and removing transcribe-speech costs -5.5% (including a catastrophic -36.5% on verbal questions). These are not small degradations — the system's performance is heavily dependent on every tool except ground-event.

Mitigation status. The paper does not discuss the coupling between orchestrator training and tool quality as a limitation. The SAGE-Flash results are presented as an aspirational ceiling ("when using Gemini-2.5-Flash as a tool... further boosts...") rather than as evidence that the default SAGE configuration is substantially tool-limited. The paper acknowledges that ground-event has "inherent inaccuracy" (Section 4.4, discussion of Table 9) but does not explore whether RL training could adapt to tool unreliability — for example, by learning to verify tool outputs or to call alternative tools when grounding fails. The tool quality dependency is not listed as a limitation or in the future work section, which focuses on "more advanced agent-centric policy optimization algorithms" and "empowering the system to select appropriate tools and synthesize new ones."

---

3. A Single Benchmark from a Single Domain with Training-Distribution Overlap Limits Generalizability Claims

The assumption or constraint. All primary experimental results are on SAGE-Bench, a benchmark curated by the authors from YouTube entertainment videos using the same Gemini-2.5-Flash-based synthetic data pipeline that generated the training data. While the specific QnA pairs are disjoint between training and evaluation, the videos are drawn from the same 13 YouTube channels and the questions are generated by the same model (Gemini-2.5-Flash) using the same prompt template. This creates a potential for stylistic overlap: both training and evaluation questions share the linguistic patterns, difficulty distribution, and question-type biases of Gemini-2.5-Flash's generation. A model trained on this data may learn to exploit these stylistic cues rather than developing general video reasoning capabilities. The paper provides only one cross-benchmark evaluation (MINERVA, Table 5), which uses a different base model (Qwen2.5-VL-7B-Instruct rather than the primary Qwen3-VL-8B-Instruct) and evaluates on a training subset of 1,473 samples, not an independently curated test set — making it a generalization check across video domains but not a clean evaluation on a fully external benchmark.

The consequence. The paper's claims about SAGE's effectiveness — particularly the absolute accuracy numbers and the comparisons to baseline models — may not transfer to video domains outside entertainment, to videos from sources not represented in the training channels, or to question styles not captured by Gemini-2.5-Flash's generation patterns. This is especially concerning for the open-ended question results, since the LLM-as-judge evaluation (GPT-4o) may share biases with the Gemini-2.5-Flash data generator (both are large proprietary models with potentially similar notions of what constitutes a good answer). A practitioner deploying SAGE on, say, surveillance footage, medical procedure videos, or user-generated content from a different platform cannot infer expected accuracy from the SAGE-Bench numbers.

The entertainment-video focus also means the tool set and strategies SAGE learns may not transfer. Web search is highly effective for Formula 1 videos (where external knowledge about teams and drivers is readily available online) but might be useless or actively misleading for proprietary corporate videos or personal recordings. The transcribe-speech tool was critical on SAGE-Bench (Table 9: -36.5% on verbal questions when removed), but this reflects the dialogue-heavy nature of comedy and talk-show content — a user deploying SAGE on silent surveillance footage would find this tool contribution nonexistent.

What evidence exists in the paper. The MINERVA evaluation (Table 5) is the only cross-benchmark data point, and it paints a more nuanced picture than SAGE-Bench. On MINERVA, SAGE SFT+RL (32.0%) approximately matches the base Qwen2.5-VL-7B-Instruct (32.7%) overall, and the SFT-only model (28.3%) substantially underperforms the base — unlike on SAGE-Bench where SFT-only for Qwen2.5-VL-7B improves over base (61.1% vs. 58.6%, Table 4). The cross-benchmark pattern suggests SAGE's benefits are not uniform: on MINERVA, SAGE provides gains only on long videos (>600s: +2.6%) while losing on short videos (<600s: -3.1%), similar to SAGE-Bench but with lower absolute improvements. The paper does not evaluate on any standard video reasoning benchmark besides MINERVA — no results on MLVU, Video-MME, EgoSchema, or other widely-used benchmarks that would allow comparison to the broader literature beyond the baselines the authors re-evaluated on SAGE-Bench.

Mitigation status. The paper does not acknowledge the single-benchmark limitation. The future work section mentions "training on data from broader domains to handle more use cases" as a natural advancement, implicitly recognizing the current domain restriction, but there is no discussion of whether the current results should be interpreted as specific to entertainment videos or as evidence for general video reasoning capability. The MINERVA results are presented as validation ("validating the effectiveness of our approach for long video reasoning") without caveats about the lower absolute improvements or the different base model used. The statement that SAGE-Bench is designed "with a focus on open-ended questions simulating the needs for real-world use-cases for entertainment videos" (Section 4.2) is honest about domain scope, but the paper's broader claims ("demonstrating the scalability of our system design") implicitly assume domain transfer that is not tested.

---

4. The Cold-Start SFT Dependency Means the Approach Cannot Bootstrap Without a Stronger Expert Model

The assumption or constraint. SAGE's training pipeline critically depends on a cold-start supervised fine-tuning stage using trajectories generated by a significantly more capable expert model (Gemini-2.5-Flash acting as SAGE-MM). The paper demonstrates (Table 13) that applying RL directly to the base Qwen3-VL-8B-Instruct without this SFT stage causes the model to collapse to single-turn behavior — it essentially stops using tools. This means the entire approach is gated on access to a model that is already capable of performing the multi-turn reasoning task well enough to generate usable training trajectories. For the paper, this expert is Gemini-2.5-Flash — a closed-source, API-gated model that the authors do not control and that may change behavior or pricing over time.

The consequence. SAGE cannot be used to bootstrap video reasoning capabilities from scratch using only open-weight models. If a research group wants to train a SAGE-like system for a new video domain (medical, industrial, educational) using only open-weight models, they face a chicken-and-egg problem: they need an expert orchestrator to generate SFT trajectories, but the orchestrator is what they're trying to build. The paper does not provide evidence that a weaker model (e.g., Qwen3-VL-30B-A3B-Instruct, which is used for tools but not as the SFT expert) could serve as the trajectory generator. The quality floor for the expert is unknown — how much better than the student does the expert need to be for behavior cloning to work? The paper's negative result on SFT performance for Qwen3-VL-8B (Table 4: SFT actually degrades accuracy from 64.9% to 63.9%) suggests that even with Gemini-2.5-Flash as the expert, the behavior cloning step is imperfect — the student model cannot fully replicate the expert's reasoning, leading to compounding errors during tool-use trajectories. A weaker expert would produce even noisier trajectories, potentially making SFT harmful rather than helpful.

More subtly, the dependency on a specific closed-source model for data generation raises reproducibility concerns. Future researchers attempting to replicate SAGE's results may find that Gemini-2.5-Flash's behavior has changed (due to model updates), producing different QnA pairs or tool-call trajectories that alter the training data distribution. The paper's data generation prompt is provided (Figure 6), but the model generating the data is not under the researchers' control, making exact reproduction impossible.

What evidence exists in the paper. Table 13 is the key evidence: RL without SFT on Qwen3-VL-8B-Instruct produces 1,727 single-turn and only 17 multi-turn trajectories, with accuracy collapsing to 56.6% — well below any other configuration. Even the base model without any fine-tuning achieves 63.2% by engaging in substantial multi-turn reasoning (942 multi-turn trajectories). The RL process without SFT thus destroys existing multi-turn capability rather than enhancing it. The paper's hypothesis — that "the base model's training objective... strongly biases it toward directly producing final answers" — is plausible but not empirically validated (no diagnostic experiments on why RL fails without SFT). Table 4 shows that SFT on Gemini-2.5-Flash trajectories improves some base models (Qwen2.5-VL-7B: +2.5%, Qwen3-VL-4B: +1.9%) but degrades the strongest one (Qwen3-VL-8B: -1.0%), indicating that even the expert trajectories contain behaviors the student cannot learn productively.

Mitigation status. The paper does not frame the expert dependency as a limitation. The data generation pipeline is presented as a contribution (Section 3.2: "cost-effective synthetic QnA pipeline") without discussion of the reliance on a closed-source model that may not be available to all practitioners. The future work section does not mention reducing expert dependency — it focuses on "more advanced agent-centric policy optimization algorithms" and "empowering the system to select appropriate tools and synthesize new ones," both of which implicitly assume continued access to strong expert models for training. The paper does not explore whether the expert trajectories could be filtered, distilled, or augmented to work better with a weaker student, nor does it test whether iterative self-training (using the SFT model to generate new trajectories for further training) could reduce or eliminate the expert dependency over multiple rounds.

---

5. The RL Reward Design Is Calibrated by Hand Without Systematic Ablation, Leaving Key Training Dynamics Unexplained

The assumption or constraint. The paper's RL recipe (Section 3.3) uses a multi-component reward function with six distinct reward terms (format, reasonable-tool, args-repeat, args-valid, and two accuracy-reward cases) across four reward magnitudes (0.05, 0.10, 0.25 via sqrt scaling, and 1.0/1.25 for accuracy). The authors state that "the values for the step rewards [were set] such that the accumulated step-level reward for a trajectory with 10 steps would be comparable to the accuracy reward." This calibration principle is asserted but not empirically validated. The paper does not report any experiments that vary the reward weights, remove individual step-level reward components, or test alternative reward structures. The reward function is presented as a single working configuration whose individual components' contributions are unknown.

The consequence. A practitioner attempting to adapt SAGE's RL recipe to a new domain or model cannot determine which reward components are essential and which are incidental. Is the reasonable-tool reward (judged by GPT-4o at every step) worth its computational cost? Could the same any-horizon behavior be achieved with only the accuracy reward and the format reward? The paper's finding that RL adds little in Direct mode (+0.5% over SFT, Table 6) but substantially more in Agent mode (+3.8% over SFT) suggests the step-level rewards are doing work, but it's unclear whether all of them are necessary or whether a simpler reward structure would suffice. The args-repeat penalty, for instance, uses a sqrt(num-repetitions) form that is never ablated against a simple linear penalty — the square root was a design choice whose justification ("diminishing marginal penalties") is plausible but untested.

The reasonable-tool reward is particularly concerning from a reproducibility standpoint. It requires calling GPT-4o at every step of every trajectory during RL training to judge whether the current tool call "is reasonable given the question and previous tool calls." For 7.68k training samples × 8 rollouts per sample × up to 11 steps per trajectory, this could require up to ~675,000 GPT-4o API calls during training. The paper does not report the cost, latency, or reliability of this judgment process, yet it is a core component of the training loop. If GPT-4o's reasonableness judgments are noisy or biased, the RL training could be steered in unintended directions. The paper also does not report whether the reasonable-tool judge's decisions correlate with downstream trajectory success — a trajectory might have perfectly reasonable tool calls at every step yet still produce a wrong answer, or might have an unconventional but effective tool-use strategy that the judge incorrectly penalizes.

What evidence exists in the paper. No reward ablation experiments are reported. The paper does not provide learning curves showing how individual reward components evolve during training, nor does it analyze which reward terms correlate most strongly with final trajectory success. The closest thing to a reward analysis is Table 8, which shows the behavioral effect of RL (shift in single-turn vs. multi-turn distribution, accuracy improvements in both modes), but this is an aggregate outcome that could be produced by multiple different reward configurations. Table 9 (tool ablation) shows which tools matter for final accuracy, but this is an inference-time analysis that doesn't speak to which rewards shaped the training dynamics. The calibration claim — that step rewards are "comparable" to the accuracy reward — is not accompanied by any analysis of actual reward magnitudes observed during training. The paper doesn't report the mean, variance, or distribution of R_i across trajectories, so the reader cannot verify whether the calibration principle held in practice.

Mitigation status. The authors do not acknowledge the lack of reward ablation as a limitation. The RL recipe is presented as a complete, validated methodology, and the specific reward weights are given without discussion of how they were arrived at or whether they are robust to perturbation. The future work section mentions "integrating more advanced agent-centric policy optimization algorithms" as a promising direction, which implicitly acknowledges that the current GRPO setup could be improved, but this is framed in terms of algorithmic advancement rather than as a gap in the current paper's empirical validation. A reader seeking to understand why SAGE's RL works — which reward components are load-bearing and which are decorative — finds no answers in the paper.

---

6. The Latency-Variance Tradeoff of Multi-Turn Agent Behavior Is Not Characterized

The assumption or constraint. SAGE's multi-turn reasoning loop introduces variable and potentially unbounded latency into the inference process. Each tool call adds API latency (web search, speech transcription) or model inference time (grounding, analysis), and the number of tool calls is determined dynamically by SAGE-MM during inference. The paper reports an average runtime of 8.6 seconds per sample (Table 10) but does not report the variance of this runtime — the minimum, maximum, or any percentile breakdown. For a system deployed in interactive applications (where users expect responses within a few seconds) or batch processing pipelines (where tail latency determines throughput), the average is insufficient to characterize the user experience or the infrastructure requirements.

The consequence. In an interactive setting, a user asking a simple question about a short video might receive an answer in ~3-4 seconds (single-turn, Stage-1 only) — comparable to a Direct model. But a user asking a complex question about a long video might wait 20-30 seconds or more while SAGE iterates through web searches, speech transcription, temporal grounding, frame extraction, and analysis. This creates an unpredictable user experience where response time varies by an order of magnitude depending on the question. The paper does not analyze whether the long-tail latency is concentrated on questions where SAGE eventually succeeds or fails — if the longest trajectories are predominantly failure cases (the model searches extensively but still produces a wrong answer), the user experience is particularly poor: a long wait followed by an incorrect response.

The latency variance also creates deployment challenges for throughput-oriented applications. If SAGE is serving requests in a batch setting, the slowest requests determine pipeline latency. With $N_{max}=11$ maximum steps and each step potentially triggering an API call (web search, parse-website) with unpredictable network latency, a small fraction of requests could take dramatically longer than the average, creating straggler problems. The paper does not discuss whether $N_{max}$ can be reduced at deployment time to cap tail latency, and if so, what the accuracy cost would be (Table 15 provides some guidance: reducing $N_{max}$ from 11 to 6 costs -1.4% accuracy for SAGE SFT+RL).

What evidence exists in the paper. Table 10 reports only the mean runtime (8.6 seconds for SAGE). No variance, percentile, or distribution information is provided. Table 14 shows the average number of turns by duration bucket (ranging from 1.74 for <60s videos to 2.77 for 2400+s videos after RL), which gives a rough sense of how latency scales with video length but does not capture within-bucket variance. Table 15 shows the no-answer rate at different $N_{max}$ values, which indirectly speaks to latency — at $N_{max}=11$, 1.3% of samples fail to produce an answer within the step budget, meaning they run the full 11 steps and then fail, representing worst-case latency with a negative outcome. The paper does not report the no-answer rate broken down by video duration or question type, so it's unclear whether these worst-case trajectories cluster in particular regimes.

The paper's comparison to Agent baselines in Table 10 highlights SAGE's relative speed advantage (3-168× faster than prior agent systems), but this comparison is against systems with fundamentally different preprocessing requirements, not against an apples-to-apples latency SLA. More importantly, the comparison doesn't address the variance issue — a system with lower average but higher variance latency may be less deployable than one with higher average but predictable latency.

Mitigation status. The paper does not acknowledge latency variance as a concern. The $N_{max}$ parameter is discussed in terms of accuracy-no-answer tradeoffs (Table 15) rather than latency capping, and the paper does not suggest $N_{max}$ as a deployment-time knob for controlling tail latency. The future work section does not mention latency optimization, real-time performance, or user-facing deployment considerations. The any-horizon framing emphasizes the benefit of adaptive computation (less compute wasted on simple queries) but does not discuss the cost of unpredictable computation from the user's perspective. This is a significant gap given that the paper motivates itself through real-world entertainment video use cases where interactive response times would be expected.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the video reasoning field from a paradigm of architecture-as-destiny — where you either build a Direct model or an Agent system — toward a paradigm of learned behavioral meta-strategy, where the operating mode (single-turn vs. multi-turn) is a policy decision optimized through reinforcement learning rather than baked into the system design. This is not merely an incremental combination of existing ideas; it is a conceptual reframing of what it means to build a video reasoning system. Prior work implicitly assumed that the choice between Direct and Agent paradigms was a design-time decision reflecting fundamentally different philosophies about how video understanding should work. SAGE demonstrates that this choice can and should be deferred to inference time and learned from data — the same model can and should do both, switching modes per-query based on estimated difficulty and information needs.

The magnitude of this shift is significant but bounded. It is not a paradigm shift on the scale of the Transformer replacing recurrent architectures, nor on the scale of large-scale pretraining replacing task-specific computer vision. It is better characterized as a methodological reframing with direct practical consequences: the paper provides a complete recipe (data generation, cold-start SFT, multi-reward RL) for building any-horizon video agents, and demonstrates that this recipe produces consistent improvements across four base model families (Qwen2.5-VL-7B, Qwen3-VL-4B, Qwen3-VL-8B, Molmo2-8B) with gains concentrated on long videos where the Direct paradigm breaks down. The fact that the recipe transfers across architectures suggests it captures something general about agent optimization rather than being tuned to a specific model's quirks.

The paper resolves a specific empirical contradiction that has been brewing in the literature. Prior work showed that (1) RL-trained Direct models improve on MCQ-heavy video benchmarks (Video-R1, VideoRFT, LongVILA-R1), (2) Agent systems underperform Direct models on open-ended questions despite their architectural flexibility (VideoMind, VideoExplorer, LVAgent in Table 4), and (3) RL alone fails to induce multi-turn agent behavior (Table 13: collapse to single-turn). These three findings appeared contradictory: if RL helps reasoning, and agents are more flexible than Direct models, why do RL-trained agents not dominate? SAGE's resolution is that RL for agents requires a specific cold-start initialization and a multi-component reward structure that prior work did not provide. The contradiction wasn't about RL versus agents — it was about the specific conditions under which RL can optimize agent behavior. By identifying those conditions (SFT before RL, balanced single-turn/multi-turn training data, process rewards alongside accuracy rewards), the paper converts an apparently contradictory literature into a coherent picture with clear boundary conditions.

This reframing makes certain research directions substantially more attractive and others less so:

More attractive: Training orchestrator models that adaptively switch between fast and slow reasoning modes across any domain (not just video) becomes a natural extension. The any-horizon concept is domain-agnostic — any task where problem difficulty varies and where tool use incurs overhead can benefit from learned mode selection. The paper's negative result on RL-without-SFT (Table 13) also makes understanding cold-start requirements for RL agent training a high-priority research question: under what conditions can RL discover multi-turn behavior from scratch, and when is behavior cloning necessary? This connects directly to the broader RL-for-reasoning literature (DeepSeek-R1, etc.) and suggests that chain-of-thought reasoning and tool use occupy different points on a spectrum of "discoverability" through pure RL.

Less attractive: Building increasingly complex agent architectures with hardcoded multi-turn pipelines (the VideoAgent/VideoExplorer/LVAgent lineage) becomes harder to justify when a simpler architecture with learned mode selection achieves 13-26 percentage point higher accuracy (Table 4) at 3-168× lower inference latency (Table 10). The marginal return from engineering more sophisticated preprocessing pipelines or retrieval mechanisms appears low compared to investing in better orchestrator training. Similarly, the paper's demonstration that RL-trained Direct models using string-matching rewards actually degrade on open-ended questions (Video-R1: 57.6% vs. base 58.6% for Qwen2.5-VL-7B, Table 4) makes the continued use of MCQ-only training with option-matching rewards difficult to justify for any system intended to handle real user queries. If your deployment use case includes open-ended questions, training exclusively on MCQ with letter-matching rewards is actively harmful — a finding that should redirect the video RL community toward LLM-as-judge evaluation during training.

The paper also establishes SAGE-Bench as a new evaluation standard that fills a genuine gap. By focusing on open-ended questions (54% of the benchmark) drawn from entertainment videos with an average duration of 727 seconds, SAGE-Bench tests capabilities that existing MCQ-heavy benchmarks (MLVU, Video-MME, LongVideoBench) do not. The benchmark's construction — synthetic QnA generation followed by manual verification with <5% edit rate — provides a template for cost-effective benchmark creation in new domains. However, the single-domain nature of SAGE-Bench also means the community should not over-index on it; the paper's results on MINERVA (Table 5) show smaller and less consistent gains, indicating that SAGE's benefits are partially domain-dependent.

---

Follow-Up Research This Work Enables

Characterizing the expert-student gap in behavior cloning for multi-turn tool use. The paper reveals a puzzling result: SFT on Gemini-2.5-Flash expert trajectories improves some base models (Qwen2.5-VL-7B: +2.5%, Qwen3-VL-4B: +1.9%) but degrades the strongest one (Qwen3-VL-8B: -1.0%, Table 4). This suggests a non-monotonic relationship between student capability and behavior cloning effectiveness that the paper does not analyze. A strong follow-up would systematically vary the expert model quality (e.g., using Gemini-2.5-Flash, GPT-4o, Qwen3-VL-30B, and Qwen3-VL-8B itself as experts) and the student model scale (2B, 4B, 8B, 30B parameter variants) to map out when behavior cloning helps versus hurts. The key measurement would be per-step action agreement between student and expert, broken down by trajectory success — does the student fail on trajectories where it deviates from the expert, or does it fail even when following the expert's actions because it cannot execute them reliably? This would directly inform whether the expert-student gap is about policy mismatch (students should follow expert decisions but can't) or capability mismatch (experts make decisions the student fundamentally cannot execute), with different mitigation strategies for each.

Reward structure ablation for multi-turn agent RL. The paper's reward function (Section 3.3, Equation 2) combines four step-level reward components with a four-tier accuracy reward, but no individual component is ablated. A systematic follow-up would train SAGE-MM variants with incrementally simpler reward structures: (a) accuracy-only, (b) accuracy + format, (c) accuracy + format + reasonable-tool, (d) accuracy + format + reasonable-tool + args-repeat, (e) full reward. The key outcomes to measure are: final accuracy on SAGE-Bench, the single-turn vs. multi-turn action distribution (does the reasonable-tool reward specifically prevent tool undercalling?), training stability (do certain reward combinations cause the policy to diverge?), and per-step tool-use quality. A particularly diagnostic experiment would train with only the accuracy reward and see whether the any-horizon behavior emerges — or whether, as the paper's SFT-to-RL shift suggests, step-level rewards are necessary to calibrate tool-use frequency. This ablation would transform the paper's RL recipe from a "working configuration" into a set of understood principles about which reward components are load-bearing.

Iterative self-training to eliminate the closed-source expert dependency. SAGE's training pipeline depends on Gemini-2.5-Flash for both QnA generation and cold-start SFT trajectories. This makes the approach inaccessible to researchers without API access to strong proprietary models and creates a reproducibility risk as those models change. A natural extension would test whether iterative self-training can bootstrap from a weaker expert or even from no expert at all. The protocol: (1) train SAGE-MM using the paper's recipe with a weaker open-weight expert (e.g., Qwen3-VL-30B-A3B-Instruct) for SFT; (2) use the resulting model to generate new tool-call trajectories on unlabeled videos; (3) filter these trajectories by outcome (keep only those producing correct answers); (4) retrain SAGE-MM on the expanded, self-generated dataset; (5) repeat. The key measurement is whether performance approaches the Gemini-2.5-Flash-expert baseline over multiple rounds, and how many rounds are needed. If self-training works, it provides a path to building domain-specific video agents without any proprietary model dependency. If it fails — perhaps because the weaker expert produces trajectories that reinforce bad strategies — that negative result would establish a hard lower bound on the expert quality needed for this approach.

Cross-domain generalization of any-horizon reasoning. The paper's results are entirely on entertainment videos from 13 YouTube channels. A critical follow-up would evaluate the same trained SAGE-MM weights on video domains with fundamentally different characteristics: instructional videos (HowTo100M), egocentric video (Ego4D), surveillance footage, sports broadcasts with different commentary styles, and procedural videos (assembly, cooking). The core question is whether SAGE has learned general any-horizon reasoning or has specialized to the temporal structure and information density of entertainment content. Entertainment videos have a particular rhythm — dialogue interspersed with visual action, narrative arcs, recurring characters — that may not generalize. A surveillance video has long stretches of nothing happening punctuated by brief events; a cooking video has dense procedural steps with strong temporal ordering. Does SAGE's learned tool-use strategy (when to transcribe, when to ground temporally, when to search the web) transfer, or does it need domain-specific retraining? The MINERVA results (Table 5) provide a weak positive signal but are on an overlapping training distribution; a clean evaluation on fully held-out domains with no training overlap would determine whether SAGE represents a general advance or a domain-specific one.

Scaling the orchestrator model size to determine whether tool-use capability follows a scaling law. The paper trains SAGE-MM at 4B, 7B, and 8B parameter scales and finds consistent improvements from the SAGE recipe. But it does not test whether larger orchestrators (30B, 70B, or larger) would benefit proportionally more or less. The key question: does the gap between SAGE and the Direct baseline grow or shrink with model scale? If larger models are inherently better at tool use (because they have stronger reasoning capabilities), the SAGE recipe might provide superlinear gains — a 30B SAGE-MM might substantially outperform a 30B Direct model. Conversely, if larger models are already better at compressing video information into a single forward pass, the marginal benefit of agent infrastructure might shrink with scale, and the overhead of multi-turn reasoning might increasingly outweigh its benefits. A scaling study mapping orchestrator size against accuracy (both Direct and Agent modes) would reveal whether the any-horizon approach is a transitional technology that matters most at smaller scales, or a permanent architectural advantage that persists across scale.

Testing whether the any-horizon behavior can be elicited through prompting rather than RL. The paper demonstrates that RL with a specific reward structure produces any-horizon behavior, but it does not test whether simpler methods could achieve similar results. A diagnostic experiment would compare: (a) the base Qwen3-VL-8B-Instruct with a carefully engineered system prompt that instructs it to "answer directly when possible, use tools only when necessary"; (b) the SFT model trained on expert trajectories; (c) the SFT+RL model. If (a) approaches (c) in performance, then the any-horizon capability is primarily a prompting discovery rather than a training achievement, and the paper's complex RL recipe is unnecessarily expensive. If there is a large gap between (a) and (c), as the paper's results on base-model function calling suggest (Table 13: base Qwen3-VL-8B achieves 63.2% but with suboptimal action distribution), then the training recipe is genuinely necessary. This experiment would also test whether the paper's claim about RL being "essential for instilling any-horizon reasoning ability" (Section 3.3) is robust to strong prompting baselines.

---

Practical Applications and Downstream Use Cases

Cost-effective video Q&A for entertainment platforms. Streaming services and video platforms (YouTube, Netflix, sports broadcasters) could deploy SAGE-like systems to power natural-language Q&A interfaces over their content libraries. A user watching a Formula 1 race replay could ask "Did Hamilton pit before or after Verstappen in the first round of stops?" and receive an answer without manually scrubbing through hours of footage. The paper's runtime numbers (8.6 seconds average, Table 10) make this viable for asynchronous or near-real-time interaction — not fast enough for live commentary during playback, but practical for post-viewing queries. The key economic insight is that SAGE-Flash (71.8% accuracy, Table 4) achieves this using an 8B open-weight orchestrator with Gemini-2.5-Flash as a tool backend. For a platform serving millions of daily queries, the cost differential between processing every query with a closed-source Direct model versus routing only hard queries to expensive tools through a cheap orchestrator could be substantial. The paper's finding that SAGE's advantage is concentrated on long videos (63.2% vs. 55.0% in the 600-1200s bucket, Table 7) means the system provides the most value exactly where users need it most — for lengthy content that is prohibitively time-consuming to navigate manually.

Targeted video segment retrieval for content moderation and compliance. Organizations that need to review long videos for policy violations, brand safety, or regulatory compliance currently rely on either human reviewers watching full videos (expensive, slow) or keyword-based transcript search (misses visual violations). A SAGE-like system could accept natural-language queries about policy-relevant events ("Does this video contain any scenes of dangerous driving?" or "Are there any shots where the product logo is obscured?") and perform targeted visual inspection through ground-event and analyze tool calls, only extracting and examining the relevant segments. The paper's tool ablation (Table 9) shows that extract-video-parts and analyze together account for ~9.6 percentage points of accuracy, demonstrating the system's ability to answer visual queries through targeted extraction rather than full-video processing. The 8.6-second average inference time (Table 10) means a single GPU could process thousands of videos per day — a throughput level that makes automated compliance review economically viable for platforms handling user-generated content at scale. The web-search tool could additionally cross-reference claims made in videos against external fact-checking databases, though the paper does not test this specific use case.

Training data generation for video-language models through targeted question answering. The synthetic data pipeline (Section 3.2) that the paper uses for its own training is itself a deployable system. Content owners with large video archives could use Gemini-2.5-Flash (or future open-weight equivalents) with the paper's QnA generation prompt to automatically produce high-quality question-answer pairs spanning their entire video catalog at approximately 100× lower cost than human annotation. The resulting datasets could fine-tune video-language models for domain-specific applications — medical procedure videos with clinician-facing Q&A, industrial training videos with technician queries, or educational content with student-level questions. The paper's verification finding (<5% error rate across 1,700+ manually checked samples) provides a credible quality floor, and the percent_video_parsed mechanism (Figure 3) ensures temporal coverage that naive generation approaches miss. The cost advantage is dramatic: at ~$30 per video for human annotation and 20 minutes per video for subclip processing (Section 1, A1), generating data for a library of 10,000 hour-long videos would cost ~$300,000 via human annotation or require ~200,000 GPU-minutes via subclip processing. The Gemini-2.5-Flash single-pass approach reduces this to approximately $3,000 in API costs and ~20,000 GPU-minutes — making video QnA dataset creation feasible for organizations that could never afford the traditional approaches.

---

When to Prefer This Method

The paper does not articulate an explicit tradeoff framework against named alternatives, but its empirical results support a clear set of decision rules for practitioners choosing between Direct models, prior Agent systems, and SAGE for video reasoning:

• Prefer SAGE over prior Agent systems (VideoAgent, VideoMind, VideoExplorer, LVAgent, LongVT) when: (1) the deployment requires answering open-ended questions, not just multiple-choice — prior agent systems achieve 28-39% on open-ended vs. SAGE's 55.6% (Table 4); (2) inference latency matters — SAGE averages 8.6 seconds per sample vs. 24.7-1,445 seconds for prior agents (Table 10); (3) the video domain includes verbal content (dialogue, commentary) that transcription can capture — SAGE's transcribe-speech tool provides critical accuracy gains on verbal questions (82.8% with vs. 46.3% without, Table 9) that prior agents cannot match; (4) questions require external knowledge not present in the video frames, where web-search provides information that temporal-grounding-only agents cannot access.

• Prefer SAGE over Direct models (Qwen3-VL, Video-R1, etc.) when: (1) the video distribution includes substantial long-form content (>600 seconds), where SAGE's accuracy advantage grows to +8.2% or more (Table 7) while Direct models saturate or degrade with increased frame counts (Table 10: base model peaks at 66.1% with 256 frames, then declines to 60.8% at 1536 frames); (2) questions require integrating information from multiple, temporally distant segments — SAGE's targeted extract-video-parts and analyze tool calls can examine specific moments without processing the full video; (3) the deployment can tolerate variable inference latency (3-30 seconds depending on query complexity) rather than requiring uniform, predictable response time — the any-horizon design means simple queries are fast but complex queries may take substantially longer.

• Prefer Direct models over SAGE when: (1) the video distribution is predominantly short (<180 seconds), where SAGE underperforms the base model by 2-4.5 percentage points (Table 7) due to agent overhead costs; (2) inference latency must be strictly bounded and predictable — Direct models have fixed runtime for a given frame budget regardless of query complexity; (3) the deployment cannot access strong proprietary models for either training (Gemini-2.5-Flash for SFT trajectory generation) or inference (Gemini-2.5-Flash for tools in SAGE-Flash), since SAGE's default tool backend (Qwen3-VL-30B) leaves ~3.8 percentage points of accuracy on the table (68.0% vs. 71.8%, Table 4) and the SFT expert dependency is unresolved; (4) the video domain lacks clean correctness signals for training — if LLM-as-judge accuracy evaluation during RL training is unreliable due to domain-specific terminology or subjective answer criteria, the entire RL reward structure becomes noisy.

• Prefer SAGE-Flash (Gemini-2.5-Flash tools) over standard SAGE when: tool execution quality is the primary bottleneck (as the +3.8% gain for the same orchestrator weights demonstrates, Table 4) and the deployment can afford API calls to a closed-source model at inference time. This configuration represents the current accuracy ceiling for the SAGE architecture and should be the default choice for applications where accuracy matters more than cost or data privacy.