ArXiv: 2512.04000

🎯 Pitch

For long-form video questions, the cleverest frame-searching algorithms are often a waste of time. They actually hurt accuracy on global questions—the very tasks where simple uniform sampling shines—and are only necessary for localized queries targeting specific moments.

1. Executive Summary

This paper analyzes how the choice of frame selection strategy for long-form video understanding depends critically on the type of query being asked, proposing DIG, a training-free framework that adaptively routes queries to either uniform sampling or a specialized visual search pipeline based on a query typology distinguishing between global queries (requiring holistic understanding, e.g., "What title best summarizes this video?") and localized queries (targeting specific temporal segments, e.g., "What kind of bike is the man riding?"). The pipeline for localized queries combines Content-Adaptive Frame Selection (CAFS, which extracts representative frames using DINOv2 feature similarity and topographic prominence filtering), LMM-based reward assignment (where the vision-language model itself scores each candidate frame's relevance to the query using a two-dimensional criterion assessing both direct utility and contextual adjacency), and video refinement (an iterative reward-guided selection procedure that merges continuous temporal segments around high-scoring frames before final uniform sampling). Across MLVU, LongVideoBench, and VideoMME with Qwen2.5-VL-7B and Qwen2.5-VL-32B, DIG consistently outperforms uniform sampling and existing query-aware baselines (AKS, Q-Frame), achieving gains of up to 7.68% accuracy on MLVU at 32 frames over uniform sampling, while scaling robustly to 256 input frames where competing methods degrade below the uniform baseline—establishing that query-aware frame selection is effective only for localized queries, while for global queries uniform sampling remains both sufficient and more efficient.

2. Context and Motivation

The Core Problem: Frame Selection Strategy Must Depend on Query Type

The fundamental question this paper tackles is deceptively simple: when processing a long video to answer a question, which frames should you show to the model? This matters because—unlike standard image understanding where the entire input fits within context limits—video creates an immediate bottleneck. A typical long-form video might span 30–60 minutes. At 2 frames per second, that yields 3,600–7,200 raw frames. Even the most capable large multimodal models (LMMs) can only process a few hundred frames at most within their context windows, and each frame consumes dozens to hundreds of visual tokens. The gap between available video data and what can be fed into the model is enormous, often 10–100×. Somehow, this gap must be bridged by selecting a subset of frames that preserves the information needed to answer the query.

This is not merely a computational inconvenience—it is a fundamental information retrieval problem embedded within a reasoning task. Getting frame selection wrong means the model simply cannot answer the question, regardless of its reasoning capabilities. As the paper observes (Section 3, Figure 2), throwing more frames at the problem does not monotonically improve performance: accuracy peaks at some optimal frame count and then degrades. This is counterintuitive—why would adding more visual information hurt? The answer, as the paper demonstrates, is that irrelevant frames act as noise, diluting the model's attention and making it harder to locate the relevant evidence. This creates a tension: too few frames means missing critical content; too many means burying the signal in noise.

The paper argues that this tension cannot be resolved with a one-size-fits-all frame selection strategy. Instead, the optimal approach depends on what kind of question is being asked—specifically, whether the question requires understanding the video holistically ("What is the main theme?") or zooming in on specific moments ("What color was the car at the intersection?"). This dependency is what the paper sets out to characterize, exploit, and operationalize.

Why This Problem Matters: Practical and Conceptual Significance

The paper's framing makes clear that this is both an urgent practical problem and a conceptually under-explored one.

Practical urgency. Video understanding with LMMs is rapidly moving from research benchmarks to real-world deployment. Use cases span video question answering (VQA), automated content moderation, video summarization, lecture comprehension, surveillance analysis, and egocentric assistance (Section 1). In all these settings, videos are long, context budgets are limited, and the cost of processing many visual tokens is high—both in compute FLOPs and in wall-clock latency. The paper notes that existing models process each input frame with a fixed token budget (56 tokens per frame in their experiments; Section 5.1), meaning that a 256-frame input already requires 14,336 visual tokens—nontrivial even for models with 128K+ context windows once you account for text tokens and the quadratic attention cost.

Moreover, the paper identifies a scaling problem (Section 1): "In well-resourced environments, performance analysis at minimal frame counts (e.g., 8 or 16) offers limited practical insight, as applications typically seek to maximize frame utilization within given constraints." In other words, the field has largely validated methods in low-frame regimes (<64 frames), but real deployments push to 256, 512, or more frames. The paper's experiments at 256 and even 768 frames (Appendix F, Table 5) represent a more realistic and demanding test of whether a frame selection strategy genuinely scales. The finding that baselines like AKS and Q-Frame degrade below uniform sampling at high frame counts—while DIG continues to improve—is not just an academic benchmark comparison; it speaks to deployability.

Conceptual significance. Beyond the practical concerns, the paper addresses a conceptual gap in how the community thinks about frame selection. Prior work largely treats query-aware frame selection as a universal good: more query-relevance is better, and more sophisticated selection mechanisms should uniformly improve performance. The paper challenges this assumption head-on, arguing that the value of query-aware selection is contingent on query type (Section 1, Section 3). For global queries—which ask about the video's overall theme, topic, or narrative—uniform sampling already provides comprehensive temporal coverage, and more targeted selection may actually remove necessary context. For localized queries, by contrast, uniform sampling is actively harmful because it includes frames irrelevant to the specific temporal segment of interest.

This contingency is similar in spirit to the idea that different problems require different inference-time strategies (as explored in some LLM reasoning literature), but the paper is the first to formalize it for the video frame selection problem with empirical rigor. The finding is not merely taxonomic—it directly motivates an architectural choice (conditional routing) that yields substantial practical gains.

Where Prior Approaches Fall Short

The paper organizes prior work into several categories and identifies specific limitations in each (Section 2).

The Standard Baseline: Uniform Sampling

Uniform sampling—selecting frames at fixed temporal intervals—is the default approach used by most video LMMs (Section 2.2, Section 3). Its advantages are clear: it is simple, requires no additional computation, and maximizes temporal coverage of the video. For global queries, the paper shows it works well (Figure 5, right panels): performance on global queries remains stable or even slightly improves with more uniformly sampled frames, since holistic understanding benefits from seeing scenes distributed across the entire timeline.

However, uniform sampling has a critical failure mode for localized queries. The paper demonstrates this empirically in Figure 3: as frame count increases, performance on localized queries degrades substantially, while global query performance remains flat. The mechanism is straightforward but important: localized queries require information from specific temporal segments. When more uniformly sampled frames are added, the proportion of frames inside the relevant segment decreases, and the model must sift through growing amounts of irrelevant visual information to find the needle in the haystack. This is the video analog of the "lost in the middle" phenomenon observed in long-context LLM processing—relevant information gets diluted, and attention mechanisms struggle.

The paper also notes a subtler problem: uniform sampling with a fixed frame count does not adapt to video length (Section 6.2, Figure 6). A 2-minute video with 32 uniformly sampled frames has dense coverage (one frame every 3.75 seconds); a 60-minute video with the same 32 frames has coverage so sparse (one frame every 112.5 seconds) that entire scenes can be missed entirely. This non-linear relationship between video length and information density means that uniform sampling cannot be tuned once and applied universally—but existing methods rarely account for this.

Token Compression

Token compression methods (Section 2.2) aggregate visual information across frames or within frames to produce a compact representation, rather than selecting individual frames. Examples include memory bank approaches, temporal redundancy reduction, and hierarchical compression (VideoChat-Flash, LongVU, etc.). The paper acknowledges the efficiency of these methods but identifies a fundamental tension: "token compression techniques may often lead to excessive summarization, resulting in the loss of critical fine-grained visual details."

This matters particularly for localized queries, where the answer often depends on small visual details—a specific object, a facial expression, a text label, a momentary action. Compression that averages or pools features across frames risks smoothing away exactly the details that distinguish the correct answer from distractors. The paper also notes that "highly query-related segments may be either compressed or overly generalized"—that is, the compression itself may be query-agnostic, failing to preserve the features most relevant to answering the specific question.

Prior Query-Based Frame Selection

The paper groups existing query-aware selection methods into a common three-step template (Section 2.2):

  1. Candidate generation: uniformly sample candidate frames from the video, or segment the video into chunks and sample one per chunk.
  2. Relevance scoring: assess each candidate's relevance to the query, typically using CLIPScore (cosine similarity between CLIP embeddings of the frame and query text), object detectors, or learned relevance models.
  3. Selection: apply an algorithm (Top-K, thresholding, or more complex optimization) to pick the most relevant frames.

The paper identifies three specific limitations in this template:

Limitation 1: Static candidate generation. By sampling candidates uniformly or at a fixed frame rate, prior methods inherit the same information-sparsity-vs-redundancy dilemma that plagues naive uniform sampling. "Low-rate sampling may yield a sparse representation that misses critical events, while high-rate sampling produces a large and redundant frame set" (Section 4.2). This is the motivation for the paper's Content-Adaptive Frame Selection (CAFS), which generates candidates based on the video's own semantic structure—identifying scene boundaries and sampling representatively from each segment—rather than blindly at fixed intervals.

Limitation 2: Shallow relevance metrics. Methods relying on CLIPScore or object detectors "are often severely constrained by mere surface-level feature matching and reliance on fixed vocabularies, which fundamentally limits their ability to capture complex contextual reasoning and broader world knowledge" (Section 4.3). This is a well-documented problem: CLIP embeddings encode coarse semantic similarity but cannot reliably distinguish between "a man holding a red object that is relevant to the question" and "a man holding a red object that is irrelevant." Similarly, object detectors can locate "car" or "person" but cannot assess whether the presence of that entity actually helps answer the query. The paper addresses this by using the LMM itself as the relevance assessor, leveraging its reasoning capabilities rather than relying on similarity metrics.

Limitation 3: Sparse frame selection loses temporal continuity. Most prior methods select individual, isolated frames rather than continuous segments. The paper argues this is problematic because "queries, particularly those involving 'why' or 'how', cannot be fully addressed by a single frame" (Section 4.3). Actions unfold over time; causal relationships span multiple frames; context before and after an event is often essential. By selecting only sparse keyframes, prior methods discard the temporal context that surrounding frames provide. The paper addresses this through its video refinement step, which takes selected representative frames and expands them back into continuous video segments for final sampling.

Specific Baselines: AKS and Q-Frame

The paper evaluates against two representative recent methods:

AKS (Adaptive Keyframe Sampling) samples candidate frames at 1 fps, computes frame-query similarity via BLIP (a bootstrapped vision-language model), and applies an adaptive selection algorithm. The paper's experiments (Table 1) show that AKS can outperform uniform sampling at moderate frame counts (e.g., +4.5% on MLVU at 32 frames with Qwen2.5-VL-7B) but degrades at high frame counts—for example, underperforming uniform sampling by 1–2% on LongVideoBench at 128 frames. The paper attributes this to the static candidate sampling and the limitations of BLIP as a relevance scorer.

Q-Frame uses a fixed candidate pool of 128 frames and applies CLIPScore-based selection with a fixed frame count strategy. The paper notes that Q-Frame is "restricted to budgets of {8, 16, 32, 64, 128}" because of this hard limit on the initial candidate pool (Appendix F.1). More critically, Q-Frame fails to consistently outperform uniform sampling: "Q-Frame fails to outperform uniform sampling as frame counts exceed 32" (Appendix G.1). At 128 frames on Qwen2.5-VL-7B, Q-Frame achieves lower accuracy than uniform sampling across multiple benchmarks.

These failures at scale are important because they demonstrate that the challenge is not just selecting query-relevant frames—it is doing so without introducing harmful selection bias, redundancy, or over-pruning as the frame budget grows. The paper's ability to maintain gains at 256 and 768 frames (Appendix F, Table 5) distinguishes DIG from methods that work only in low-frame regimes.

How This Paper Positions Itself

The paper positions itself not as proposing an entirely new paradigm, but as identifying and exploiting a fundamental contingency that prior work overlooked: the dependence of optimal frame selection strategy on query type (Section 1, Section 3).

This positioning is articulated through several key arguments:

The negative result is as important as the positive one. The paper explicitly demonstrates that for global queries, "deploying more complex selection methods is often inefficient and yields diminishing returns" (Section 1). This is not a failure mode to be fixed—it is a finding that motivates the paper's dual-path architecture. By recognizing that uniform sampling is already optimal for a substantial fraction of queries (global ones), DIG avoids unnecessary computation and avoids the risk of over-selecting—a form of premature information filtering that could discard context needed for holistic understanding.

Query typology is a design principle, not an afterthought. Rather than treating query type as a post-hoc analysis dimension, the paper makes it the routing mechanism at the architecture's entry point (Section 4.1). An LLM classifies the query first; this classification determines whether the video undergoes expensive query-aware processing or cheap uniform sampling. This "classify-then-route" design is the paper's central architectural contribution, and it is what enables the framework to be both effective (for localized queries) and efficient (for global queries).

Training-free by design. The paper repeatedly emphasizes that DIG is training-free (Section 1, Section 4, Section 7). This is a deliberate positioning choice: in a landscape where many video understanding improvements come from fine-tuning or architectural modifications, DIG can be applied to any existing LMM without retraining, using off-the-shelf components (DINOv2 for features, an LLM for query classification, the LMM itself for reward assignment). This makes it a drop-in module compatible with any video LMM, which the paper demonstrates across Qwen2.5-VL-7B, Qwen2.5-VL-32B, and Qwen3-VL-8B.

Scalability is a first-class evaluation criterion. By testing at frame counts up to 256 (and 768 in appendix experiments), the paper positions itself against a specific failure mode of prior work: methods that work at 32 frames but collapse at realistic deployment scales. The experiments in Table 5 (Appendix F) with Qwen3-VL-8B at 768 frames—where DIG maintains +4.7% on MLVU, +3.7% on LongVideoBench, and +3.5% on VideoMME-Medium over uniform sampling—are particularly compelling because they show the method continues to provide value even as the base model and context lengths improve.

The framework is modular and the components are independently validated. The paper's ablation structure (Section 6) reflects a deliberate design philosophy: each component (query classification, CAFS, LMM-based reward assignment, video refinement via window length) is evaluated in isolation so that its contribution is clear and its failure modes are understood. This is important because it allows future work to improve or replace individual components without discarding the overall architecture. For instance, the finding that LMM-based reward assignment consistently outperforms CLIPScore (Section 6.3, Table 2) is a modular result that applies beyond DIG to any frame selection method that needs relevance scoring.

In sum, the paper positions itself at the intersection of recognizing a previously overlooked query-type dependency in frame selection, proposing a modular and training-free architecture that exploits this dependency, and validating that the approach scales to frame budgets where prior methods fail. The framing is deliberately practical—this is not a new model architecture or training paradigm, but a inference-time framework that can be applied to existing systems to improve their video understanding capabilities without retraining.

3. Technical Approach

3.1 Reader Orientation

System built: DIG is a training-free, modular inference pipeline that sits between a long-form video and a Large Multimodal Model (LMM), deciding which frames to feed to the LMM based on the user's question. Problem and solution shape: The core problem is that no single frame-selection strategy works well for all types of video questions — uniform sampling excels at capturing holistic context but fails at pinpointing specific moments, while query-aware search finds relevant moments but is computationally expensive and can harm global understanding. DIG solves this by acting as a decision router: it first classifies the incoming question as either global or localized, then branches into the appropriate processing path — cheap uniform sampling for global queries, and a multi-stage visual search pipeline for localized ones.

3.2 Big-Picture Architecture (Diagram in Words)

The DIG framework consists of five major components connected in a branching pipeline (Figure 4):

  1. Query Classifier — Takes the user's natural-language question as input and, using an LLM with chain-of-thought reasoning, outputs a binary label: isGlobal: true or isGlobal: false. This is the routing decision that determines which branch the video processing follows.

  2. Uniform Sampling Branch (for Global queries) — If the query is classified as global, the video is uniformly sampled at fixed intervals across its entire duration, producing exactly N frames. These frames are fed directly to the downstream LMM for answer generation. No additional processing occurs.

  3. Content-Adaptive Frame Selection (CAFS) — If the query is classified as localized, the video is first sampled at 2 fps. Each frame is embedded using DINOv2, and pairwise cosine distances between consecutive frames are computed to detect scene boundaries. A topographic prominence filter selects the most significant boundaries, and the midpoints between these boundaries become r-frames — a compact set of semantically representative frames that summarize the video's visual content.

  4. LMM Reward Assigner — Each r-frame is presented individually to the same (or a different) LMM, along with the original question, the video duration, and the frame's timestamp. The LMM generates a structured output containing a description of the frame's relevance and a reward score (0–100) that rates both the frame's direct utility and whether adjacent frames are likely to contain useful context.

  5. Video Refinement Module — Taking the r-frame indices, the peak boundary indices, and their assigned rewards, this module applies an iterative reward-guided selection algorithm to identify the most query-relevant r-frames. For each selected r-frame, it expands the selection outward by a temporal window wlen to form continuous video segments. The union of these segments becomes a refined video — a shortened version of the original video concentrated on query-relevant temporal regions. The final N input frames are then uniformly sampled from this refined video and passed to the downstream LMM.

Information flows sequentially through these components for localized queries: raw video → CAFS → r-frames + boundaries → LMM reward assignment → rewards → video refinement → refined video → uniform sampling → LMM inference. For global queries, the path is simply: raw video → uniform sampling → LMM inference.

3.3 Roadmap for the Deep Dive

  • First, the compute-optimal allocation formulation (Equation 1), which defines what "optimal" means and why difficulty is the key variable.
  • Second, the difficulty estimation mechanism, since it is the linchpin that enables adaptive allocation and is shared across both the search and revision pipelines.
  • Third, the PRM verifier — how it is trained, how it scores solutions, and how scores are aggregated — since all search methods depend on it.
  • Fourth, the three search algorithms (best-of-N, beam search, lookahead search), their mechanics, their cost model, and their difficulty-dependent behavior.
  • Fifth, the revision model — how it is trained, how it generates sequential revisions, and how sequential and parallel sampling are combined.
  • Sixth, the FLOPs-matched comparison framework that enables the pretraining-vs-inference tradeoff analysis.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that the optimal frame selection strategy for video question answering is conditional on the query's scope — whether it targets the whole video or specific moments — and that a routing-based architecture exploiting this contingency can achieve both stronger performance and better efficiency than approaches that use a single strategy for all queries.

The Query Typology: Global vs. Localized Queries

Before examining the technical mechanisms, we must understand the two query categories that drive DIG's routing decision. The paper defines these categories in Section 3, validating them through both human annotation and empirical performance analysis.

Global queries (GQ) are questions that require understanding the video in its entirety. A canonical example from the paper is: "What title best summarizes this video?" or "What is the video mainly about?" These queries ask about overarching themes, narratives, topics, or the primary focus of the complete video. The key property is that answering them does not require focusing on any specific temporal segment — the relevant information is distributed across the entire video timeline, and excluding any substantial portion risks missing context needed for a correct answer.

Localized queries (LQ) are questions whose answers can be found by identifying and analyzing specific segments of the video. The paper gives examples such as "What kind of bike is the man riding?" and "How does the cartoon sponge walk?" These queries contain referents — specific entities, actions, or events that anchor the question to particular temporal locations. The key property is that frames outside the relevant segments are not just unhelpful but actively harmful: they introduce noise that makes it harder for the model's attention mechanism to focus on the evidence-bearing frames.

The paper validates this typology empirically in Figure 3. Using manually annotated labels on MLVU and VideoMME (described in Appendix B), the authors evaluate three LMMs (Qwen2.5-VL-7B, InternVL3-8B, LLaVA-OneVision-7B) and plot accuracy deviation from a baseline as frame count increases, separately for GQ and LQ. The result is striking: global query performance remains relatively flat as frames increase (with slight upward trends in some cases), while localized query performance drops substantially. For Qwen2.5-VL-7B on MLVU, LQ accuracy drops by approximately 6 percentage points when moving from the lowest to highest frame count, while GQ accuracy shifts by less than 1 point. The mechanism is clear: adding more uniformly sampled frames to a localized query dilutes the signal-to-noise ratio, since the proportion of frames from the relevant temporal segment shrinks as the total frame count grows.

This typology is not simply a post-hoc categorization — it is the operational backbone of DIG's architecture. The system's first action upon receiving a query is to classify it as global or localized (Section 4.1), and this classification determines the entire downstream processing path.

Query Type Identification (Section 4.1)

The classifier that performs this routing decision is an LLM prompted with a carefully structured chain-of-thought reasoning template. The full prompt is provided in Figure 11 of Appendix C. The LLM is instructed to:

  1. Understand the query's core intent — read the query and grasp its general meaning.
  2. Infer the video's hypothetical style — based on the query's phrasing, reason about what type of video is likely being asked about (narrative film, educational lesson, documentary, etc.).
  3. Identify specific referents — entities (persons, objects), actions, events, or specific pieces of information mentioned in the query, adjusting what counts as a referent based on the inferred video style.
  4. Evaluate referents in context — apply a two-part decision rule:

The decision rule is:

  • Global if: the query lacks a specific referent (e.g., summary-based queries like "primary focus," "in summary," "what is the video about?"); OR the query has a referent but answering still requires holistic understanding (e.g., "what is the boy's overall role?").
  • Localized if: the query has specific referents AND the answer can be found by focusing on specific related segments where they appear (e.g., entity-based referents like "the person in the red shirt," action-based referents like "what is X doing," temporal referents like "at the beginning," "after the explosion").

The LLM outputs a JSON object containing its step-by-step analysis and a boolean field: "isGlobal": true for global queries, "isGlobal": false for localized queries.

Model choice and accuracy. The paper uses Qwen3-Next-80B-A3B-Instruct as the classifier in main experiments (Section 5.1) and compares several alternatives in Appendix D (Table 4). Classification accuracy against human annotations is measured on MLVU, LongVideoBench, and VideoMME. The key results:

  • Overall accuracy exceeds 80% for nearly all evaluated LLMs across all benchmarks.
  • Localized query accuracy is consistently higher than global query accuracy. For Qwen3-Next-80B-A3B-Instruct: LQ accuracy is 87.02% (MLVU), 97.53% (LongVideoBench), 89.13% (VideoMME); GQ accuracy is 38.26% (MLVU), N/A (LongVideoBench has no GQ), 65.76% (VideoMME).
  • The lower GQ accuracy is explicitly acknowledged as acceptable by the paper because "this has a negligible impact on final model performance; it primarily incurs a minor computational overhead" — when a global query is misclassified as localized, it undergoes unnecessary processing, but the performance difference between the two processing paths is minimal for global queries (as demonstrated in Figure 5, right panels). The critical metric is LQ accuracy, which exceeds 90% for most models.

Design rationale for using an LLM rather than a simpler classifier. The paper does not use a rule-based classifier (e.g., keyword matching for "summary," "overall," "entire video") because query phrasing is diverse and referents can be ambiguous. A query like "What is the boy's overall role?" contains both a referent ("the boy") and a holistic term ("overall role"), requiring reasoning about which aspect dominates. Similarly, "What happens after the explosion?" contains a temporal anchor ("after the explosion") but might require understanding events distributed across a significant portion of the remaining video. The chain-of-thought reasoning in the LLM prompt is designed to handle these edge cases by making the reasoning explicit: the model must state why it believes the query requires holistic understanding, not just output a label.

Efficiency motivation. The query classifier is the efficiency linchpin of DIG. Table 11 in Appendix G quantifies the savings: on VideoMME, applying the expensive query-aware pipeline universally (without query identification) costs 384.2 minutes; with query identification routing global queries to cheap uniform sampling, the cost drops to 307.9 minutes — a 19.9% reduction. On MLVU, the savings are 13.3%. These savings come from skipping the CAFS + LMM reward assignment + video refinement stages for queries where those stages provide no benefit. On LongVideoBench, where 97.8% of queries are localized, the query identification step adds a marginal 3.6% overhead (from the classification computation itself), but the overhead is trivial compared to the gains from targeted frame selection.

Content-Adaptive Frame Selection (CAFS) — Technical Mechanics (Section 4.2, Appendix E)

CAFS is the first stage of the localized-query pipeline and addresses a fundamental problem: before we can select query-relevant frames, we need a manageable set of candidate frames to evaluate. Exhaustively scoring every frame in a long video (which could contain thousands of frames) is computationally infeasible. Prior methods generate candidates through uniform sampling or fixed-rate sampling, which creates an inescapable tradeoff: sparse sampling risks missing critical events that occur between samples; dense sampling produces a redundant candidate set where many frames are near-duplicates.

CAFS resolves this by making the candidate selection content-adaptive — it uses the video's own visual structure to determine how many frames to sample and where, rather than imposing a fixed schedule.

Step 1: Feature extraction and distance computation. The video is first sampled at 2 frames per second to produce a sequence of M frames with their temporal indices {Ii}ⁱᴹᵢ₌₁. Each frame f_{Ii} is processed through DINOv2 (Oquab et al., 2024) to extract a robust visual feature vector V_{Ii}. DINOv2 is chosen because it produces features that are semantically meaningful (capturing objects, scenes, and high-level visual content) while being robust to low-level variations like lighting changes or small camera motions. The paper does not fine-tune or adapt DINOv2 — it is used as a frozen feature extractor.

For each consecutive pair of frames, a dissimilarity score d_i is computed:

di=1−sim(VIi,VIi+1)d_i = 1 - \text{sim}(V_{I_i}, V_{I_{i+1}})

where $\text{sim}(\cdot, \cdot)$ denotes cosine similarity (the dot product between L2-normalized feature vectors, producing a value in $[-1, 1]$ where 1 means identical direction, 0 means orthogonal, and -1 means opposite direction). This produces a sequence of distances $\{d_i\}^{M-1}_{i=1}$.

What it computes: For each pair of temporally adjacent frames (at 0.5-second intervals given the 2 fps sampling), the cosine distance between their DINOv2 embeddings. A value near 0 means the frames are visually and semantically similar (same scene, same objects); a value near 1 or higher (the maximum is 2, when vectors are opposite) means a significant visual change has occurred — potentially a scene cut, a major camera movement, or the appearance/disappearance of large objects.

Why this form: Cosine distance is chosen over alternatives like Euclidean distance because it is invariant to the overall magnitude of the feature vector, focusing only on the direction. This matters because DINOv2 features can vary in magnitude depending on image complexity, but scene similarity is better captured by the angular relationship. Using raw cosine similarity $\text{sim}$ and computing $1 - \text{sim}$ converts the similarity (where 1 = identical) into a distance (where 0 = identical), making the subsequent peak detection more intuitive.

Step 2: Peak detection with topographic prominence filtering. The distance sequence {d_i} is scanned for local maxima (peaks): index i is a peak if d_{i-1} < d_i and d_i > d_{i+1}. These peaks correspond to frames where the visual content changes abruptly relative to their immediate neighbors — typically scene boundaries, camera cuts, or major content transitions.

However, raw peak detection produces many spurious peaks from minor fluctuations (e.g., a character moving slightly, a brief occlusion). To filter these, CAFS applies topographic prominence filtering. For each candidate peak at index j, the algorithm:

  1. Finds the lowest distance value to the left (lmin): starting from the peak, move leftward, tracking the minimum distance encountered, stopping only when a distance larger than the peak's height d_j is found. The lowest point in this left valley is lmin.
  2. Finds the lowest distance value to the right (rmin): symmetric procedure moving rightward.
  3. Computes prominence: p_prom = d_j - max(lmin, rmin).

The prominence quantifies how much the peak "stands above" the surrounding landscape. A peak that is only slightly higher than its neighbors has low prominence; a sharp spike in the distance sequence (indicating a genuine scene change) has high prominence. Peaks with prominence greater than 0.1 are retained. This threshold is stated to "have been found effective through empirical observation" (Section 4.2).

The indices of valid peaks are denoted {K_j}ⱼᴺ_{j=1}, where N is the number of detected boundaries and N < M (typically much smaller — a video might have hundreds or thousands of frames but only dozens of scene changes).

Why prominence filtering rather than simpler thresholding: Simple thresholding (keeping peaks where d_i > τ) would fail to adapt to videos with different baseline visual dynamics. A fast-paced action sequence might have consistently high pairwise distances, where even small fluctuations exceed a fixed threshold; a static lecture video might have consistently low distances where even genuine scene changes barely exceed a modest threshold. Prominence is self-normalizing — it measures how exceptional a peak is relative to its local context, not relative to an absolute scale. This makes it robust to the overall visual dynamics of different video genres.

Step 3: R-frame selection from between-boundary midpoints. The validated peaks {K_j} partition the video into segments — the interval [K_j, K_{j+1}] represents a continuous span of visually consistent content (frames with low pairwise distances between boundaries). From each segment, CAFS selects exactly one representative frame (r-frame) to serve as the segment's content summary. The r-frame index is:

Ij′=Kj+Kj+12I'_j = \frac{K_j + K_{j+1}}{2}

for j = 1, 2, ..., N-1. This is the temporal midpoint between consecutive peaks.

Why the midpoint and not the first or last frame of the segment: The frames near boundaries may contain transitional content — a partially faded scene, an object entering or leaving — that is atypical of the segment as a whole. The midpoint is most likely to capture the stable, characteristic content of the segment. This is a simple heuristic but well-motivated by the structure of edited video, where cuts typically separate distinct shots and the middle of a shot is most representative.

Output characteristics. The CAFS output is a set of r-frame indices {I'_j}^{N-1}_{j=1} and the boundary indices {K_j}ⱼᴺ_{j=1}. The number of r-frames N-1 is entirely determined by the video's content — videos with many scene changes produce many r-frames; static videos produce few. Figure 10 (Appendix E.2) shows this non-linearity: on MLVU, videos in the 0–10 minute range average 105.3 r-frames, while videos in the 10–20 minute range average 180.8 r-frames (despite being only moderately longer, with average durations of 7.1 vs. 12.7 minutes). This confirms that "the r-frame count does not scale linearly with video duration" — information density matters more than raw length.

The paper validates CAFS against uniform sampling and fixed-FPS sampling using two metrics (Section 6.2, Figure 6):

  • Localized Coverage (LoC): For each r-frame f_{I_i}, four neighboring frames are sampled uniformly from the surrounding temporal window. LoC is the average cosine similarity between each r-frame and its four neighbors, averaged across all r-frames. Higher LoC means r-frames are more representative of their local temporal context.

LoC=14N∑i=1N∑j=03sim(VIi,VMi,j)LoC = \frac{1}{4N} \sum_{i=1}^{N} \sum_{j=0}^{3} \text{sim}\left(V_{I_i}, V_{M_{i,j}}\right)

where $M_{i,j} = I_i + (j - 1.5) \cdot \lfloor (I_{i+1} - I_{i-1})/6 \rfloor$. The neighbor indices are spaced to cover the segment evenly.

  • Global Coverage (GlC): 200 frames are randomly sampled from the video. For each sampled frame, the maximum cosine similarity to any r-frame is computed. GlC is the average of these maxima. Higher GlC means the r-frames collectively cover the visual diversity of the entire video well.

GlC=1∣X∣∑x∈Xmax⁡i∈[1,N]sim(VIi,Vx)GlC = \frac{1}{|X|} \sum_{x \in X} \max_{i \in [1, N]} \text{sim}(V_{I_i}, V_x)

where X is the set of 200 randomly sampled frames.

On both metrics, CAFS outperforms uniform sampling and fixed-FPS sampling, especially for videos over 10 minutes. The advantage widens with duration because uniform sampling with a fixed frame budget becomes increasingly sparse for long videos, while CAFS adapts its sampling density to content complexity.

Ablation within DIG (Figure 7). When CAFS's r-frames are replaced with standard uniformly sampled frames in the DIG pipeline (keeping all other components identical), performance drops across all benchmarks. The drop widens with more input frames, confirming that uniform candidate generation becomes increasingly inadequate as the final frame budget grows.

LMM-Based Reward Assignment (Section 4.3)

Once CAFS has produced a manageable set of r-frames {I'_j}^{N-1}_{j=1}, the next stage evaluates each r-frame's relevance to the specific query. This is the relevance scoring step that determines which r-frames (and by extension, which temporal segments) deserve further attention.

The two-dimensional reward criterion. The paper designs a reward prompt (Appendix C, Figure 12) that asks the LMM to consider two factors jointly:

  1. Direct usefulness: How directly does this frame help answer the query? Can the question be partially or fully resolved from the visual content of this single frame?
  2. Contextual adjacency: Does the frame suggest that adjacent frames (immediately before or after) contain supplementary information needed for a complete answer? This acknowledges that many queries — especially "why" and "how" questions — require information spanning multiple frames, and a frame that only partially addresses the query can still be valuable if it indicates where the complete evidence is located.

The LMM receives the video duration, the specific frame timestamp, and the query text. It is instructed to:

  1. Describe the frame's visual content, focusing on elements relevant to the query.
  2. Assign a reward score between 0 and 100 based on the two-dimensional criterion.

The output format is JSON: {"description": str, "reward": int}.

Why LMM-based scoring over CLIPScore. The paper explicitly argues against using CLIPScore (cosine similarity between CLIP embeddings of frame and query text) or object detectors for relevance assessment. The fundamental limitations are:

  • Surface-level feature matching: CLIPScore captures broad semantic alignment (e.g., "this frame contains a person and the query is about a person") but cannot determine whether the specific person in the frame is the one relevant to the query. A query about "the man in the red shirt" matched against a frame containing "a man in a blue shirt" may still receive a moderately high CLIPScore because both contain a person — a false positive that misleads downstream selection.
  • Fixed vocabulary: Object detectors can only recognize categories in their training set. If the query asks about a specific object, action, or visual relationship not covered by the detector's vocabulary (e.g., "how does the cartoon sponge walk?" — "walk" is an action, "cartoon sponge" is not in standard object taxonomies), the detector provides no useful signal.
  • No reasoning about utility: Even when CLIPScore or a detector correctly identifies that the frame contains query-relevant entities, they cannot assess informational utility. A frame showing the relevant person from behind or in poor lighting may match the query text but provide no useful information for answering it. The LMM, through its instruction tuning, can reason about whether the visual content actually helps.

Better LMMs yield better rewards (Table 2). The paper tests this by comparing rewards from Qwen2.5-VL-32B (larger) against Qwen2.5-VL-7B (smaller) and CLIPScore, using Qwen2.5-VL-7B as the final inference model in all cases. Results in Table 2 show:

  • At 128 frames: Qwen2.5-VL-32B rewards achieve 72.6% on MLVU, vs. 71.4% for Qwen2.5-VL-7B and 69.6% for CLIPScore.
  • At 256 frames: Qwen2.5-VL-32B achieves 74.3%, vs. 72.5% (7B) and 71.2% (CLIPScore).

The consistent advantage of the larger LMM as reward assigner — even though the inference model is the same — demonstrates that more capable models provide more precise relevance signals. This also validates a key architectural choice: the reward assignment LMM and the inference LMM can be different models. The paper notes this enables a "decoupled" setup where a stronger but more expensive model evaluates frame relevance, while a cheaper model performs the final answer generation. This is practically important because reward assignment processes frames one at a time (linear cost in number of r-frames) rather than the quadratic attention cost of processing all frames together.

Computational cost of reward assignment. Each r-frame receives one LMM forward pass for reward scoring. For a video with 200 r-frames, this means 200 separate LMM calls. The paper reports in Table 10 (Appendix G) that for MLVU, the reward assignment stage takes 218.9 minutes — the dominant component of DIG's overhead. The query identification and CAFS stages are relatively cheap (11.3 and 25.9 minutes, respectively). This cost structure means that improving CAFS to reduce the number of r-frames (by more aggressive filtering) would directly reduce the dominant cost, making it a natural target for future optimization.

Video Refinement (Section 4.4)

The final stage of the localized-query pipeline takes the r-frame indices {I'_j} and their associated rewards {R_j} and constructs a refined video — a shortened version of the original video that concentrates on query-relevant temporal regions. The refined video is then uniformly sampled to produce the final N input frames for the LMM.

This stage has two sub-steps: iterative reward-guided selection (which chooses which r-frames to keep) and segment combination (which expands each selected r-frame into a continuous temporal segment).

Iterative reward-guided selection.

The paper argues against Top-K selection (keeping the K r-frames with highest rewards) because it requires choosing a hyperparameter K that is inherently video- and query-dependent. Instead, DIG uses a parameter-free iterative thresholding procedure:

Initialize: Start with the reward set {R_j}^{N-1}_{j=1} from the LMM reward assigner.

Iterate until convergence:

  1. Compute the mean: $\bar{R} = \frac{1}{N-1} \sum_{j=1}^{N-1} R_j$ — the average reward across all r-frames.
  2. Threshold below the mean: For each r-frame, compute a new reward value by subtracting the mean and clamping at zero:

Rj′=max⁡(Rj−Rˉ,0)∀j=1,…,N−1R'_j = \max(R_j - \bar{R}, 0) \quad \forall j = 1, \ldots, N-1

This sets any r-frame whose reward is below the current mean to zero (removing it from consideration) and reduces the rewards of remaining r-frames by the mean (lowering their values relative to the new, higher mean that will be computed in the next iteration).

  1. Check convergence: Let S be the set of indices where R'_j > 0 (the r-frames that survived this iteration). If S is identical to the set from the previous iteration, terminate. Otherwise, set {R_j} = {R'_j} and go back to step 1.

Upon termination, the selected r-frames are those with positive final rewards: I_f = {I'_j \mid R'_j > 0}.

What this procedure does, operationally: It is an iterative elimination process where, in each round, r-frames with below-average relevance (relative to the current set of survivors) are discarded, and the mean is recomputed on the survivors. Because the mean of the survivors will be higher than the mean of the original set, some previously above-average frames may now fall below the new mean. The process continues until the survivor set stabilizes — meaning all remaining r-frames have rewards above the mean of the survivor set.

Why this specific form matters:

  • The $\max(R_j - \bar{R}, 0)$ operation implements a soft threshold at the mean. Frames exactly at the mean become 0 (eliminated); frames above the mean retain their relative advantage; frames below the mean are eliminated. This is more principled than a fixed threshold because the mean adapts to the reward distribution of each video-query pair.
  • The iterative nature handles multimodal reward distributions. If a video has two clusters of relevant segments (e.g., the query involves two events that occur at different times), the initial mean might be between the clusters. The first iteration eliminates the below-mean cluster; the second iteration recomputes the mean on the surviving cluster and may further prune marginal frames within that cluster. This adaptivity cannot be achieved with a single-pass threshold.
  • The convergence criterion ensures the procedure is self-terminating — no hyperparameter for the number of iterations is needed.

Segment combination.

Simply feeding the selected r-frames as-is to the LMM would discard fine-grained temporal information. Each r-frame is a single still image; actions, causal sequences, and events that unfold over multiple frames would be lost. To address this, DIG expands each selected r-frame into a continuous video segment.

For a selected r-frame indexed by I'_j, the natural temporal segment is the interval between the two surrounding scene boundaries: [K_j, K_{j+1}]. This segment corresponds to a visually consistent shot or scene — the frames between these boundaries are highly similar to the r-frame (as confirmed by the low pairwise distances that CAFS used to detect the boundaries in the first place).

Additionally, DIG incorporates a temporal context window of length wlen on each side. The expanded segment for r-frame I'_j spans [K_{j-\text{wlen}}, K_{j+\text{wlen}+1}]. With wlen = 2 (the default used in experiments), the window extends two r-frames before and after the selected one, incorporating the segments corresponding to those neighboring r-frames as well. This captures context like setup actions before an event, reactions after an event, or multi-shot sequences that span several scene boundaries.

The final refined video is the union of all these expanded segments. Because segments may overlap (when selected r-frames are close together), the union operation merges them, producing contiguous blocks of query-relevant video rather than fragmented pieces. From this refined video, the final N frames are uniformly sampled and passed to the LMM.

Why expansion rather than direct r-frame input:

  • Temporal continuity matters for reasoning. Many queries, especially "why" and "how" questions, require understanding causal or procedural sequences that cannot be inferred from isolated keyframes. The expanded segments preserve the temporal flow, enabling the LMM to observe actions unfolding, transitions occurring, and contextual cues before and after key moments.
  • Fine-grained details are preserved. A single r-frame might show the start of an action; the surrounding frames show its completion. If only the start frame is included, the model must guess the outcome. Including the continuous segment gives the model access to all the evidence.
  • The union operation avoids fragmentation. By merging overlapping segments, the refined video is a set of coherent, query-relevant temporal blocks rather than scattered isolated slices. This makes the final uniform sampling step more effective, since frames will be drawn from a continuous region of interest rather than from disjoint points.

Window length ablation (Figure 8). The paper tests wlen ∈ {0, 2, 4, 8} and finds:

  • wlen = 0 (selected r-frames only, no expansion) yields the lowest performance across all benchmarks. This confirms that isolated keyframes are insufficient.
  • wlen = 2 achieves the best results on most benchmarks. It strikes a balance between including useful context and avoiding irrelevant noise.
  • wlen = 8 performs worse than wlen = 2 and wlen = 4 on most benchmarks. The paper attributes this to "an excessively large window introduces irrelevant contextual information, creating noise that is detrimental to localized queries." This validates a central premise: for localized queries, expanding too far from the relevant segment defeats the purpose of targeted selection by reintroducing irrelevant frames.

The Final Inference Step

For localized queries, the pipeline produces a refined video — a temporally compressed version of the original, concentrated on query-relevant segments. The final N input frames are uniformly sampled from this refined video:

A=LMM([f1′;f2′;…;fN′;Q])A = \text{LMM}([f'_1; f'_2; \ldots; f'_N; Q])

where {f_i'}^{N}_{i=1} are the uniformly sampled frames from the refined video, Q is the query text, and A is the auto-regressively generated answer. The LMM processes these frames identically to how it would process uniformly sampled frames from the original video — no architectural modifications, no fine-tuning, no special tokens. The only difference is which frames are presented.

For global queries, where the query classifier assigns isGlobal: true, the pipeline is simply: uniformly sample N frames from the entire original video and pass them to the LMM. No CAFS, no reward assignment, no video refinement.

Contrast with baselines. The paper explicitly contrasts this final sampling from the refined video with prior methods that feed the selected frames directly. By using uniform sampling from a refined video rather than direct frame selection, DIG preserves the benefits of uniform temporal coverage within the region of interest while eliminating frames from irrelevant regions. This is a subtle but important design choice: uniform sampling within the refined video ensures the LMM sees frames distributed across the relevant temporal span, rather than clustered or gapped in ways that could mislead temporal reasoning.

Design Summary and Key Hyperparameters

Fixed hyperparameters used in experiments (Section 5.1):

  • Video sampling rate for CAFS: 2 fps
  • DINOv2 model: frozen, no fine-tuning, used only for feature extraction
  • Topographic prominence threshold: 0.1
  • Window length for segment expansion: wlen = 2
  • Frames per input to LMM: 56 tokens each
  • LLM for query classification: Qwen3-Next-80B-A3B-Instruct (main experiments)
  • LMM for reward assignment: Qwen2.5-VL-32B or Qwen2.5-VL-7B (depending on experiment)
  • LMM for final inference: Qwen2.5-VL-7B or Qwen2.5-VL-32B (matching or decoupled)
  • Hardware: 8 NVIDIA A100 GPUs, vLLM backend for accelerated inference

Why training-free matters. Every component in DIG — DINOv2 for features, the LLM for classification, the LMM for reward assignment and final inference — is used off-the-shelf without fine-tuning. This is an explicit design choice that ensures DIG is a plug-and-play module compatible with any video LMM, independent of its training data, architecture, or fine-tuning regime. The paper demonstrates this compatibility by testing with three different LMM families (Qwen2.5-VL, InternVL3-8B in preliminary experiments, Qwen3-VL-8B in appendix) across three benchmarks without any per-model adaptation.

Why routing rather than ensembling. An alternative design would be to always run both paths (uniform sampling and query-aware selection) and ensemble or fall back based on confidence. The paper's routing design is simpler and computationally cheaper — global queries save the entire cost of the localized pipeline, which Table 10 shows is substantial (256.3 minutes for MLVU). The key empirical finding that justifies this design is Figure 5 (right panels): for global queries, the performance curves for uniform sampling and the query-aware pipeline are essentially overlapping across all frame counts, meaning there is no accuracy cost to routing global queries to the simpler path. This is the paper's central negative result, and it is what makes the routing architecture Pareto-optimal: it achieves the performance of the best method for each query type at the cost appropriate to that type.

4. Key Insights and Innovations

Innovation 1: The Query Typology as a Routing Principle, Not a Post-Hoc Observation

The paper's most fundamental intellectual contribution is not the mechanism of CAFS or the reward model — it is the empirically validated claim that the optimal frame selection strategy is conditional on query scope, and that this contingency is strong enough to serve as the architectural organizing principle for an entire inference pipeline.

Prior work — both the uniform sampling baseline and query-aware methods like AKS, Q-Frame, and Bolt — implicitly assumes that a single frame selection strategy should be applied to all queries. The choice of strategy is treated as a system-level design decision: you either optimize for temporal coverage (uniform sampling) or for query relevance (adaptive selection), and you live with the tradeoffs. The paper challenges this assumption at its root, demonstrating that the tradeoffs themselves are query-dependent. For global queries, uniform sampling is not a "good enough" baseline that could be improved with more sophisticated methods — it is the correct approach, and adding query-aware selection adds cost without benefit (and sometimes with harm). For localized queries, uniform sampling is not merely suboptimal — it is actively destructive at scale, with performance degrading as more frames are added because the signal-to-noise ratio collapses.

This is a reframing of the frame selection problem from "what is the best strategy?" to "under what conditions is each strategy best?" — a move from optimization over strategies to optimization over the strategy-selection function itself. It parallels the shift in machine learning from training a single model to training a meta-learner or router, but applied at the inference-time pipeline level for video understanding. The field has long known intuitively that different questions require different kinds of visual evidence; this paper is the first to operationalize that intuition as a pre-classification step that gates computationally expensive processing.

Evidence anchoring the claim: Figure 3's difficulty-bin analysis shows the performance divergence between global and localized queries under uniform sampling — localized query accuracy drops by approximately 6 percentage points for Qwen2.5-VL-7B on MLVU as frame count increases, while global query accuracy shifts less than 1 point. Figure 5's right panels then demonstrate that for global queries, the uniform sampling and query-aware pipeline curves overlap almost perfectly, confirming that the expensive pipeline provides no benefit. This dual finding — harm for one type, zero benefit for the other — is what justifies the routing architecture as Pareto-optimal: you get the best of both worlds at the cost appropriate to each.

The significance extends beyond this specific paper: the query typology is a diagnostic lens that future work can use to evaluate frame selection methods. Rather than reporting aggregate benchmark scores that conflate global and localized queries, researchers can and should report per-type performance. The paper's appendices model this, with per-task breakdowns (Tables 6–8) showing that gains are concentrated in perception-heavy localized tasks (PlotQA, NeedleQA) while temporal reasoning tasks (Action Order, L2-Relation) remain stagnant regardless of method — itself a diagnostic finding about the limits of frame selection as a solution.

Innovation 2: The Negative Result That Global Queries Don't Need Query-Aware Selection

A substantial fraction of the paper's intellectual contribution is a negative result — the demonstration that for a whole class of queries (global), deploying expensive query-aware frame selection is not merely unnecessary but often counterproductive. This is a distinctive type of contribution because the field's default assumption, encoded in the very existence of methods like AKS and Q-Frame, is that more query-awareness is always better. The paper's refutation of this assumption is careful, evidence-backed, and practically consequential.

The significance of this negative result lies in what it prevents — both over-engineering and misleading research conclusions. Without it, a natural research trajectory would be to make query-aware selection ever more sophisticated (better relevance models, more complex search algorithms, learned selection policies) and evaluate on benchmarks that may be dominated by localized queries, reporting aggregate gains while silently degrading performance on global queries. The paper explicitly demonstrates this risk with the failure of baselines like AKS and Q-Frame at high frame counts (Table 1: AKS underperforms uniform sampling by 1–2% on LongVideoBench at 128 frames with Qwen2.5-VL-7B). By identifying that these failures are systematic — concentrated in global queries and in the high-frame-count regime where irrelevant frame injection accelerates — the paper provides a conceptual guardrail against this class of over-engineering.

Comparison to prior work: Prior frame selection methods (Bolt, AKS, Q-Frame, Frame-Voyager, MDP3) uniformly apply query-aware mechanisms to all queries without distinguishing global from localized. Their evaluations report aggregate performance without per-type breakdowns. This paper's contribution is the diagnostic move of separating the evaluation by query type — a move that reveals the methods' strengths are concentrated in localized queries and their weaknesses in global ones. The negative result is thus a methodological contribution as much as an empirical one: it demonstrates that per-type evaluation is necessary to understand what a frame selection method actually does.

The practical implication — that DIG routes global queries to a zero-overhead uniform sampling path, saving 13–20% of processing time (Table 11) — is the direct operationalization of this negative result. But the intellectual contribution is broader: it establishes that query-awareness has a domain of applicability, and that operating outside that domain is actively harmful. This is a conceptual boundary condition that future work must respect.

Innovation 3: Content-Adaptive Candidate Generation as a Replacement for Static Sampling

While individual elements of CAFS (DINOv2 features, peak detection, prominence filtering) are not individually novel, their combination into a content-adaptive candidate generation module represents a genuine conceptual advance over the dominant paradigm in prior query-aware frame selection.

The prior paradigm (shared by AKS, Q-Frame, Bolt, VideoAgent, and most agent-based methods) generates candidate frames through static sampling — typically uniform sampling at a fixed interval or fixed frames-per-second rate. The paper identifies the fundamental dilemma this creates (Section 4.2): "low-rate sampling may yield a sparse representation that misses critical events, while high-rate sampling produces a large and redundant frame set." This is not merely an efficiency concern — it is a structural limitation of static sampling that cannot be resolved by tuning the sampling rate, because the optimal rate depends on video-specific content density which is unknown a priori.

CAFS resolves this by making the sampling responsive to the video's semantic structure. Rather than imposing a uniform temporal grid, it detects where meaningful visual changes occur (through DINOv2 feature distances and prominence-filtered peak detection) and samples representatively from the stable segments between changes. The number of r-frames is determined by the video's content complexity, not by a fixed budget — Figure 10 shows that r-frame counts do not scale linearly with duration, with videos in the 10–20 minute range on LongVideoBench averaging 226.4 r-frames (from 17.1-minute average videos) versus 47.9 for 0–10 minute videos (3.6-minute average). This non-linearity confirms that information density varies independently of duration, and a static sampling scheme cannot adapt to this variation.

Why this is more than an engineering improvement. The paper's own ablation (Figure 7) shows that replacing CAFS r-frames with uniformly sampled frames in the DIG pipeline degrades performance across all benchmarks, with the gap widening at higher final frame counts. This demonstrates that the quality of candidate generation compounds through the pipeline: better candidates mean the reward model evaluates more informative frames, which means the video refinement step operates on a better representation of the video's structure, which means the final uniformly sampled frames are drawn from more precisely identified query-relevant regions. The whole pipeline benefits from the initial content-adaptive step in a way that is not merely additive.

The conceptual contribution is the principle of content-adaptive candidate generation: that frame selection for video understanding should first understand the video's own structure before trying to understand the query's relationship to that structure. This inverts the typical priority in query-aware methods, which often apply query relevance scoring directly to uniformly sampled frames — essentially asking "is this frame relevant?" without first asking "does this frame represent a distinct piece of the video's content?" CAFS answers the second question first, providing a structurally meaningful candidate set for the relevance scoring to operate on.

Innovation 4: LMMs as General-Purpose Relevance Assessors, Replacing Shallow Similarity Metrics

The paper's decision to use the LMM itself — rather than CLIPScore, BLIP, object detectors, or learned relevance models — as the frame-query relevance assessor is a conceptual shift that has implications beyond DIG. Prior query-aware frame selection methods rely on fixed similarity metrics (CLIPScore in Q-Frame and VideoAgent, BLIP in AKS, object detectors in T*) that compute relevance through embedding-space proximity or category matching. The paper argues these are "severely constrained by mere surface-level feature matching and reliance on fixed vocabularies, which fundamentally limits their ability to capture complex contextual reasoning and broader world knowledge" (Section 4.3).

The shift is from shallow similarity to reasoned assessment. CLIPScore can tell you that a frame containing a kitchen is similar to a query about cooking, but it cannot distinguish between "this kitchen contains the specific tool being asked about" and "this kitchen is generically similar but irrelevant." The LMM, by contrast, can describe the frame's content, identify specific entities, assess their relationship to the query, and even evaluate whether adjacent frames are likely to contain needed context — a multi-dimensional assessment that similarity metrics cannot approximate.

What makes this a genuine innovation rather than just "use a bigger model":

  • It is decoupled from the inference model. The paper explicitly demonstrates (Table 2) that using a stronger LMM (Qwen2.5-VL-32B) for reward assignment improves the performance of a weaker inference model (Qwen2.5-VL-7B). This decoupling is architecturally significant — it means that the cost of better relevance assessment can be scaled independently of the cost of final inference, enabling asymmetric deployment where a powerful but expensive model evaluates frame relevance offline or in batch, while a cheaper model handles real-time inference.

  • It exploits instruction-tuned reasoning capabilities that similarity metrics lack. The two-dimensional reward criterion (direct usefulness + contextual adjacency) requires the assessor to perform counterfactual reasoning: "does this frame suggest adjacent frames contain relevant information?" This is not a similarity computation — it is an inference about the video's structure and the query's information needs, which only a model capable of chain-of-thought reasoning can perform.

  • The finding that better LMMs yield better rewards (Table 2) has scaling-law implications. It suggests that as base LMMs improve, the reward assignment component of DIG will automatically improve without any architectural changes, since the reward model is just the LMM itself prompted appropriately. This is a form of automatic capability inheritance that similarity-metric-based methods cannot benefit from.

The paper's ablation comparing LMM-based rewards against CLIPScore (Section 6.3, Table 2) shows the gap is substantial (e.g., on MLVU at 128 frames: 72.6% with Qwen2.5-VL-32B rewards vs. 69.6% with CLIPScore vs. 71.4% with Qwen2.5-VL-7B, all using Qwen2.5-VL-7B as the inference model) and widens at higher frame counts. This confirms that the quality of relevance assessment is a bottleneck for the entire pipeline, and that scaling the assessor independently of the inference model is a viable strategy.

Innovation 5: Temporal Segment Expansion as a Corrective to Sparse Keyframe Selection

The paper's video refinement step — which takes individually selected r-frames and expands them into continuous temporal segments, then merges overlapping segments via union — addresses a systematic weakness in prior keyframe selection methods that the paper diagnoses as temporal fragmentation. Most prior query-aware methods (AKS, Q-Frame, Bolt) select individual, isolated frames and feed them directly to the LMM. The paper argues this is problematic because actions, causal sequences, and events unfold over time, and a single frame cannot capture the temporal dynamics needed for reasoning — especially for "why" and "how" questions.

The conceptual insight is that frame relevance is not localizable to individual instants — a frame is relevant not only because of what it contains, but because of what it indicates about its temporal neighborhood. The two-dimensional reward criterion (which includes contextual adjacency) is the scoring manifestation of this insight; the video refinement step's segment expansion is the retrieval manifestation. Together, they implement a principle that query-relevant information in video is inherently spatiotemporal, not instantaneous, and that frame selection must respect this temporal extent.

Comparison to prior work: Methods that select sparse frames and concatenate them for the LMM implicitly assume that the selected frames are sufficient in isolation — that the information in each frame is self-contained, and the LMM can reconstruct any needed temporal relationships from the sequence of selected frames. The paper's window-length ablation (Figure 8) directly refutes this: wlen = 0 (r-frames only, no expansion) yields the lowest performance across all benchmarks, substantially below wlen = 2 and wlen = 4. This proves that even when the selected frames are individually highly relevant, their isolation destroys performance. The temporal context provided by the surrounding frames is not a nice-to-have — it is essential.

The union operation for merging overlapping segments is a simple but principled design choice that prevents the refined video from becoming fragmented. Without it, adjacent selected r-frames with overlapping expansion windows would create duplicate content or awkward boundaries; the union produces clean, contiguous blocks of query-relevant video. This ensures that the final uniform sampling step operates on a coherent temporal region rather than disjoint slices, which matters because LMMs process frames as a sequence and can be sensitive to temporal discontinuities.

The practical significance is that this expansion mechanism is what allows DIG to maintain gains at very high frame counts (256, 512, 768) where methods selecting sparse individual frames degrade. At 768 frames on Qwen3-VL-8B (Appendix F, Table 5), DIG achieves +4.7% on MLVU, +3.7% on LongVideoBench, and +3.5% on VideoMME-Medium over uniform sampling — gains that are impossible for methods that select frames without preserving temporal continuity, because increasing the frame count in those methods would just add more sparse, decontextualized keyframes without the surrounding context needed for reasoning. The temporal expansion principle is thus not just an accuracy improvement — it is what makes the method scalable to the frame budgets of realistic deployment.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three long-form video understanding benchmarks. MLVU (Zhou et al., 2025) contains 3,102 questions across 9 categories, using only multiple-choice questions from the dev set of 2,593 questions, excluding open-ended ones. Videos average 636.2 seconds in duration. LongVideoBench (Wu et al., 2024) contains 3,763 web-collected videos with 6,678 human-annotated multiple-choice questions across 17 fine-grained categories, designed to test referring reasoning. The paper uses only the validation set of 1,337 QA pairs. Videos average 732.2 seconds. VideoMME (Fu et al., 2024) contains 900 videos totaling approximately 254 hours with 2,700 QA pairs across 30 subdomains. The paper uses only the medium (516.8-second average) and long (2,466.3-second average) splits, excluding the short split. No subtitles or audio are used — evaluation is strictly based on pure visual understanding.

Base models. The paper tests three LMM families: Qwen2.5-VL-7B and Qwen2.5-VL-32B (Bai et al., 2025) as primary evaluation models, Qwen3-VL-8B (Bai et al., 2025) for extended scalability experiments in Appendix F, and InternVL3-8B (Zhu et al., 2025) and LLaVA-OneVision-7B (Li et al., 2024) in preliminary analyses (Figures 2–3). The 7B-scale models are chosen as representative of widely deployed open-source LMMs; the 32B model tests whether DIG benefits persist at larger scale. Qwen3-VL-8B is used specifically for its longer context window, enabling tests at 512 and 768 input frames.

Metrics. The primary metric is accuracy (%) — the fraction of multiple-choice questions answered correctly by selecting the option letter (A, B, C, D) that matches the ground truth. For VideoMME, results are reported separately for medium and long splits. For the query-type analysis in Section 6.1, accuracy is broken out by global queries (GQ) and localized queries (LQ) using the manually annotated classifications described in Appendix B. For the CAFS evaluation in Section 6.2, two additional metrics are introduced: Localized Coverage (LoC) and Global Coverage (GlC) (Equations 4 and 5), which measure how well selected r-frames represent their local temporal context and the video's overall visual diversity, respectively.

Baselines. The paper compares against three baselines. (1) Uniform sampling (UNI): frames are selected at fixed intervals across the entire video duration, with no query awareness. This is the standard default in most video LMM deployments. (2) AKS (Adaptive Keyframe Sampling, Tang et al., 2025): samples candidate frames at 1 fps, computes frame-query similarity via BLIP (Li et al., 2022), and applies an adaptive selection algorithm. Evaluated at frame budgets of {32, 64, 128, 192, 256}; budgets of 8 and 16 are excluded because the algorithm occasionally yields null returns at these low settings (Appendix F.1). (3) Q-Frame (Zhang et al., 2025): uses a fixed candidate pool of 128 uniformly sampled frames and applies CLIPScore-based selection with a fixed frame count strategy. Limited to frame budgets of {8, 16, 32, 64, 128} due to the hard cap on the candidate pool.

Generation budget / compute accounting. The "budget" is the number of input frames N fed to the LMM for final inference, swept across values from 8 to 256 in main experiments and up to 768 in extended experiments. Each frame is represented using 56 tokens (Section 5.1). Computational cost is measured in two ways: (1) FLOPs for the end-to-end pipeline per QA pair (Figure 9), comparing DIG against uniform sampling at varying frame counts, and (2) wall-clock time for each stage of the pipeline (Table 10 in Appendix G), broken down into query identification, CAFS, reward assignment, and video refinement. The FLOPs comparison in Figure 9 reveals that uniform sampling saturates at approximately 62.5% accuracy on LongVideoBench regardless of additional computation, while DIG surpasses this ceiling once total computation exceeds approximately 720 TFLOPs.

Cross-validation / statistical protocol. There is no explicit cross-validation or statistical significance testing reported. The manual annotation protocol for query classification on VideoMME used majority voting across human annotators (Appendix B). For MLVU, query types are determined by the benchmark's intrinsic task categorization (holistic tasks → global queries; single-detail and multi-detail tasks → localized queries). For LongVideoBench, all queries are classified as localized since the benchmark is explicitly designed for referring reasoning. The paper reports single-run results without confidence intervals or error bars in most tables and figures. The small test set sizes — for example, 2174 QA pairs in MLVU after filtering, 1337 in LongVideoBench-val, and 900 each in VideoMME-Medium and VideoMME-Long — mean that percentage differences of 1–2% correspond to small absolute numbers of questions, and the paper does not report whether observed differences are statistically significant.

Main Quantitative Results

Overall Comparison Against Baselines (Table 1, Table 5)

The headline result is that DIG consistently outperforms uniform sampling and both AKS and Q-Frame across nearly all frame budgets and benchmarks, for both Qwen2.5-VL-7B and Qwen2.5-VL-32B.

With Qwen2.5-VL-7B at 32 frames (Table 1, right): DIG achieves 67.20% on MLVU versus 59.52% for uniform sampling (+7.68 percentage points), 60.43% on LongVideoBench versus 56.92% (+3.51 points), and 61.62% on VideoMME-Medium versus 59.08% (+2.54 points). AKS achieves 65.07% on MLVU (competitive but below DIG) and 59.31% on LongVideoBench (below DIG's 60.43%). Q-Frame achieves 60.03% on MLVU (barely above uniform's 59.52%) and 56.39% on LongVideoBench (below uniform).

With Qwen2.5-VL-32B at 32 frames (Table 1, left): DIG achieves 70.69% on MLVU versus 61.91% for uniform (+8.78 points), 61.86% on LongVideoBench versus 57.89% (+3.97 points), and 60.87% on VideoMME-Medium versus 57.89% (+2.98 points). AKS achieves 66.42% on MLVU (below DIG's 70.69%).

Scaling to high frame counts (Table 1, Qwen2.5-VL-7B): At 128 frames, AKS achieves 68.68% on MLVU versus uniform's 67.31% — a marginal gain of +1.37 points. On LongVideoBench, AKS achieves 60.36% which is below uniform's 61.86% (a degradation of −1.50 points). Q-Frame at 128 frames achieves 68.03% on MLVU (below AKS and uniform) and 59.76% on LongVideoBench (below uniform). In contrast, DIG achieves 71.40% on MLVU (+4.09 points over uniform) and 63.13% on LongVideoBench (+1.27 points over uniform). At 256 frames, DIG achieves 72.46% on MLVU (+3.31 over uniform's 69.15%) and 64.62% on LongVideoBench (+3.14 over uniform's 61.48%), while AKS degrades to 61.03% on LongVideoBench (−0.45 points below uniform).

With Qwen3-VL-8B at 768 frames (Table 5, Appendix F): DIG achieves 72.2% on MLVU (+4.7 over uniform's 67.5%), 64.6% on LongVideoBench (+3.7 over uniform's 60.9%), and 67.8% on VideoMME-Medium (+3.5 over uniform's 64.3%). AKS at 768 frames achieves 65.3% on MLVU (−2.2 below uniform), 58.3% on LongVideoBench (−2.6 below uniform), and 62.3% on VideoMME-Medium (−2.0 below uniform). This demonstrates that DIG is the only method among those tested that maintains positive gains over uniform sampling at extreme frame budgets; both AKS and Q-Frame systematically degrade below the uniform baseline as frame count increases.

Performance at low frame counts (Table 1): At 8 frames, DIG with Qwen2.5-VL-32B achieves 61.55% on MLVU versus uniform's 55.93% (+5.62 points). This is notable because 8 frames represents a very sparse sampling of long videos — the fact that DIG achieves substantial gains even here indicates that its refined video approach succeeds in concentrating those few frames on the most relevant content.

Performance by Query Type: The Core Contingency (Figure 5)

The paper's central architectural claim — that uniform sampling suffices for global queries while query-aware selection is essential for localized queries — is directly tested in Figure 5, which plots accuracy versus frame count separately for LQ and GQ, comparing uniform sampling against DIG's full query-aware pipeline.

For localized queries (Figure 5, left three panels): DIG substantially and consistently outperforms uniform sampling across all frame counts and all three benchmarks, for both Qwen2.5-VL-7B and Qwen2.5-VL-32B. On MLVU LQ with Qwen2.5-VL-32B, DIG achieves approximately 75% at 256 frames versus uniform's approximately 67% — a gap of approximately 8 percentage points that widens with frame count. On LongVideoBench (which is 97.8% localized queries per Table 11), DIG with Qwen2.5-VL-32B achieves approximately 66% at 256 frames versus uniform's approximately 63% — a more modest gap consistent with LongVideoBench's emphasis on fine-grained temporal reasoning, which the paper identifies as a bottleneck independent of frame selection quality (Appendix F.3).

For global queries (Figure 5, right two panels): The DIG pipeline and uniform sampling produce essentially overlapping curves. On MLVU GQ with Qwen2.5-VL-32B, both methods achieve approximately 89% at 256 frames. On VideoMME GQ with Qwen2.5-VL-32B, uniform sampling achieves approximately 75% at 256 frames — slightly above DIG, though the difference is small and the curves are nearly parallel. This confirms the paper's negative-result claim: the query-aware pipeline provides zero benefit for global queries, and the routing decision to bypass it is correct.

Asymmetric scaling behavior: On localized queries, the gap between DIG and uniform sampling tends to widen with increasing frame count (e.g., on MLVU LQ with Qwen2.5-VL-32B, the gap grows from approximately 3 points at 8 frames to approximately 8 points at 256 frames). This is because uniform sampling's signal-to-noise ratio degrades as more irrelevant frames are added, while DIG's refined video approach concentrates additional frames within query-relevant regions, making the added frames helpful rather than harmful. On global queries, the curves remain flat and overlapping regardless of frame count, consistent with the idea that holistic understanding benefits from broadly distributed temporal coverage.

CAFS Effectiveness: Ablation Within DIG (Figure 7) and Coverage Metrics (Figure 6)

Replacing CAFS with uniform sampling within DIG (Figure 7): When the CAFS-extracted r-frames are replaced with standard uniformly sampled frames (keeping all other pipeline components — reward assignment, video refinement, final uniform sampling — identical), performance drops across all benchmarks. For Qwen2.5-VL-7B on MLVU at 256 frames, DIG with CAFS achieves approximately 72.5% versus approximately 69% with uniform candidate generation — a drop of approximately 3.5 percentage points. The performance gap widens with increasing frame count: at 8 frames, the gap is small (approximately 1 point on MLVU); at 256 frames, it is substantial. This confirms the paper's claim that CAFS's adaptive candidate generation becomes increasingly important as the frame budget grows, because uniform candidate generation becomes progressively less adequate at capturing the video's content structure.

Coverage metrics compared to baselines (Figure 6): On both MLVU and VideoMME, CAFS achieves higher Localized Coverage (LoC) and Global Coverage (GlC) than uniform sampling and fixed-FPS sampling, especially for videos longer than 10 minutes. For VideoMME videos in the 20–30 minute range, CAFS achieves LoC of approximately 0.85 versus approximately 0.75 for uniform sampling and approximately 0.78 for FPS sampling. The FPS baseline maintains more stable performance than uniform sampling as video duration increases — uniform's coverage degrades sharply for long videos because a fixed frame budget becomes increasingly sparse — but CAFS consistently outperforms both. The number of r-frames produced by CAFS (shown as bars in Figure 6, right y-axis) increases with video duration but sublinearly: CAFS extracts more frames from longer videos, but not proportionally to length, reflecting the non-linear relationship between duration and content complexity.

LMM vs. CLIPScore for Reward Assignment (Table 2)

Headline result (Table 2): LMM-based reward assignment (using Qwen2.5-VL-7B or Qwen2.5-VL-32B as the reward model) consistently outperforms CLIPScore-based reward assignment when all other DIG components are held constant. The base inference model in all cases is Qwen2.5-VL-7B.

At 32 frames on MLVU: Qwen2.5-VL-32B rewards achieve 67.9%, Qwen2.5-VL-7B rewards achieve 67.2%, and CLIPScore achieves 65.4%. The gap between LMM-based and CLIPScore-based rewards is 1.8–2.5 percentage points. On VideoMME-Long: 53.1% (32B), 53.2% (7B), 51.2% (CLIPScore).

At high frame counts (128+ frames): The gap widens. At 256 frames on MLVU: 74.3% (32B), 72.5% (7B), 71.2% (CLIPScore). On VideoMME-Long at 256 frames: 59.1% (32B), 57.8% (7B), 57.0% (CLIPScore). The consistent advantage of the larger reward model (32B vs. 7B) — approximately 1–2 percentage points across most configurations — provides evidence for the paper's claim that "better LMMs yield superior rewards." The fact that even the weaker LMM (7B) outperforms CLIPScore confirms that the LMM's reasoning capabilities, not just its scale, drive the improvement.

Interesting pattern on LongVideoBench: At 256 frames, Qwen2.5-VL-7B rewards (64.6%) slightly outperform Qwen2.5-VL-32B rewards (64.5%), and the CLIPScore gap is relatively small (61.9% vs. 64.5–64.6%). This is consistent with the paper's observation that LongVideoBench emphasizes fine-grained temporal reasoning (L2-Relation, temporal ordering tasks) where frame selection quality matters less because the bottleneck is the model's temporal reasoning capability rather than the presence or absence of relevant visual evidence (Appendix F.3: "Temporal reasoning remains a fundamental bottleneck").

Window Length Ablation (Figure 8)

At wlen = 0 (selected r-frames only, no temporal expansion): Performance is the lowest across all benchmarks and frame counts. On MLVU with Qwen2.5-VL-32B at 256 frames, wlen = 0 achieves approximately 65%, while the best setting (wlen = 2) achieves approximately 74% — a gap of roughly 9 percentage points. This directly supports the paper's claim that isolated keyframes are insufficient and that temporal context is essential.

At wlen = 2: This achieves the best or near-best performance across all benchmarks. It provides enough context to capture actions and causal sequences while avoiding the injection of irrelevant frames from distant temporal regions.

At wlen = 4: Performance is slightly below wlen = 2 on most benchmarks (e.g., approximately 72% vs. 74% on MLVU at 256 frames), but still substantially above wlen = 0.

At wlen = 8: Performance degrades further, approaching or falling slightly below wlen = 4. On VideoMME-Medium at high frame counts, wlen = 8 drops below wlen = 4, and on MLVU the curves for wlen = 4 and wlen = 8 nearly overlap. The paper attributes this to excessive context introducing noise — for localized queries, expanding the window too far begins to include frames from temporally distant and irrelevant segments, undermining the precision of the targeted selection.

Benchmark-specific optimal wlen: The pattern suggests that the optimal wlen depends on the temporal granularity required by the benchmark. LongVideoBench, which emphasizes reasoning over extended temporal contexts, might benefit from larger windows; VideoMME, which includes many short-duration perception tasks, benefits from tighter windows.

Efficiency Analysis (Figure 9, Tables 9–11)

Performance-efficiency tradeoff (Figure 9): The plot of accuracy versus TFLOPs on LongVideoBench with Qwen2.5-VL-7B reveals a clear saturation pattern for uniform sampling: accuracy peaks at approximately 62.5% at around 400 TFLOPs and does not improve with additional computation or frame count. DIG, operating at a higher computational budget (≥680 TFLOPs due to the overhead of CAFS, reward assignment, and video refinement), surpasses the uniform sampling ceiling once computation exceeds 720 TFLOPs and continues to improve, reaching approximately 65% accuracy at 900–1000 TFLOPs. This demonstrates that DIG's additional computational cost buys access to a performance regime that uniform sampling cannot reach regardless of budget — the extra computation is not merely shifting the curve but enabling fundamentally better scaling.

Absolute runtime costs (Table 10, Appendix G): For MLVU with Qwen2.5-VL-7B, DIG's total frame selection overhead is 256.3 minutes (sum of query identification: 11.3 min, CAFS: 25.9 min, reward assignment: 218.9 min, video refinement: 0.2 min). AKS requires ≥720 minutes for frame selection across any dataset, making DIG approximately 3× faster than AKS on MLVU. Q-Frame requires 122.1 minutes on MLVU — faster than DIG, but the paper argues this speed advantage is negated by Q-Frame's failure to outperform uniform sampling at higher frame counts (Table 1). The reward assignment stage dominates DIG's overhead (218.9 of 256.3 minutes, or approximately 85%), suggesting that reducing r-frame count or using a smaller reward model would be the most impactful optimization.

Query identification savings (Table 11): Applying the expensive query-aware pipeline universally (without query-type routing) would cost 295.7 minutes on MLVU; with query identification routing global queries to uniform sampling, the cost drops to 256.3 minutes — a 13.3% reduction. On VideoMME, the savings are 19.9% (384.2 → 307.9 minutes). On LongVideoBench, where 97.8% of queries are localized, query identification adds a small overhead of 3.6% (134.1 → 138.9 minutes) because almost all queries undergo the expensive pipeline anyway; the cost of the classification step itself is included in this figure.

Inference latency comparison (Table 9): The base LMM inference time with uniform sampling grows from 3.2 minutes (8 frames) to 43.4 minutes (256 frames) on MLVU with Qwen2.5-VL-7B. In comparison, DIG's frame selection overhead (256.3 minutes for MLVU) is substantially larger than the inference time at all but the highest frame counts, meaning that DIG trades significantly increased total latency for improved accuracy. The paper does not discuss this latency tradeoff explicitly, but the numbers in Tables 9 and 10 make clear that DIG is not a real-time method — it is designed for accuracy maximization under a frame budget, not latency minimization.

Ablation Studies and Robustness Checks

Query classification accuracy across LLMs (Table 4): Four LLMs are evaluated on their ability to classify queries as global vs. localized, compared to human annotations. Overall accuracy ranges from 78.52% (Qwen3-Next-80B-A3B) to 87.28% (DeepSeek-R1-Distill-Qwen-32B) on MLVU, and from 82.95% to 87.28% on VideoMME. Critical localized-query accuracy — where misclassification could harm performance — exceeds 90% for nearly all models: 93.65% (Llama-3.1-8B) on MLVU LQ, 99.18% (DeepSeek-R1-Distill-Qwen-32B) on LongVideoBench LQ. Global query accuracy is lower (24.01% to 74.93%), which the paper argues is acceptable because misclassified global queries incur only computational overhead, not accuracy loss.

CAFS vs. uniform sampling within DIG (Figure 7): Replacing CAFS-extracted r-frames with uniformly sampled frames degrades performance across all four evaluated settings (MLVU, LongVideoBench, VideoMME-Medium, VideoMME-Long). The degradation increases with frame count, confirming that CAFS's adaptive candidate generation is most important when many final frames need to be sampled — because uniform candidate generation becomes increasingly sparse relative to the video's content.

LMM reward assigner scale (Table 2): Across all four VideoMME splits and both MLVU and LongVideoBench, Qwen2.5-VL-32B as reward assigner outperforms Qwen2.5-VL-7B in the majority of configurations, with an average gap of approximately 1–2 percentage points at high frame counts. The consistent direction of this gap supports the claim that better LMMs provide better relevance signals. The exceptions (e.g., LongVideoBench at 256 frames where 7B slightly outperforms 32B) suggest the advantage is not universal and may depend on the interaction between reward quality and the downstream task's sensitivity to frame selection precision.

CLIPScore as reward baseline (Table 2): CLIPScore consistently underperforms both LMM-based reward assigners, with the gap widening at higher frame counts. On VideoMME-Short at 256 frames: 76.8% (32B), 76.3% (7B), 75.0% (CLIPScore). On MLVU at 256 frames: 74.3% (32B), 72.5% (7B), 71.2% (CLIPScore). This supports the paper's argument that shallow similarity metrics cannot capture the nuanced relevance assessment needed for effective frame selection.

Window length wlen (Figure 8): The ordering wlen = 0 < wlen = 8 < wlen = 4 < wlen = 2 holds across most settings, with wlen = 2 consistently the best or tied for best. The degradation at wlen = 8 confirms that excessive temporal expansion introduces noise. The fact that wlen = 0 performs worst across all benchmarks and frame counts confirms the paper's central claim that temporal context is necessary — isolated keyframes, even when highly relevant, are insufficient.

Query type breakdowns per task (Tables 6–8): The per-task performance breakdowns reveal that DIG's gains are concentrated in perception-heavy localized tasks. On MLVU (Table 6, Qwen2.5-VL-7B), DIG at 32 frames improves over uniform sampling by: +10.6 points on PlotQA (70.3% vs. 59.7%), +11.6 points on NeedleQA (80.6% vs. 69.0%), +19.4 points on Action Count (42.2% vs. 22.8%), and +3.0 points on Ego Reasoning (59.1% vs. 54.0%). For global tasks, gains are minimal or nonexistent: Topic Reasoning at 32 frames shows 87.5% (DIG) vs. 84.4% (uniform) — a +3.1 point gain; Anomaly Recognition shows 73.0% vs. 74.5% — a −1.5 point loss. On temporal reasoning tasks like Action Order, gains remain stagnated across all methods, suggesting that frame selection cannot compensate for fundamental limitations in the LMM's temporal reasoning capability.

Scaling behavior of baselines at high frame counts (Table 1, Table 5): Both AKS and Q-Frame exhibit a pattern of initial gains over uniform sampling at low-to-moderate frame counts (32–64), followed by degradation below uniform at higher frame counts (128–256). This is most pronounced on LongVideoBench with Qwen2.5-VL-7B (Table 1, right): AKS is above uniform at 32 and 64 frames (+2.39 and +1.72 points respectively) but below uniform at 128, 192, and 256 frames (−1.50, −0.67, and −0.45 points). Q-Frame is above uniform at 8 and 16 frames (+3.00 and +2.92 points) but below uniform at 32, 64, 128 frames. This cross-over pattern is a key empirical finding: it demonstrates that current query-aware methods are not robust to increasing frame budgets, and reinforces the paper's argument that scaling behavior should be a primary evaluation criterion.

Model family robustness (Tables 1 vs. Table 5, Figure 2 vs. Figure 3): DIG's advantages are demonstrated across three LMM families (Qwen2.5-VL-7B/32B, Qwen3-VL-8B, and preliminary analyses with InternVL3-8B and LLaVA-OneVision-7B in Figures 2–3). The consistent pattern of performance degradation with increasing uniform frame count (Figure 2) and query-type-dependent scaling (Figure 3) across all tested models suggests that the underlying phenomenon — localized queries suffering from frame dilution, global queries benefiting from broad coverage — is model-agnostic rather than specific to a particular architecture.

Negative result: ReST-EM revision model (not applicable to this paper): The paper does not include reinforcement learning or iterative self-improvement experiments; all components are training-free. The closest to a negative result in the ablations is the failure of CLIPScore as a reward mechanism (Table 2) and the failure of wlen = 0 (Figure 8), both of which support the paper's design choices by demonstrating the inadequacy of simpler alternatives.

Critical Assessment

Does the evidence support the paper's central claims?

Claim 1: "We identify a query typology (global vs. localized) and demonstrate that the efficacy of frame selection strategies is highly contingent on this classification."

The evidence for this claim is strong and multi-faceted. Figure 3 demonstrates the query-type-dependent performance degradation under uniform sampling across three different LMMs and two benchmarks, with localized queries degrading substantially as frame count increases while global queries remain stable. Figure 5 then demonstrates the converse — that the DIG query-aware pipeline substantially improves localized queries while providing no benefit (and occasional slight harm) for global queries, across both Qwen2.5-VL-7B and Qwen2.5-VL-32B. The per-task breakdowns in Tables 6–8 confirm that gains are concentrated in localized perception tasks (PlotQA, NeedleQA) while global tasks (Anomaly Recognition, Topic Reasoning) show minimal differences.

However, there are important caveats to how this claim is validated:

  • The classification scheme is partially dataset-dependent. MLVU's task structure maps directly to global vs. localized (holistic tasks → global, detail tasks → localized). LongVideoBench is treated as entirely localized because of its "referring reasoning" design. Only VideoMME required manual annotation. This means the claim's generality depends on whether these three benchmarks adequately represent the space of possible query types in real-world video understanding. The paper does not evaluate on, for example, open-ended video QA, video summarization, or dialogue about videos — all of which might have query types that don't cleanly fit the binary classification.

  • Global query classification accuracy is low (38.26% on MLVU for Qwen3-Next-80B-A3B, Table 4). The paper argues this is acceptable because misclassified global queries only incur computational overhead. But this means that in practice, a substantial fraction of queries that the system routes to the expensive pipeline would have been adequately served by uniform sampling — the efficiency savings are smaller than they could be if classification were perfect. A system that misclassifies 60%+ of global queries as localized is substantially less efficient than the theoretical optimum.

  • The query typology is binary, but the examples suggest a spectrum. A query like "What is the boy's overall role?" is classified as global by the paper's definition (has a referent but still requires holistic understanding), but might benefit from some degree of targeted selection — seeing the scenes where the boy appears more densely than scenes where he's absent. The binary routing decision may be too coarse for intermediate query types, and the paper does not explore whether a continuous query-scope estimate would improve performance.

Claim 2: "DIG, a training-free frame selection framework, adapts to query type by employing uniform sampling for global queries and a specialized pipeline to extract query-relevant frames for localized queries."

The claim that DIG works and is training-free is supported: all components (DINOv2, LLM for classification, LMM for reward assignment and inference) are used off-the-shelf without fine-tuning. The architecture's dual-path design is clearly motivated and validated by the query-type analysis.

Weaknesses in the evidence:

  • The "specialized pipeline" is only validated as a monolithic block in the query-type comparison (Figure 5). While individual components (CAFS, reward model, wlen) are ablated, the paper never compares alternative pipeline designs (e.g., different ordering of stages, skipping video refinement and using r-frames directly with temporal ordering, using the reward model to directly score uniformly sampled frames without the CAFS intermediary). The claim that this particular pipeline is optimal among possible designs is not tested.

  • The training-free property is technically true but misleading in a practical sense. DIG requires a capable LMM for reward assignment (Qwen2.5-VL-32B for best results), an LLM for query classification, and DINOv2 as a feature extractor. While none of these require training for DIG specifically, they are all large pretrained models. The paper's efficiency analysis (Tables 9–11) shows that the reward assignment stage alone takes 218.9 minutes on MLVU — this is "training-free" in the sense of not requiring gradient updates, but it is not zero-cost or lightweight.

  • The decoupling of reward model and inference model is demonstrated (Table 2 shows Qwen2.5-VL-32B rewards + Qwen2.5-VL-7B inference outperforms Qwen2.5-VL-7B rewards + Qwen2.5-VL-7B inference), but the paper does not explore the tradeoff space systematically. How much does the reward model's quality matter relative to its cost? Could a very small, specialized reward model (e.g., a fine-tuned CLIP variant) achieve most of the benefit at a fraction of the cost? These questions are not addressed.

Claim 3: "Experiments on three long-form video understanding benchmarks show that DIG consistently outperforms existing baselines and robustly improves LMM's performance, even when scaling input frame count to 256."

This claim is the best-supported in the paper, with comprehensive evidence across Tables 1 and 5, Figures 5–9, and the extended Qwen3-VL-8B experiments at 512 and 768 frames.

However, several limitations should be noted:

  • "Consistently outperforms" has exceptions. Table 1 shows that at some configurations, AKS is competitive or slightly better — e.g., on VideoMME-Medium at 64 frames with Qwen2.5-VL-7B (right side), AKS achieves 62.94% vs. DIG's 62.61%, a marginal difference but technically a case where DIG does not outperform the baseline. The paper does not flag these exceptions, instead emphasizing the aggregate pattern of DIG's superiority. This is not a serious weakness — the overall trend is clear — but the claim of "consistently" is slightly overstated.

  • The baselines (AKS, Q-Frame) are recent but not state-of-the-art at time of publication. The paper does not compare against Bolt (Liu et al., 2025), Frame-Voyager (Yu et al., 2024), MDP3 (Sun et al., 2025), or T* (Ye et al., 2025) — all query-aware frame selection methods published before or contemporaneously with DIG. While the paper positions itself as addressing limitations common to this class of methods (static candidate generation, shallow similarity metrics, sparse frame selection), a direct comparison against the strongest contemporary methods would strengthen the claim of superiority.

  • The "scaling to 256 frames" claim is relative to baselines that degrade at scale. The improvement at 256 frames is partly due to DIG improving and partly due to baselines worsening. On Qwen2.5-VL-7B at 256 frames on MLVU (Table 1, right), DIG achieves 72.46% vs. uniform's 69.15% (+3.31). But AKS achieves 71.50% — actually above uniform and only slightly below DIG. The dramatic gap the paper emphasizes is on LongVideoBench, where AKS degrades below uniform while DIG maintains gains. The claim that DIG "robustly improves" at scale is true in aggregate, but the margin varies substantially by benchmark, and on some benchmarks (VideoMME-Medium with Qwen2.5-VL-32B at 256 frames, Table 1 left: DIG 67.66% vs. uniform 66.31% — a +1.35 point gain) the improvement is modest.

Potential missing experiments that would strengthen the paper:

  1. Direct comparison against more baselines — particularly Bolt (which also uses LMM-based relevance assessment) and T* (which addresses the temporal search problem specifically). This would clarify whether DIG's gains come from its specific pipeline design or from the general principle of LMM-based relevance scoring.

  2. Sensitivity analysis of the prominence threshold (0.1) and the 2 fps sampling rate. The paper states the prominence threshold "has been found effective through empirical observation" but does not show what happens with different thresholds. A threshold that is too low produces too many r-frames (increasing reward assignment cost); too high produces too few (missing content). Similarly, the choice of 2 fps for the initial sampling is not ablated — would 1 fps or 4 fps produce different r-frame sets?

  3. Analysis of failure cases. The paper reports quantitative improvements but does not show examples where DIG fails — cases where the refined video excludes the relevant segment, or where the reward model assigns high scores to irrelevant frames. Understanding failure modes would clarify the method's limitations and guide future improvements.

  4. Statistical significance testing. With test sets of 900–2174 questions and accuracy differences sometimes as small as 1–2 percentage points (which could represent <20 questions), confidence intervals or significance tests would help readers assess whether observed differences are reliable or within sampling noise.

  5. Latency-constrained evaluation. The paper reports total processing time but does not evaluate whether DIG's gains persist under a wall-clock time budget. If an application requires answers within, say, 30 seconds, can DIG's pipeline complete in time? What is the tradeoff between accuracy and latency at fixed time budgets?

Assessment of specific experimental design choices

The choice to measure only accuracy, not open-ended generation quality. All benchmarks use multiple-choice questions. This makes evaluation straightforward and unambiguous but limits the findings' applicability to open-ended VQA, where the model must generate free-form answers. The frame selection problem is likely similar for open-ended questions, but the paper provides no evidence.

The exclusion of subtitles and audio. The paper explicitly evaluates "pure visual understanding" without subtitles. This is a deliberate scoping choice that simplifies the frame selection problem (no need to worry about synchronizing with text or audio tracks) but also limits ecological validity — many real-world video understanding tasks involve multimodal inputs where subtitles provide crucial information. An interesting extension would test whether DIG's frame selection complements or conflicts with subtitle-based reasoning.

The manual annotation protocol for query types (Appendix B). On VideoMME, query types were determined by majority vote among human annotators. The paper does not report inter-annotator agreement (e.g., Cohen's kappa), which is important because query type classification is subjective — the paper itself notes edge cases where a query has a referent but still requires holistic understanding. Without agreement statistics, it's unclear whether the "ground truth" labels are reliable or whether the LLM's low GQ accuracy (Table 4) partly reflects inherent ambiguity rather than classifier error.

The choice of 56 tokens per frame. This is stated as a fixed configuration (Section 5.1) but not ablated. Different LMMs and different tokenization strategies could change the optimal frame count and the relative benefit of query-aware selection. A model that represents frames more compactly (fewer tokens per frame) could process more frames and might benefit less from targeted selection; a model with coarser frame representation might benefit more. This interaction is unexplored.

In summary, the paper's experimental evidence strongly supports its core claim that frame selection strategy effectiveness depends on query type, and provides convincing evidence that DIG outperforms uniform sampling, AKS, and Q-Frame across a range of frame budgets and benchmarks. The scaling results at 256–768 frames are particularly valuable because they demonstrate robustness in a regime where competing methods fail. The primary limitations are the binary query classification's imperfect accuracy (especially for global queries), the incomplete ablation of design hyperparameters, the absence of comparisons against the strongest contemporary baselines, and the lack of statistical significance reporting. The experiments demonstrate that DIG works, but leave open questions about why specific design choices matter, how much each component contributes at the margin, and under what conditions the method might fail.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in Headline Efficiency Numbers

The assumption or constraint. The compute-optimal allocation policy depends on estimating each prompt's difficulty before deciding how to spend the inference budget. The paper uses two methods: oracle difficulty (requiring ground-truth labels and 2048 samples per question) and predicted difficulty (replacing ground-truth correctness with PRM scores on the same 2048 samples). The paper explicitly acknowledges this cost in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The predicted difficulty method still requires generating 2048 complete solutions per question and scoring each with the PRM — an operation that consumes more compute than the largest test-time budgets studied (which max out at 256–512 generations). The headline 4×4\times efficiency gain over best-of-N is computed after difficulty is known, without amortizing the cost of learning it.

The consequence. In a realistic deployment, the total cost would equal difficulty estimation cost plus strategy execution cost. Since the estimation cost (2048 generations) is 4–8× larger than the largest strategy budgets (256–512 generations), the actual end-to-end efficiency could be substantially worse than the reported 4×4\times figure suggests. A system using predicted difficulty bins would need to sample 2048 solutions, score them with the PRM, bin the question, then spend an additional 16–256 generations executing the strategy. The estimation cost dominates, and the marginal benefit of the strategy choice over simply using best-of-2048 in the first place is unclear — if you've already generated 2048 solutions for difficulty estimation, you could instead just apply best-of-N weighted selection to those 2048 solutions directly.

Furthermore, the difficulty estimation itself introduces a cold-start problem: for a new question, the system has no difficulty estimate, so it must burn significant computation before making any allocation decision. This is a form of exploration-exploitation tradeoff that the paper flags in Section 3.2 but does not resolve:

"we acknowledge that this is a key area for future exploration as it ultimately incurs a tradeoff between compute spent assessing question difficulty and compute spent answering the question"

What evidence exists in the paper. The paper demonstrates that predicted difficulty bins largely overlap with oracle bins in Figures 4 and 8 — but this only shows that the ranking of questions by PRM score is correlated with the ranking by ground-truth pass@1. It does not show that the difficulty estimation cost is amortizable, that fewer than 2048 samples would suffice, or that the total end-to-end cost (estimation + execution) is lower than simply using best-of-N with the combined budget. No experiment measures total FLOPs including difficulty estimation. The paper never compares "difficulty estimation + compute-optimal strategy" against "best-of-N with an equivalent total budget."

Mitigation status. The paper acknowledges this limitation explicitly and suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), or reducing the number of samples needed for difficulty estimation. Neither approach is developed or evaluated. The current framework requires a method that can estimate difficulty with substantially fewer than 2048 samples before the 4×4\times claim translates to end-to-end savings. Until such a method is demonstrated, the reported efficiency gains represent an upper bound conditioned on free difficulty estimation.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Create Capability

The assumption or constraint. The compute-optimal framework assumes that the base model's proposal distribution contains correct solutions at some non-trivial rate — that is, the model already "knows how" to solve problems of this type, and test-time compute is amplifying existing capability. The paper is explicit about this boundary in Section 7:

"test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine."

This is not so much an oversight as a fundamental capability bound — but it has practical consequences that the paper's headline claims do not fully surface.

The consequence. For the hardest problems (difficulty bin 5), the compute-optimal policy provides essentially no benefit regardless of budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, and in many configurations the larger pretrained model substantially outperforms the smaller model with test-time compute (e.g., −52.9% relative disadvantage for PRM search on hard problems at R≫1R \gg 1).

This has two implications. First, the compute-optimal framework provides no path forward for problems that exceed the base model's training distribution — if a model has never seen problems of a certain reasoning complexity, scaling test-time compute will not help. This is a hard ceiling, not a soft tradeoff. Second, the fraction of problems in each difficulty bin is benchmark-dependent, so the overall benefit of compute-optimal scaling depends on the problem distribution. On a benchmark skewed toward hard problems (e.g., competition-level math where the base model's pass@1 is near zero), the framework would yield minimal gains regardless of budget.

What evidence exists in the paper. Figures 3 (right), 7 (right), and 9 all show the bin 5 performance floor consistently across search methods, revision strategies, and FLOPs-matched comparisons. The paper also provides qualitative evidence in Appendix M (Figures 29–31) showing degenerate search outputs for hard problems — repetitive low-information steps and overly short solutions that score highly under the PRM but are factually incorrect.

Mitigation status. The paper acknowledges this limitation clearly in the Section 7 takeaway box, noting that "test-time compute is most effective when problems are within the base model's reach." However, it does not provide guidance on how to determine a priori whether a problem is in bin 5 without the expensive difficulty estimation that bin 5 problems would fail anyway. The framework routes compute to hard problems (via the compute-optimal policy lookup) but gets nothing back — a practitioner would want to know when to not spend any compute and instead escalate to a larger model or human. The paper does not explore this "abstention" decision.

The Model Is a Sparse Keyframe Selector That Sacrifices Temporal Continuity for Localized Queries

The assumption or constraint. DIG's query-aware pipeline for localized queries selects representative frames (r-frames) based on visual dissimilarity peaks, then expands them into continuous segments via the window parameter wlen. However, the underlying selection mechanism operates on individual frames scored in isolation, not on temporal segments scored jointly. The reward assignment step (Section 4.3) presents each r-frame individually to the LMM with the query and asks for a relevance score. There is no mechanism for the LMM to evaluate whether a sequence of frames jointly contains the needed information — each frame is assessed independently, and the video refinement step stitches the selected frames back together post hoc.

The consequence. For queries that require reasoning about temporal relationships — action sequences, causal chains, before/after comparisons, or event ordering — the frame-level scoring may fail to identify relevant segments because no single frame contains the answer. The paper itself observes this limitation in Appendix F.3 (per-task breakdowns, Tables 6–8): on tasks requiring temporal logic (e.g., Action Order in MLVU, L2-Relation in LongVideoBench), "performance remains stagnant across all methods. Even when provided with query-relevant visual information, model performance does not improve." This suggests either that the selected frames lack the temporal information needed, or that the LMM's temporal reasoning is a bottleneck independent of frame selection — but the paper does not disentangle these explanations.

More subtly, the expansion window wlen is symmetric and fixed — it expands the same number of r-frames before and after each selected frame, regardless of whether the query implies a forward-looking causal relationship (where frames after the key event matter most) or a backward-looking explanatory one (where frames before the key event provide context). For a query like "What caused the explosion?", frames leading up to the event are critical; for "What happened after the explosion?", frames following it are critical. A symmetric window treats both directions equally, potentially including irrelevant context in one direction while missing critical context in the other.

What evidence exists in the paper. The window length ablation (Figure 8) shows that wlen = 0 (no temporal expansion) performs substantially worse than wlen = 2 or wlen = 4, confirming that temporal context matters. However, the paper does not ablate asymmetric windows, nor does it evaluate whether the reward model's second scoring dimension (contextual adjacency) successfully identifies which direction of expansion is needed. The per-task breakdowns in Tables 6–8 show that DIG's gains are concentrated in static perception tasks (PlotQA, NeedleQA, Object Recognition) rather than temporal reasoning tasks (Action Order, Action Count, L2-Relation), which is consistent with the hypothesis that frame-level scoring struggles with temporally extended reasoning — but the paper does not explicitly investigate whether this is a frame selection problem or a model capability problem.

Mitigation status. The paper does not address this limitation. The reward prompt's two-dimensional criterion (asking the LMM to assess "whether it suggests adjacent frames may contain relevant context") is a partial acknowledgment that temporal context matters, but it does not change the fact that each frame is scored independently. A more principled approach would score temporal segments jointly or allow the LMM to specify which adjacent frames are relevant rather than relying on a fixed symmetric window. The paper identifies temporal reasoning as "a fundamental bottleneck" in Appendix F.3 but does not connect this to a limitation of the frame selection pipeline itself.

The Global Query Classification Accuracy Is Low, and Misclassification Cost Is Asymmetric

The assumption or constraint. DIG routes queries to either uniform sampling (global) or the expensive query-aware pipeline (localized) based on an LLM classifier. The paper reports in Table 4 (Appendix D) that global query classification accuracy is substantially lower than localized query accuracy across all tested LLMs. For the model used in main experiments (Qwen3-Next-80B-A3B-Instruct), GQ accuracy is 38.26% on MLVU and 65.76% on VideoMME. This means that on MLVU, over 60% of global queries are misclassified as localized and undergo the expensive pipeline unnecessarily.

The consequence. The paper argues this is acceptable because "this has a negligible impact on final model performance; it primarily incurs a minor computational overhead" (Appendix D). However, the computational overhead is not minor: Table 10 shows that for MLVU, the total frame selection overhead for DIG is 256.3 minutes, of which reward assignment alone accounts for 218.9 minutes. If 60% of global queries are routed to this expensive pipeline, a substantial fraction of that computation is wasted — the pipeline runs on queries where Figure 5 (right panels) shows it provides zero accuracy benefit over uniform sampling.

Furthermore, the misclassification cost is asymmetric: misclassifying a localized query as global (rare, since LQ accuracy is ~90%+) means the system falls back to uniform sampling, which performs poorly on localized queries (as demonstrated in Figure 3 and Figure 5, left panels). Misclassifying a global query as localized (common) means the system burns computation for no accuracy gain. Both types of error matter, but they have different consequences — the first sacrifices accuracy, the second wastes computation. The paper's focus on LQ accuracy alone overlooks the fact that poor GQ accuracy directly undermines one of DIG's claimed advantages: efficiency through routing.

What evidence exists in the paper. Table 4 reports classification accuracies. Table 11 quantifies the efficiency savings from query identification (13.3% on MLVU, 19.9% on VideoMME) compared to applying the expensive pipeline universally. The paper does not report what the actual classification accuracy would need to be to achieve these savings, nor does it report how the savings degrade with lower classification accuracy. The reported savings are based on the LLM's actual classification decisions, so they already incorporate the imperfect GQ accuracy — but this means the savings are substantially lower than what a perfect classifier would achieve. A hypothetical perfect classifier would route 100% of global queries to uniform sampling, potentially doubling the savings on MLVU (since GQ accuracy is only ~38%, meaning ~62% of global query computation is currently wasted).

Mitigation status. The paper acknowledges the low GQ accuracy in Appendix D but dismisses its importance, focusing on LQ accuracy as the "critical metric." It does not propose improvements to the classifier (e.g., fine-tuning on query-type data, using a different prompt, ensembling multiple classifiers) or explore whether the computational overhead of misclassification could be reduced (e.g., by using a cheaper reward model for queries where classification confidence is low). The current design accepts that a majority of global queries will be processed through the expensive pipeline, which partially contradicts the efficiency motivation for the routing architecture.

The Method Is Validated on a Single Family of Math Benchmarks with a Single Model Family

The assumption or constraint. All experiments use the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* (Codey) as the base model. The paper argues in Section 4 that this model is "representative of the capabilities of many contemporary LLMs" and that MATH is appropriate because "test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences." However, the paper does not demonstrate that the findings generalize beyond this specific combination of task domain and model family.

The consequence. Several aspects of the findings could be specific to math reasoning with PaLM 2-S*:

  • The difficulty-dependent behavior of search algorithms (beam search hurting easy problems, helping medium ones) depends on the PRM's calibration and over-optimization characteristics. A PRM trained on a different model's outputs, or on a different task domain (e.g., code generation, scientific reasoning), might exhibit different failure modes — for instance, over-optimization might occur at different difficulty levels or manifest through different degenerate outputs.

  • The revision model's effectiveness depends on the base model's ability to condition on its own incorrect outputs and produce improved answers. This capability is likely model-dependent — some models may be better at in-context self-correction, others may ignore the revision history or become confused by incorrect in-context examples. The paper's revision training procedure (paired incorrect-correct trajectories with edit distance selection) may also be sensitive to the base model's output distribution.

  • The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning with ground-truth answers that can be verified by a grading function. This enables both the PRM training pipeline (Monte Carlo rollouts with automatic correctness checking) and the difficulty estimation (pass@1 computation). In domains without clean correctness signals — open-ended generation, dialogue, creative writing, abstractive summarization — neither the PRM nor the difficulty estimation could be trained or evaluated in the same way.

  • The specific numerical thresholds (five difficulty quintiles, 2048 samples for difficulty estimation, the budget ranges where beam search degrades) are likely benchmark- and model-specific. A practitioner deploying on a different task or model would need to recalibrate these parameters, but the paper provides no guidance on how.

What evidence exists in the paper. The paper provides zero cross-domain or cross-model evaluation. The "representative" claim about PaLM 2-S* is stated without evidence (Section 4), and the paper does not test the framework on, for example, code generation benchmarks (HumanEval, MBPP), scientific reasoning (ARC, GPQA), or any non-math domain. The FLOPs-matched comparison uses a second PaLM 2 model with approximately 14× more parameters, but this stays within the same model family and training paradigm. The paper does not test whether the findings replicate on models from different families (e.g., LLaMA, Qwen, DeepSeek) or with different pretraining recipes.

Mitigation status. The paper does not claim cross-domain or cross-model generality, but it does not sufficiently caveat the single-domain, single-model nature of the evidence either. The "representative" claim in Section 4 implies a level of generality that the experiments do not support. A more careful framing would acknowledge that the findings are demonstrated for math reasoning with PaLM 2-S* and that extension to other domains and models is a direction for future work. The paper's title and abstract do not restrict the scope to math, which could lead practitioners to overestimate the evidence for generality.

Revisions and Search Are Studied Independently, Never Combined

The assumption or constraint. The paper studies two mechanisms for test-time compute — PRM-guided search (Section 5) and iterative revisions (Section 6) — as separate, independently evaluated strategies. The compute-optimal allocation selects between search algorithms (best-of-N, beam search, lookahead search) or between sequential-to-parallel ratios (for revisions). The paper never combines PRM search with the revision model as the proposal distribution. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The two mechanisms have complementary strengths demonstrated in the paper: revisions are most effective on easy problems where the model's outputs need local refinement (Section 6, Figure 7), while PRM search is most effective on medium-hard problems where the model needs to explore different solution strategies (Section 5, Figure 3 right). Combining them — for instance, using the revision model within beam search so that each beam expansion conditions on previous rejected branches, or using the PRM to score revision steps rather than independently generated solutions — could yield performance beyond either mechanism alone.

The current results therefore represent a lower bound on what a fully integrated system could achieve. The difficulty-dependent routing that the compute-optimal policy performs (best-of-N for easy, beam search for medium) could be seen as a crude approximation of combining both mechanisms — but it switches between them per question rather than integrating them within a single question's processing. A question that benefits from both exploration (to find the right high-level approach) and refinement (to polish the solution once the approach is found) cannot receive both types of computation under the current framework.

What evidence exists in the paper. The paper provides suggestive evidence that combination would be valuable: Figure 7 (right) shows that even for hard questions, a balanced sequential-to-parallel ratio is optimal, implying that both exploration (parallel sampling) and exploitation (sequential refinement) contribute. Figure 3 (right) and Figure 7 (right) together show that the difficulty regimes where search excels (medium-hard) and where revisions excel (easy-medium) partially overlap, creating a region where both mechanisms offer gains. But the paper provides no direct experimental evidence for the combined approach.

Mitigation status. The paper acknowledges this gap in Section 8 as a direction for future work but does not minimize its importance. The separate-study design is a reasonable scoping choice for a first systematic analysis of test-time compute scaling, but it leaves open the central question of whether the two mechanisms are additive, redundant, or synergistic. A practitioner deploying a test-time compute system would likely want to combine the best of both approaches, and the current paper provides only indirect guidance on how to do so.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the frame selection conversation from "what is the best selection strategy?" to "which strategy is best for which query type?" — a reframing analogous to the move from training a single model to training a router or mixture of experts, but applied at the inference-time pipeline level for video understanding. This is an incremental but diagnostically important shift, not a paradigm change: the individual components (DINOv2 feature extraction, LMM-based relevance scoring, temporal segment expansion) are not individually novel, but their integration into a query-conditional routing architecture establishes a new design principle that future frame selection methods must contend with.

The magnitude of this shift lies in two specific contributions:

First, the finding that query-aware frame selection has a domain of applicability. The field's default assumption — encoded in the design of AKS, Q-Frame, Bolt, Frame-Voyager, and most other recent methods — has been that more query-awareness is uniformly better. The paper's demonstration that for global queries, adding query-aware selection provides zero benefit (Figure 5, right panels: overlapping performance curves between uniform sampling and DIG's expensive pipeline across all frame counts) and sometimes active harm (when expansion windows or selection heuristics strip away necessary context) constitutes a corrective to this assumption. This is not merely a negative result — it is a boundary condition that designers of future frame selection systems must respect. The research implication is that evaluation of frame selection methods should always report per-query-type breakdowns, and that aggregate benchmark scores that conflate global and localized queries can be misleading — a methodological contribution that the paper models in its per-task tables (Tables 6–8).

Second, the reconciliation of conflicting intuitions about uniform sampling. The community has long observed that uniform sampling sometimes works surprisingly well (leading some practitioners to dismiss more complex selection methods as unnecessary) and sometimes fails dramatically (leading others to advocate for query-aware approaches exclusively). The paper resolves this contradiction by showing that both camps are correct, but for different query types. Uniform sampling is genuinely optimal for global queries — its broad temporal coverage is exactly what holistic understanding requires, and targeted selection would remove necessary context. It is genuinely harmful for localized queries — its signal-to-noise ratio collapses as frame count increases. This diagnosis converts a confusing set of contradictory anecdotal experiences into a coherent picture with clear boundary conditions, analogous to how the test-time compute scaling laws paper (Snell et al., 2024) resolved contradictions about whether self-correction and search "work" by showing they work only on specific difficulty tiers.

Third, the identification of temporal continuity as a critical design axis. Prior frame selection methods (AKS, Q-Frame, Bolt) select individual, isolated keyframes and feed them directly to the LMM. The paper's window-length ablation (Figure 8) provides the first systematic evidence that isolated keyframes are substantially worse than keyframes expanded into continuous segments, with wlen = 0 underperforming wlen = 2 by approximately 9 percentage points on MLVU at 256 frames. This finding redirects attention from "how do we pick the best individual frames?" to "how do we identify and extract query-relevant temporal segments?" — a shift in the unit of selection from frame to segment. This aligns with the broader video understanding intuition that actions, causal sequences, and events unfold over time, but the paper provides the concrete empirical showing that the gap is large enough to dominate other design choices.

Research directions that become more attractive:

  • Segment-level relevance scoring — rather than scoring individual frames and expanding post-hoc, design reward models that assess entire temporal segments jointly, leveraging the LMM's understanding of temporal dynamics.
  • Query-type-adaptive architectures — the routing principle demonstrated for frame selection could apply to other video processing decisions (frame rate, resolution, use of audio/subtitles, compression aggressiveness).
  • Difficulty or scope estimation for video queries — the paper's query classifier is binary and imperfect (38% GQ accuracy on MLVU); a continuous query-scope estimator could enable finer-grained routing and potentially handle intermediate query types that the binary classification struggles with.

Research directions that become less attractive:

  • Ever-more-complex fixed strategies applied uniformly. The paper's demonstration that AKS and Q-Frame degrade below uniform sampling at high frame counts (Table 1) suggests that adding complexity to a single-strategy approach has diminishing and eventually negative returns — the bottleneck is not strategy sophistication but strategy-to-query matching. Research effort is better spent on routing and adaptation mechanisms than on refining a one-size-fits-all selection algorithm.
  • Pure sparse keyframe selection without temporal expansion. The wlen ablation establishes that temporal context is essential; methods that select isolated frames and concatenate them are architecturally limited regardless of how good the individual frame selection is.

Follow-Up Research This Work Enables

Direct comparison of LMM-based vs. CLIP-based relevance scoring in a controlled setting. The paper shows that LMM-based reward assignment outperforms CLIPScore (Table 2), but the comparison confounds multiple differences: the LMM sees the full query text and frame metadata (duration, timestamp), reasons via chain-of-thought, and uses a two-dimensional criterion. A controlled experiment would isolate which factor drives the gain — for instance, comparing CLIPScore against (a) CLIPScore with the query expanded via an LLM to include referent descriptions, (b) the LMM's reward without chain-of-thought, (c) the LMM's reward without the contextual adjacency dimension, and (d) a smaller LMM (e.g., Qwen2.5-VL-3B) to test whether scale or instruction-tuning matters more. The paper's modular architecture makes this experiment straightforward: swap the reward assignment module and measure downstream accuracy on localized queries from MLVU and VideoMME across frame budgets of 32, 128, and 256. The expected outcome is that chain-of-thought reasoning and the two-dimensional criterion each contribute independently, suggesting that the reward prompt design is as important as the model scale.

Query-scope estimation as a continuous regression problem. The paper's binary global-vs-localized classifier has low GQ accuracy (38% on MLVU with Qwen3-Next-80B-A3B, Table 4), which the paper accepts because misclassified global queries only incur computational overhead. But a continuous query-scope estimate — for instance, the fraction of the video's duration that contains information relevant to answering the query — could enable graduated routing: queries with scope >0.8 (holistic) get pure uniform sampling; queries with scope <0.2 (highly localized) get the full DIG pipeline; intermediate queries get a cheaper variant (e.g., CAFS + CLIPScore rewards + smaller wlen). A strong experiment would annotate 500+ queries from MLVU and VideoMME with continuous scope estimates (using temporal annotations of which segments contain answer-relevant information, already available in some benchmarks), fine-tune a small classifier (e.g., Llama-3.1-8B with LoRA) to predict scope from query text alone, and measure whether the graduated routing achieves better accuracy-efficiency Pareto frontiers than the binary routing on held-out queries. The paper's existing efficiency analysis (Table 11) provides the baseline: 13.3% savings on MLVU with binary routing; a graduated scheme might achieve 20–25% by avoiding the expensive pipeline for intermediate queries where a cheaper variant suffices.

Stress-testing temporal expansion with asymmetric windows and query-dependent direction. The paper demonstrates that wlen = 2 outperforms wlen = 0 (Figure 8), confirming that temporal context matters, but uses symmetric windows that expand equally forward and backward. Many video queries have temporal asymmetry: "What caused X?" requires pre-event context; "What happened after X?" requires post-event context; "How does X change over the video?" requires both but perhaps not equally. A natural extension would modify the reward model to output not just a scalar relevance score but also a temporal direction indication (e.g., "relevant context is primarily before this frame" or "primarily after"), then use asymmetric expansion windows. The experiment would compare symmetric wlen = 2 (the current default) against asymmetric variants on MLVU and VideoMME queries that have been manually annotated for temporal direction. The paper's per-task analysis (Tables 6–8) already shows that temporal reasoning tasks benefit less from DIG than static perception tasks — this experiment could determine whether the gap is due to insufficient temporal context (which asymmetric windows would address) or fundamental LMM reasoning limitations (which no frame selection can fix).

Scaling the reward model independently: cost-quality Pareto frontier. The paper demonstrates that a stronger reward model (Qwen2.5-VL-32B) improves the performance of a weaker inference model (Qwen2.5-VL-7B) in Table 2. This decoupling is architecturally significant but unexplored in terms of the cost-quality tradeoff: how much does reward model quality matter relative to its computational cost? A systematic experiment would sweep reward models from very cheap (CLIPScore, requiring ~0 TFLOPs per frame beyond feature extraction) through progressively more expensive LMMs (Qwen2.5-VL-0.5B, 1.5B, 3B, 7B, 32B, 72B), measuring both downstream accuracy (on MLVU and LongVideoBench at 128 and 256 frames) and total FLOPs (including reward assignment overhead). The expected result is a concave Pareto frontier with diminishing returns — the jump from CLIPScore to Qwen2.5-VL-3B might capture 70% of the maximum gain at 20% of the cost of the 32B model. This would provide practical guidance for deployment: if reward model inference is 10× cheaper than final inference, even a modest improvement in reward quality is worth the cost; if it's comparable, the sweet spot shifts toward cheaper reward models.

Does CAFS's prominence threshold transfer across video domains? The paper uses a fixed prominence threshold of 0.1 "found effective through empirical observation" (Section 4.2) on MLVU, LongVideoBench, and VideoMME — all relatively high-production-value videos with identifiable scene boundaries. The threshold determines how many r-frames CAFS produces (and thus downstream reward assignment cost), but its sensitivity to video domain is unknown. A stress test would evaluate CAFS with thresholds swept from 0.05 to 0.5 on videos with different editing characteristics: (a) minimally edited egocentric video (e.g., Ego4D), where scene boundaries are gradual rather than sharp cuts; (b) rapidly edited content (e.g., movie trailers, TikTok-style compilations with multiple cuts per second); (c) static-camera footage (e.g., surveillance, lecture capture). The hypothesis is that the optimal prominence threshold is domain-dependent — lower thresholds for gradual-transition domains (to avoid missing subtle scene changes), higher thresholds for rapid-cut domains (to avoid generating excessive r-frames). The experiment would also test whether an adaptive threshold (e.g., set to the 80th percentile of the distance distribution for each video) performs robustly across domains without per-domain tuning, which would make DIG more deployment-ready.

Practical Applications and Downstream Use Cases

Cost-efficient batch video QA for content moderation and annotation. Organizations that process large video libraries — social media platforms reviewing flagged content, media companies generating metadata for archival footage, educational platforms grading video-based assignments — routinely run video QA models across thousands of videos. The paper's efficiency analysis (Figure 9) shows that uniform sampling saturates at ~62.5% accuracy on LongVideoBench regardless of additional FLOPs, while DIG surpasses this ceiling at ~720 TFLOPs and continues improving. In a batch processing setting where queries are predominantly localized (e.g., "Does this video contain violence at any point?", "What product is shown at 3:15?"), DIG's 3–8 percentage point accuracy gains over uniform sampling (Table 1: +7.68% on MLVU at 32 frames, +3.51% on LongVideoBench at 32 frames with Qwen2.5-VL-7B) translate directly to fewer missed violations, more accurate metadata, and reduced manual review costs. The query identification module's efficiency savings (13–20% reduction in processing time on MLVU and VideoMME per Table 11) are particularly relevant here: if 20% of queries in a moderation pipeline are global (e.g., "Summarize this video's content"), routing them to cheap uniform sampling saves substantial computation without accuracy loss. The modular design also enables asymmetric deployment: a powerful LMM (Qwen2.5-VL-32B) performs CAFS and reward assignment as an offline preprocessing step, while a cheaper LMM (Qwen2.5-VL-7B) runs final inference, as validated in Table 2.

On-device or edge video understanding with small models. Mobile devices, AR headsets, and robots increasingly run small LMMs locally for privacy and latency reasons. These small models are more sensitive to frame selection quality because their reasoning capabilities are limited — they cannot compensate for poor input selection by "thinking harder." The paper's experiments with Qwen2.5-VL-7B (Tables 1, 5) show that DIG's gains are larger in absolute terms for the 7B model than for the 32B model on some benchmarks — for example, on MLVU at 32 frames, DIG improves the 7B model by +7.68 points (59.52 → 67.20) versus +8.78 points for the 32B model (61.91 → 70.69), but the relative improvement is larger for the smaller model. This suggests that small, on-device video LMMs may benefit disproportionately from query-aware preprocessing. A practical deployment architecture would run the query classifier (using a tiny LLM or distilled classifier) and CAFS (using a lightweight feature extractor like MobileNet- or EfficientNet-based DINO variants) on-device, then send only the selected r-frames to a cloud-based reward assignment service, with final inference performed on-device from the refined video. The paper does not test this split-deployment scenario directly, but its modular design and the decoupling result (Table 2: reward model and inference model can differ) make it immediately feasible to prototype.

Video-grounded dialogue and assistive agents. In applications like video-based customer support ("Watch this recording and tell me what error code appeared at 2:30"), video-grounded tutoring ("What mistake did the student make when solving this problem on the whiteboard at the end?"), or accessibility tools ("Describe what happened after I left the room"), users ask a sequence of questions about a shared video. The queries in such sessions are predominantly localized — users ask about specific moments, objects, or actions — making DIG's specialized pipeline directly applicable. Moreover, the shared video context in multi-turn dialogue creates an amortization opportunity: CAFS r-frames are video-dependent, not query-dependent, so they need to be computed once per video and can be reused across all queries in the session. The reward assignment and video refinement are query-specific but operate on the cached r-frames. This means the dominant cost (reward assignment, ~85% of DIG's overhead per Table 10) remains per-query, but the CAFS cost is amortized. A practical system for video-grounded dialogue would precompute DINOv2 features and r-frames at video ingestion time, maintain them in a session cache, and run only query identification + reward assignment + video refinement per user question. The paper's runtime breakdown (Table 10) enables a precise cost model for this architecture.

When to Prefer This Method

The paper explicitly positions DIG against two named alternatives — uniform sampling and existing query-aware methods (AKS, Q-Frame) — and the evidence supports clear decision rules based on query type distribution, frame budget, and available LMM quality:

  • Prefer DIG's routed architecture when: (1) The query distribution contains a mix of global and localized queries (the routing saves 13–20% computation on mixed datasets per Table 11, with zero accuracy cost for global queries per Figure 5 right panels). (2) You are deploying at high frame budgets (128+, up to 768) where uniform sampling's signal-to-noise ratio degrades for localized queries (Figure 3) and competing methods degrade below uniform (Table 1, Table 5 — AKS and Q-Frame fall below the uniform baseline at 128+ frames on LongVideoBench). (3) You have access to a strong LMM for reward assignment that is substantially better than CLIPScore at relevance assessment (Table 2: LMM-based rewards outperform CLIPScore by 1–3 percentage points at high frame counts), and the reward model can be decoupled from the inference model. (4) The domain has identifiable visual scene boundaries that CAFS can detect — the method was validated on edited, high-production-value video (MLVU, LongVideoBench, VideoMME), and its behavior on egocentric, surveillance, or live-stream content is untested.

  • Prefer uniform sampling when: (1) The query distribution is predominantly global (e.g., video summarization benchmarks, topic classification tasks) — Figure 5 (right) shows uniform and DIG overlap almost perfectly, so the overhead buys nothing. (2) The frame budget is very low (≤16 frames) where uniform sampling's temporal coverage advantage matters more than targeted precision, and the relative gains from DIG shrink (Table 1: DIG improves over uniform by +2.38 points at 8 frames with Qwen2.5-VL-32B on LongVideoBench vs. +3.97 at 32 frames). (3) Latency constraints are tight — DIG's frame selection overhead is 256.3 minutes on MLVU (Table 10) versus ~3–9 minutes for LMM inference (Table 9), making it impractical for real-time or near-real-time applications.

  • Prefer existing query-aware methods (AKS) only when: Frame budgets are moderate (32–64 frames) and queries are predominantly localized, where AKS is competitive with DIG (Table 1: AKS at 32 frames on MLVU with Qwen2.5-VL-7B achieves 65.07% vs. DIG's 67.20%) but computationally cheaper. However, the paper shows AKS degrades at higher frame counts, so this preference is fragile to budget scaling.

A practical decision heuristic: if your deployment processes >80% localized queries at >64 frames with a capable LMM available for preprocessing, implement DIG; if your queries are >30% global or you're at <32 frames, uniform sampling is sufficient; if you're at moderate frames and compute-constrained, AKS is a reasonable fallback but will not scale with future context window improvements.