Unified Spatio-Temporal Token Scoring for Efficient Video VLMs

ArXiv: 2603.18004

🎯 Pitch

Standard video VLMs waste 62% of their computation on redundant visual tokens, yet simply dropping half of them randomly causes severe performance collapse. By training a lightweight scorer to identify which patches are spatially uninteresting and temporally redundant, STTS achieves that 62% efficiency gain with almost no accuracy loss—and even outperforms the baseline on long videos when the saved compute is reinvested into more frames.

1. Executive Summary

This paper introduces Spatio-Temporal Token Scoring (STTS), a lightweight plug-in module that prunes vision tokens across both the ViT encoder and LLM in video VLMs without text-conditioned selection or token merging. Built on the Molmo2 backbone with a SigLIP 2 ViT and Qwen3-4B LLM, STTS learns to score tokens along two complementary axes—spatial saliency via downstream task gradients (bias-injected attention that prioritizes foreground objects over static backgrounds) and temporal redundancy via an auxiliary neighboring-frame cosine similarity loss (guiding the scorer to drop near-identical patches across adjacent frames)—then packs surviving tokens into dense tensors for genuine hardware acceleration. At 50% pruning, STTS achieves a 62% improvement in training and inference throughput with only a 0.7% drop in average performance across 13 short and long video QA benchmarks, and applying test-time scaling to pruned models yields 0.5–1% gains on long-video benchmarks by trading spatial redundancy for additional temporal frames, establishing that architecture-wide token pruning remains highly effective only when both spatial and temporal scoring signals are learned jointly rather than applied as static heuristics.

2. Context and Motivation

The Core Problem: Video VLMs Are Computationally Bottlenecked by Vision Tokens

The fundamental challenge this paper addresses is deceptively simple: processing video with modern vision-language models (VLMs) requires handling an enormous number of visual tokens, and most of those tokens are redundant across time. Each frame of video is decomposed into hundreds of patch tokens by a vision transformer (ViT), and when you string dozens or hundreds of frames together to capture temporal context, the resulting token sequences become quadratically expensive under transformer attention. This creates a computational bottleneck that hits twice—first in the ViT encoder that processes all frames independently, and again in the large language model (LLM) that must attend over all encoded visual tokens alongside text tokens.

This gap matters for several concrete reasons that the paper surfaces in Section 1:

The quadratic attention curse in video. A single image processed through a ViT might produce, say, 256 patch tokens. But video reasoning requires multiple frames—if you sample 64 frames (a common budget for long-video understanding), you now have 64 × 256 = 16,384 visual tokens. Under standard self-attention, the computational cost scales with $O(N^2)$ where $N$ is the total sequence length. This means that doubling the number of frames roughly quadruples the attention cost, not doubles it. The paper's architecture—a SigLIP 2 So400M/14 384px Image ViT connected to a Qwen3-4B LLM via a connector module—exemplifies this: while the ViT processes frames independently (linear cost per frame), the aggregate token load still becomes prohibitive as frame counts increase, and the LLM's cross-attention over all visual tokens compounds the problem.

Training throughput degradation. The quadratic attention cost directly reduces the number of training examples that can be processed per unit time. In the paper's experimental setup (Section 4.3, Table 6 in Appendix B), a 128-frame baseline configuration processes only 0.1932 batches per second on 8 H100 GPUs. When you increase to 256 frames—necessary for longer videos—throughput plummets to 0.0549 batches per second (a ~3.5× drop in throughput for only 2× more frames). This slowdown fundamentally limits how quickly video VLM models can be iterated on during research and development, and it makes large-scale video pretraining economically challenging.

Inference latency in deployment. The same quadratic attention bottleneck affects inference, where latency constraints are often tighter than training throughput constraints. A model that takes seconds to process a short video clip is impractical for real-time applications like video assistants, autonomous driving perception, or live video Q&A. The paper's efficiency results (Section 4.3, Figure 5) show that at 256 frames, the baseline achieves only 0.2641 evaluation iterations per second on MLVU—meaning a single video takes nearly 4 seconds to process, excluding the LLM generation phase.

The form factor mismatch. Most ViT backbones in modern VLMs are pretrained on images, not videos. Image ViTs are optimized to extract rich spatial features from single frames. When you feed them 64 frames independently, they process each frame with equal computational effort—even though adjacent frames in a video are often nearly identical except for small regions of motion. A ViT has no built-in mechanism to say "this frame is 95% similar to the previous one, so I can skip most of the computation." The paper's insight is that this temporal redundancy is a massive optimization opportunity that existing models leave on the table.

The Gap in Existing Approaches: Pruning Happens in the Wrong Places or the Wrong Way

Prior work on token pruning for VLMs falls into two broad categories, and the paper argues that both fail to solve the video efficiency problem holistically. Understanding why requires examining the VLM architecture (Figure 2):

Input Video → ViT Encoder (layers 0..L) → Connector → LLM → Text Output
                   ↑                                 ↑
          Pre-/In-ViT pruning                  Post-ViT pruning
         (existing works)                      (existing works)

The ViT consumes raw video frames and produces encoded visual features. The connector (in Molmo2, a 3×3 spatial pooling followed by a projection) bridges the visual features to the LLM's token space. The LLM consumes both visual tokens (from the connector) and text tokens (the user's question) to generate an answer.

Prior approaches only prune at one of the two points marked above, never both.

Category 1: Pre-ViT and In-ViT Pruning (Section 2.1)

These methods reduce token count before or during ViT processing. The paper cites several representative examples:

SPViT (Kong et al., 2022) aggregates redundant ViT tokens into "package tokens"—essentially grouping similar patches and representing them with a single token. ToMe (Bolya et al., 2023) merges similar tokens via bipartite matching at each ViT layer, reducing the total token count progressively through the network. FastViT (Vasu et al., 2023) employs structural reparameterization and token mixing for efficiency. These methods are effective for image-level tasks because they exploit spatial redundancy: a blue sky region has many identical patches that can be merged without information loss.

However, the paper identifies three critical limitations when these methods are applied to video VLMs:

1. No temporal awareness. These methods prune or merge tokens based only on within-frame similarity. They cannot detect that a patch in frame $t$ is near-identical to the same patch in frame $t+1$ because they process frames independently or collapse the temporal dimension into a single batch dimension. A static background region gets the same treatment in every frame, even though the information is repeated 64 times.

2. Designed for unimodal perception, not multimodal reasoning. SPViT, ToMe, and similar methods are evaluated on tasks like image classification, object detection, or semantic segmentation—where the pruning criteria (often attention scores or feature similarity) directly align with the downstream task. In a VLM, the downstream objective is language generation conditioned on vision. A patch that looks visually "unimportant" (e.g., a small clock showing the time) might be semantically critical for answering a temporal reasoning question ("What time did the event occur?"). Vision-only pruning criteria cannot capture this cross-modal importance.

3. Merge-based approaches create hybrid tokens. ToMe and similar methods don't just drop tokens—they merge them, creating new tokens that are averages or concatenations of multiple source patches. This can corrupt fine-grained spatial information that the LLM needs for tasks like "point to the object that moved left" or "describe the expression on the person's face." The paper argues (Section 2.1) that "merging tokens without sufficient structural or temporal awareness... often compromises the fine-grained details required for complex video reasoning."

DToP (Tang et al., 2023) takes a different approach: early-exiting, where "easy" tokens (e.g., background patches) stop being processed at shallow ViT layers. VLTP (Chen et al., 2025) uses a learned pruning decoder to select important tokens at specific ViT layers. Run-Length Tokenization (Choudhury et al., 2024) identifies temporally redundant patches before they enter the ViT by run-length encoding across frames—this is one of the few prior works that addresses temporal redundancy. However, the paper notes that these methods "are typically demonstrated on vision-only tasks like segmentation or action classification and have not been extended to downstream VLM, and specifically video-LLM, applications."

In summary, Category 1 methods solve the ViT bottleneck but don't account for the LLM's needs, and they're blind to temporal structure.

Category 2: Post-ViT Pruning (Section 2.2)

These methods leave the ViT untouched and prune tokens after the visual encoder but before the LLM. The paper cites a range of approaches:

Pooling-based methods. FreeVA (Wu, 2024) provides training-free temporal token aggregation—essentially averaging features across frames before passing them to the LLM. LLaVA-PruMerge (Shang et al., 2025) leverages CLIP-ViT attention scores to identify important tokens and merges the rest. Matryoshka-based approaches (Cai et al., 2025; Hu et al., 2024) compress vision tokens into multiple granularity levels using nested token representations, allowing the LLM to operate on coarser tokens.

Merging-based methods. PruneVid (Huang et al., 2024), STTM (Hyun et al., 2025), and HoliTom (Shao et al., 2025) merge tokens both spatially and temporally after the ViT, reducing the number of tokens passed to the LLM. FastVid (Shen et al., 2025) incorporates temporal segmentation into its merging process to preserve event boundaries.

Text-conditioned selection. VCM (Luo et al., 2025) and Video-XL-Pro (Liu et al., 2025) employ query-based selector modules that perform cross-attention between visual tokens and text tokens, learning to keep only the visual information relevant to the specific question being asked.

The paper identifies one critical limitation that applies to all of these methods:

"A critical limitation of all these methods is that they prune after the ViT. Consequently, the ViT must still process every frame from the input video, creating a significant computational bottleneck, especially for long inputs."

This is the central insight that motivates the paper's approach. Even if you throw away 80% of the ViT's output tokens before the LLM sees them, the ViT has already spent full compute processing all frames. In a standard video VLM, the ViT accounts for a substantial fraction of total FLOPs—especially when processing many frames. Leaving the ViT untouched means leaving significant efficiency gains unrealized.

Furthermore, the paper notes that many post-ViT methods "rely on complex merging algorithms or text-conditioned modules." Text-conditioned selection requires running cross-attention between the visual tokens and the question text, which adds its own computational overhead. Merging algorithms (bipartite matching, clustering) are also non-trivial operations that don't come for free.

A Deeper Look at the Dual Bottleneck

To understand why the gap matters, consider the computational flow in a typical video VLM like Molmo2:

• ViT processing: Each frame $f \in \{1, \ldots, T\}$ passes through all ViT layers independently (except for optional cross-frame attention in some architectures). If the ViT has $L$ layers and each layer's self-attention costs $O(N^2 \cdot D)$ per frame (where $N$ is patches per frame and $D$ is hidden dimension), the total ViT cost is approximately $O(T \cdot L \cdot N^2 \cdot D)$. This is linear in $T$ because frames are processed independently, but the constant factor is large and grows with $N^2$.

• LLM processing: The connector maps the $T \times N_{\text{pooled}}$ visual tokens into the LLM's input space. The LLM then attends over both visual tokens and text tokens (the user's question and any system prompt). If the LLM has $M$ layers and the total sequence length is $S = T \cdot N_{\text{pooled}} + S_{\text{text}}$, the LLM's self-attention costs $O(M \cdot S^2 \cdot D_{\text{LLM}})$. This is quadratic in the number of frames because visual tokens from all frames appear in the same sequence.

• In the Molmo2 baseline: With 64 frames and 3×3 spatial pooling (reducing 256 ViT patches to ~28 pooled patches per frame), the visual token count is $64 \times 28 = 1,792$. Combined with text tokens (up to 2048 in the paper's setup), the total LLM sequence length can exceed 3,800. The $S^2$ term in the LLM's attention makes this expensive.

Category 1 methods (pre-/in-ViT) address the first bullet—they reduce $N$ or $T$ within the ViT, which also indirectly helps the LLM since fewer tokens emerge. But they do so without awareness of what the LLM needs. Category 2 methods (post-ViT) address the second bullet—they reduce $S$ for the LLM—but leave the ViT cost untouched.

The paper's position is that neither approach alone is sufficient. You need to prune within the ViT (to reduce ViT computation) AND reduce tokens flowing to the LLM (to reduce LLM computation), and the pruning decisions must be informed by both spatial redundancy (intra-frame) and temporal redundancy (inter-frame), as well as the downstream multimodal task objective.

How This Paper Positions Itself

The paper frames STTS as filling a specific architectural gap: it is a unified, architecture-wide pruning module that operates at a single insertion point (after ViT layer $l$) but affects the entire downstream computation—the remaining ViT layers AND the LLM. This is captured in the paper's four stated contributions (Section 1):

1. Unified token pruning across ViT and LLM. Unlike prior work that prunes in only one location, STTS makes pruning decisions inside the ViT (after layer $l = 3$) and permanently removes those tokens from the sequence. The reduced token count then propagates naturally through the remaining ViT layers (saving compute) and through the connector to the LLM (saving more compute). No separate pruning step is needed post-ViT. The paper emphasizes that this is done "without requiring significant architectural modifications, text-conditioned selection, or complex merging algorithms"—it's a lightweight module (a self-attention pooler + 3-layer MLP) that can be inserted into any VLM with a standard ViT + LLM architecture.

2. Dual-axis scoring that learns both spatial and temporal importance. STTS scores tokens along two complementary axes:

• Spatial saliency is learned implicitly through the downstream task loss. The scores are injected as an attention bias into the subsequent ViT layer (Section 3.2), making them differentiable with respect to the final VLM training objective. This means the LLM's gradients flow back through the ViT and teach the scorer which visual regions matter for answering questions—without requiring explicit text conditioning.
• Temporal redundancy is supervised explicitly through an auxiliary loss based on neighboring-frame cosine similarity (Section 3.4). This provides a direct training signal that the downstream task loss might not supply (the paper finds that the LLM "seemed indifferent to fine-grained temporal redundancy" when optimized solely with the primary loss). The scorer learns to assign low importance to patches that are nearly identical to their counterparts in the preceding frame.

The dual-axis design is the paper's key methodological innovation. Prior methods addressed one axis or the other—spatial pruning within ViTs, temporal aggregation post-ViT—but none learned both simultaneously from a single, end-to-end trained scorer.

3. Pruning without merging. Unlike ToMe, SPViT, PruneVid, STTM, or HoliTom, STTS performs hard pruning (discarding tokens entirely) rather than merging. The paper argues this is important for video VLM tasks because merged tokens can smooth over fine-grained details that matter for reasoning. When STTS drops a token, it's gone—the surviving tokens are the original, unmodified ViT features, preserving their representational precision for the LLM.

4. Actual hardware efficiency via packing. Hard pruning creates a sparse, ragged tensor because different frames have different numbers of surviving tokens. Standard deep learning frameworks (PyTorch/TensorFlow) cannot accelerate sparse tensors efficiently for batched matrix multiplications. The paper introduces a first-fit descending packing algorithm (Section 3.3, Appendix Algorithm 1) that consolidates surviving tokens from multiple frames into a compact, dense batch. This generates genuine wall-clock speedups—not just theoretical FLOP reductions—and the paper quantifies these carefully in Section 4.3.

The paper also explicitly positions itself relative to the Molmo2 backbone (Section 3). Molmo2 already applies 3×3 spatial pooling to reduce raw ViT patch tokens before the LLM. STTS's 3-layer MLP scorer operates on these pooled features and further reduces token count beyond what pooling alone achieves. The paper notes that STTS "imposes no architecture-specific constraints, requiring only a standard ViT encoder and a token-to-LLM pathway—both ubiquitous in modern VLMs," suggesting broad applicability beyond Molmo2.

The Broader Significance

Beyond the immediate efficiency gains, the paper touches on several implications that motivate the work:

Enabling longer video contexts. If you can prune 50% of visual tokens with minimal accuracy loss, you can double the number of frames you process within the same computational budget. The paper demonstrates this via test-time scaling (Section 5.3, Table 3): a model trained with 50% pruning on 64 frames is evaluated on 128 frames, using the same total visual token budget as the unpruned baseline. This yields 0.5–1% accuracy improvements on long-video QA benchmarks. The framing is that STTS "effectively trades off spatial redundancy for temporal density," converting frequentist sampling limitations into an architectural optimization problem.

Democratizing video VLM research. Training and evaluating video VLMs currently requires substantial GPU resources—the paper's baseline training uses 8 H100 GPUs and processes only ~0.19 batches per second at 128 frames. By improving throughput by 62%, STTS makes video VLM experimentation more accessible to researchers with limited compute budgets. This is not just about cost savings; it's about enabling faster iteration cycles and more comprehensive hyperparameter sweeps.

A path toward real-time video understanding. The inference speedup results (Section 4.3, Table 7) show that at 256 frames with 50% pruning, throughput increases to 0.587 evaluations per second—a 2.22× improvement over the baseline. While this is still far from real-time (30 FPS), it substantially narrows the gap and demonstrates that architecture-aware pruning can push video VLMs closer to interactive latencies.

3. Technical Approach

3.1 Reader Orientation

STTS is a lightweight, plug-in neural module that learns to score every visual token in a video according to its importance for downstream question answering, then removes the least-important tokens before they consume computation in later layers. The problem it solves is that video VLMs waste enormous compute processing redundant visual information—static backgrounds that repeat across frames, nearly identical patches in adjacent frames, and regions irrelevant to the specific question being asked—and the solution is a single, end-to-end trainable scorer inserted early in the ViT that permanently prunes tokens from the entire downstream architecture based on dual spatial-and-temporal importance signals.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a sequential pipeline:

1. ViT Encoder (Early Layers, 0 through $l$): The first $l$ layers of the vision transformer process all $T$ video frames independently, producing initial visual features for every patch in every frame. These layers operate at full resolution—no pruning has happened yet—and their outputs serve as the input to the scoring module.

2. STTS Scorer Module (inserted after layer $l$): A compact neural network consisting of a self-attention token pooler followed by a 3-layer MLP. It takes pooled visual features from the current and previous frame (concatenated) as input and outputs a single importance score for each $w \times w$ spatial region. The scores are injected as an attention bias into ViT layer $l+1$ (making them differentiable with respect to the downstream task loss) and are also supervised by an auxiliary temporal similarity loss.

3. Hard Pruning + Packing (immediately after layer $l+1$): The bottom-$k\%$ scoring tokens (according to the STTS scores) are permanently discarded. Because this produces a sparse, ragged tensor with different numbers of surviving tokens per frame, a first-fit descending packing algorithm consolidates the survivors into a compact, dense batch that enables genuine hardware acceleration in subsequent ViT layers.

4. ViT Encoder (Remaining Layers, $l+2$ onward): The remaining ViT layers process only the packed, pruned token sequences. Because many tokens have been removed, these layers run significantly faster than they would on the full input. A block-diagonal attention mask ensures that tokens from different original frames do not attend to each other despite being packed into the same batch entry.

5. Connector + LLM: The final ViT output (which already has fewer tokens due to pruning) passes through the standard connector module (3×3 spatial pooling + projection in Molmo2) and enters the LLM alongside text tokens. The LLM's self-attention cost is quadratic in total sequence length, so the token reduction from pruning yields substantial savings here as well.

Information flows in a strict forward pass: raw video frames → early ViT layers → STTS scorer produces importance scores → attention bias flows into layer $l+1$ → hard pruning removes low-scoring tokens → packing densifies the remaining tokens → later ViT layers process the packed batch → connector maps to LLM token space → LLM generates answer from visual + text tokens. The auxiliary temporal loss provides a training signal that branches off at the scorer's output, while the main task loss flows backward through the entire network including through the attention bias connection, teaching the scorer which tokens matter for answering questions.

3.3 Roadmap for the Deep Dive

• First, the constrained optimization formulation (Equation 1), which defines what "optimal pruning" means mathematically—minimizing task loss subject to a hard sparsity constraint—and establishes the formal problem STTS solves.
• Second, the scorer architecture and spatial scoring mechanism, because understanding how scores are produced and how they become trainable via attention bias injection is foundational to everything downstream.
• Third, the hard pruning operation and packing algorithm, since this is what converts learned scores into actual computational savings—without packing, pruning would produce no wall-clock speedup.
• Fourth, the temporal auxiliary loss, which provides the explicit supervision signal for temporal redundancy that the downstream task loss alone fails to supply.
• Fifth, the end-to-end training objective and integration with the Molmo2 backbone, covering how all components are trained jointly, what hyperparameters are used, and how STTS fits into the existing VLM architecture without requiring structural changes.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-and-methods paper whose core idea is that a single, lightweight learned scorer inserted early in the ViT can prune tokens across the entire VLM architecture—reducing compute in both the ViT and LLM—if it is trained with both spatial supervision (from downstream task gradients) and temporal supervision (from an explicit inter-frame similarity loss).

The Constrained Optimization Formulation

The paper frames token pruning as a constrained optimization problem in Section 3. Let $N_{\text{total}} = T \times N$ be the total number of initial patch tokens across all $T$ frames, where each frame has $N$ patches. The goal is to find model parameters $\theta$ that minimize the total loss $\mathcal{L}$ while satisfying a hard sparsity constraint on the number of retained tokens:

$\min_{\theta} \mathcal{L}(\theta) \quad \text{subject to} \quad \|M\|_0 \leq (1 - k\%) N_{\text{total}}$

where $M \in \{0, 1\}^{T \times N}$ is a binary mask indicating which tokens survive pruning ($M_{t,i} = 1$ if the $i$-th token of frame $t$ is retained, $0$ otherwise), $\|M\|_0$ counts the number of ones in the mask (the total surviving tokens), and $k$ is the pruning ratio expressed as a percentage (e.g., $k = 50$ means at most $50\%$ of tokens survive).

What this computes: the optimization seeks the best possible model parameters such that the total number of surviving visual tokens is at most $(1 - k\%)$ of the original count. The mask $M$ is determined by the scorer's output—the bottom $k\%$ of scored tokens are assigned $M_{t,i} = 0$ and removed. Because $M$ itself depends on the scorer's parameters (which are part of $\theta$), this is a joint optimization over both the base model parameters and the scoring policy. The loss $\mathcal{L}$ encompasses both the primary VLM task loss $\mathcal{L}_{\text{task}}$ (typically next-token prediction on the correct answer) and the auxiliary temporal loss introduced in Section 3.4.

Why this form: casting token pruning as a constrained optimization rather than a post-hoc heuristic or separate preprocessing step is the key conceptual move. It means the pruning policy is not fixed a priori—it is learned jointly with the task objective, so the model can discover which tokens to keep based on actual downstream utility. The $L_0$ constraint (counting non-zero entries of $M$) directly encodes the computational budget: we want to spend attention compute only on the most informative $(1 - k\%)$ fraction of tokens. Prior pruning methods either used fixed heuristics (e.g., always keep high-attention patches) that don't adapt to the task, or applied pruning only to the ViT or only post-ViT, violating the constraint that the reduction should benefit the entire pipeline. The paper's formulation is architecture-wide because $M$ is applied inside the ViT and the resulting sparse sequence propagates through all subsequent layers naturally, satisfying the constraint at the point of maximum leverage.

Scorer Architecture and Spatial Scoring via Attention Bias

Input preparation and token pooling. Given an input $X \in \mathbb{R}^{T \times N \times D}$ representing $T$ video frames, each with $N$ patch tokens of hidden dimension $D$, the features are first passed through ViT layers $0, 1, \ldots, l$. The output at layer $l$, denoted $X_l$, is then spatially pooled with width $w = 3$ to reduce the spatial dimension from $N$ to $N/w^2$:

$X_l^{\text{pooled}} = \text{Pool}_{w \times w}(X_l)$

where $\text{Pool}_{w \times w}$ averages or max-pools non-overlapping $w \times w$ blocks of patch tokens. For the Molmo2 backbone, the ViT produces 27×27 patches from 384px images (the SigLIP 2 So400M/14 384px ViT uses 14px patch size, yielding $384/14 \approx 27$ patches per side, so $N = 27 \times 27 = 729$). After 3×3 pooling, this becomes $9 \times 9 = 81$ pooled tokens per frame, matching the dimensionality that the Molmo2 connector later uses. The pooling reduces the number of tokens the scorer must process, making it computationally lightweight.

Temporal context via concatenation. To provide the scorer with information about how frames relate to each other temporally, the input for each frame $t \geq 1$ is the concatenation of its own pooled features with those of the preceding frame $t-1$:

$\text{Input}_t^{\text{scorer}} = [X_{l,t}^{\text{pooled}} \;\|\; X_{l,t-1}^{\text{pooled}}] \in \mathbb{R}^{(N/w^2) \times 2D}$

where $[\cdot \;\|\; \cdot]$ denotes concatenation along the feature dimension. For the first frame ($t = 0$), there is no preceding frame, so it is concatenated with a zero tensor of the same shape. The paper explicitly states that "for the first frame ($t = 0$), we concatenate it with a zero-padding tensor; its scores are ignored during pruning, as it lacks a preceding frame for temporal comparison"—meaning the first frame is always kept intact regardless of its scores.

Token pooler: self-attention for feature aggregation. The concatenated features pass through a self-attention layer (the "Token Pooler") before scoring. This layer allows each pooled patch to attend to all other pooled patches within the same frame (across the two-frame concatenation), enabling the model to aggregate contextual information before making per-patch scoring decisions. The paper does not specify the number of attention heads or the exact dimensionality of this layer, but given that it operates on $N/w^2 = 81$ tokens per frame with hidden dimension $2D$, it is a standard multi-head self-attention block with residual connection and layer normalization.

3-layer MLP scorer. The output of the token pooler feeds into a 3-layer MLP that produces a single scalar score for each of the $N/w^2$ pooled spatial regions:

$S_t^{\text{pooled}} = \text{MLP}(\text{TokenPooler}(\text{Input}_t^{\text{scorer}})) \in \mathbb{R}^{N/w^2}$

where $S_t^{\text{pooled}}$ contains one importance score per $w \times w$ spatial block in frame $t$. A lower score indicates lower importance—the token is more likely to be pruned. The MLP consists of three fully-connected layers with non-linear activations (the paper does not specify the activation function or intermediate dimensions, but standard choices would be GELU activations with hidden dimensions in the range of $D$ to $4D$).

Score expansion to original resolution. To apply these scores at the original patch resolution (necessary because the ViT operates on individual patches, not pooled blocks), the pooled scores are expanded back to the full $N$ patches:

$S_t^{\text{expanded}} = \text{Expand}_{w \times w}(S_t^{\text{pooled}}) \in \mathbb{R}^N$

The expansion assigns the same score to every patch within a given $w \times w$ block. So if the scorer predicts a low importance score for the top-left 3×3 block of a frame, all 9 individual patches in that block inherit that low score and will be pruned together. This block-level granularity is a design choice: individual patch scores would be higher-resolution but would require the scorer to process $N$ tokens rather than $N/w^2$, increasing its computational cost.

Attention bias injection for end-to-end training. The critical mechanism that makes the scorer end-to-end trainable is the injection of the expanded scores as an additive bias into the attention matrix of the subsequent ViT layer ($l+1$):

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + S\right) V$

where $Q, K, V \in \mathbb{R}^{T \times N \times d_k}$ are the query, key, and value projections at layer $l+1$, $d_k$ is the per-head dimension, and $S \in \mathbb{R}^{T \times N}$ is the logarithm of the expanded scores (the paper states "the logarithm of these expanded scores, denoted as $S$, is then injected as a bias"). The log transform is important because attention scores are exponentiated inside the softmax; adding $\log(\text{score})$ as a bias means that the final attention weight is proportional to $\text{score} \times \exp(QK^T/\sqrt{d_k})$, effectively down-weighting tokens with low importance scores regardless of their query-key similarity.

What this computes: for each attention head at layer $l+1$, every query token computes attention weights over all key tokens. Without the bias, these weights depend only on dot-product similarity. With the bias, tokens that the scorer deems unimportant receive a negative additive penalty in the pre-softmax logits, reducing their influence on the output of the attention layer. Because the scores $S$ are functions of the scorer's parameters (which are part of $\theta$), the gradient of the final task loss $\mathcal{L}_{\text{task}}$ with respect to the scorer's parameters flows through this bias term, teaching the scorer to assign higher scores to tokens that are useful for downstream reasoning and lower scores to tokens that are not.

Why this form: the bias injection mechanism is elegantly minimal. It does not require modifying the ViT architecture—the attention computation is unchanged except for an added bias term. It does not require text conditioning or cross-attention with the LLM. The downstream gradients from $\mathcal{L}_{\text{task}}$ provide implicit supervision about which visual regions matter: if a region contains information that helps predict the correct answer, backpropagation through the LLM, connector, and ViT will increase the scorer's output for patches in that region. Conversely, patches that are irrelevant or distracting will receive downward gradient pressure. This is a form of implicit, task-driven attention that emerges from the optimization rather than being explicitly programmed. The alternative—text-conditioned selection via cross-attention (as in VCM or Video-XL-Pro)—would require an additional module that computes compatibility between every visual token and every text token, adding significant computational overhead that STTS avoids.

Hard Pruning and the Packing Algorithm

Hard pruning operation. Following ViT layer $l+1$ (the layer where scores were injected as bias), the system performs hard pruning: all tokens corresponding to the bottom $k\%$ of the expanded scores $S_t^{\text{expanded}}$ are permanently removed from the sequence. Formally, for each frame $t$, the set of retained token indices is:

$\mathcal{R}_t = \{i \in \{1, \ldots, N\} : S_{t,i}^{\text{expanded}} \geq \tau_t\}$

where $\tau_t$ is the $k$-th percentile of scores within frame $t$. The number of retained tokens per frame is $|\mathcal{R}_t| = (1 - k\%) N$ in expectation, but because scores vary across frames, the actual retention count is non-uniform: a static frame with high temporal redundancy might retain only $20\%$ of its tokens ($80\%$ pruned), while a dynamic frame with unique motion might retain $90\%$ ($10\%$ pruned).

The sparsity problem. This non-uniform pruning creates a computational challenge. Standard deep learning frameworks (PyTorch, TensorFlow) expect batched tensors to be dense and rectangular—every entry in the batch must have the same spatial dimensions. If frame 1 retains 200 tokens and frame 2 retains 50 tokens, simply masking out the pruned positions (setting them to zero) does not reduce computation because the matrix multiplications still operate on the full padded tensor. The paper states: "Because deep learning frameworks like PyTorch rely on dense, uniform tensors for efficient batched matrix multiplications, merely masking the pruned tokens yields no computational savings."

The packing solution: first-fit descending bin packing. To convert the sparse, variable-length token sequences into a dense tensor that enables actual hardware acceleration, the paper employs a bin-packing algorithm (Algorithm 1 in Appendix D). The algorithm treats the batch of frames $(T, N, D)$ as $T$ sequences of variable-length token vectors and packs them into a smaller number of "bins" (new frame slots in a packed batch) of fixed capacity $N$:

1. Compute valid token counts: for each frame $t$, count the number of surviving tokens $C_{\text{valid}}[t] = |\mathcal{R}_t|$.
2. Sort by descending count: sort the frames from most surviving tokens to fewest. This is the "descending" part of first-fit descending—it places the largest items first, which is a well-known heuristic that produces near-optimal packings.
3. Iterative packing: for each frame $i$ in the sorted order, find the first existing bin (packed frame) that has enough remaining capacity to hold all $C_{\text{valid}}[i]$ tokens. If no existing bin fits, create a new bin. The capacity constraint is that the total tokens in a bin cannot exceed $N$ (the original per-frame token count), ensuring each packed frame can use the same tensor dimensions as an original frame.
4. Record assignments: maintain arrays $P_{\text{assign}}[i] = j$ (original frame $i$ maps to packed frame $j$) and $P_{\text{offset}}[i]$ (the starting position of frame $i$'s tokens within packed frame $j$).
5. Scatter tokens: copy the surviving tokens from each original frame into the packed tensor at their assigned positions, and construct a block-diagonal attention mask that allows tokens within a packed frame to attend only to other tokens from the same original frame.

What this computes: given $T$ original frames with variable numbers of surviving tokens, the packing algorithm produces a new tensor of shape $(T', N, D)$ where $T' \leq T$ and each of the $T'$ packed frames contains tokens from one or more original frames concatenated along the spatial dimension. The total number of packed frames $T'$ is minimized, which maximizes the computational savings in subsequent ViT layers: instead of processing $T$ frames (many of which are sparse), the ViT processes $T'$ denser frames.

Why this form: first-fit descending is chosen because it is simple, runs in $O(T^2)$ time (where $T$ is the number of frames, typically 64), and produces packings that are close to optimal for bin-packing. The paper notes that "the overhead is negligible because $T \ll N$"—the quadratic cost of sorting and bin-packing over 64 frames is dwarfed by the $O(N^2)$ cost of self-attention on hundreds of tokens per frame. The block-diagonal attention mask is essential for correctness: it prevents tokens from different original frames from attending to each other within a packed batch entry, preserving the independence of the per-frame ViT processing. Without this mask, the ViT would blend information across frames that were never meant to interact, corrupting the temporal structure. The packing approach differs fundamentally from token merging methods (ToMe, SPViT): merging combines multiple tokens into a single averaged token, losing fine-grained information; packing keeps tokens intact and merely rearranges them spatially for computational efficiency, preserving all information in the surviving tokens.

Auxiliary Loss for Temporal Scoring

The problem: downstream loss is insufficient for temporal pruning. While the attention bias injection provides a path for spatial saliency gradients (the LLM can teach the scorer which regions matter for answering questions), the paper found in preliminary experiments that "the LLM seemed indifferent to fine-grained temporal redundancy." This means that optimizing solely with $\mathcal{L}_{\text{task}}$ does not provide an adequate signal for identifying which patches are temporally redundant—the model might learn to drop some tokens, but without explicit guidance it cannot reliably distinguish between a patch that is genuinely repeated across frames (and can be safely pruned) versus a patch that appears similar but contains subtle motion or change (and should be kept).

Neighboring-frame cosine similarity as temporal redundancy ground truth. To provide explicit supervision, the paper computes a "ground truth" temporal redundancy signal directly from the ViT features at layer $l$. First, the features $X_l$ are pooled with the same $w \times w$ pooling as used in the scorer, then L2-normalized:

$\hat{X}_{l,t}^{(i)} = \frac{X_{l,t}^{(i), \text{pooled}}}{\|X_{l,t}^{(i), \text{pooled}}\|_2}$

where $X_{l,t}^{(i), \text{pooled}}$ is the pooled feature vector for the $i$-th spatial block of frame $t$ at layer $l$, and $\hat{X}_{l,t}^{(i)}$ is its L2-normalized version (unit length). The cosine similarity between corresponding patches in adjacent frames is then:

$\text{CosSim}(\hat{X}_{l,t}^{(i)}, \hat{X}_{l,t+1}^{(i)}) = \hat{X}_{l,t}^{(i)} \cdot \hat{X}_{l,t+1}^{(i)}$

What this computes: for each spatial block $i$ in frame $t$, the cosine similarity with the same spatial block in frame $t+1$ measures how similar the ViT's internal representations are at those two positions. Because the features have been L2-normalized, the dot product is exactly the cosine of the angle between the two feature vectors, ranging from $-1$ (opposite directions) to $1$ (identical). A value close to $1$ means the ViT encodes the two patches nearly identically—they are temporally redundant. A lower value indicates the patches differ, likely due to motion, occlusion, or lighting changes.

The temporal auxiliary loss. The scorer is trained to predict low importance scores for patches with high temporal similarity. The target for patch $i$ in frame $t$ (for $t \geq 1$) is $1 - \text{CosSim}(\hat{X}_{l,t-1}^{(i)}, \hat{X}_{l,t}^{(i)})$, which is close to $0$ when the frames are similar (low redundancy = keep) and close to $1$ when they differ (high redundancy = prune). The mean squared error between the scorer's output $S_t^{(i)}$ and this target forms the per-element loss:

$\mathcal{L}_{\text{sim}}(t, i) = \left(S_t^{(i)} - \left(1 - \text{CosSim}(\hat{X}_{l,t-1}^{(i)}, \hat{X}_{l,t}^{(i)})\right)\right)^2$

where $S_t^{(i)}$ is the score for the $i$-th pooled block of frame $t$ produced by the STTS scorer (before expansion). For $t = 0$ (the first frame), $\mathcal{L}_{\text{sim}}(0, i) = 0$ for all $i$ because there is no preceding frame to compare against, and the first frame is never pruned anyway.

The total auxiliary loss is the average over all frames and all spatial blocks:

$\mathcal{L}_{\text{sim}} = \frac{w^2}{T N} \sum_{t=0}^{T-1} \sum_{i=0}^{N-1} \mathcal{L}_{\text{sim}}(t, i)$

where the scaling factor $w^2/(T N)$ accounts for the fact that there are only $N/w^2$ scored blocks per frame rather than $N$ patches (since scores are pooled), and the loss is computed per-block.

What this computes: the auxiliary loss penalizes the scorer when its predicted importance score $S_t^{(i)}$ deviates from the complement of the cosine similarity. If two corresponding patches across adjacent frames are nearly identical (cosine similarity close to 1), the target $1 - \text{CosSim}$ is close to 0, so the scorer should output a low score—signal that this patch is redundant and can be pruned. If the patches differ significantly (cosine similarity moderate or low), the target is higher, so the scorer should output a higher score—signal that this patch contains temporally novel information worth keeping.

Why this form: the cosine similarity on L2-normalized features is chosen because it is invariant to the magnitude of the feature vectors—it captures only directional similarity, which is a more robust measure of representational redundancy than Euclidean distance or unnormalized dot product. Using layer $l$ features (rather than raw pixels or early-layer features) means the similarity is computed in a semantically meaningful space where the ViT has already aggregated some contextual information. The complement $1 - \text{CosSim}$ transforms similarity into a "redundancy" signal suitable for pruning: highly similar patches get low importance scores. MSE loss is appropriate because the scorer outputs are continuous values that should be calibrated to similarity magnitudes—binary cross-entropy would be wrong here because there's no binary classification decision, just a continuous mapping from similarity to importance. The MSE also has the property that large deviations (predicting high importance for a highly redundant patch, or low importance for a changing patch) are penalized quadratically, encouraging the scorer to produce scores that are well-calibrated to the actual temporal dynamics.

The crucial design insight: why an explicit temporal loss is necessary. The paper's ablation (Table 2, "STTS (No Aux)") shows that removing this auxiliary loss causes a dramatic performance drop—the model actually performs worse than random pruning. This validates the assumption that the downstream VLM task loss alone cannot drive effective temporal pruning. The reason is likely that the LLM's gradients through the connector and ViT are too coarse to provide per-patch temporal feedback: the LLM sees the aggregated visual representation after pooling and projection, so it has limited ability to distinguish between "this patch was pruned because it was temporally redundant" versus "this patch was pruned because it was spatially irrelevant." The auxiliary loss provides a direct, per-patch signal that explicitly encodes temporal redundancy, enabling the scorer to separate the two axes of pruning.

End-to-End Training Objective and Integration with Molmo2

Combined loss function. The final training objective is the sum of the primary VLM task loss and the temporal auxiliary loss:

$\mathcal{L} = \mathcal{L}_{\text{task}} + \frac{w^2}{T N} \sum_{t=0}^{T-1} \sum_{i=0}^{N-1} \mathcal{L}_{\text{sim}}(t, i)$

where $\mathcal{L}_{\text{task}}$ is the standard autoregressive next-token prediction loss used to train the LLM on the correct answer given the visual and text context. The two losses are simply added—there is no balancing hyperparameter mentioned in the paper, suggesting that the scales are naturally compatible or that a coefficient of $1.0$ was found to work well without tuning.

What this computes: during each training step, the model processes a batch of video-question-answer triples. The input video passes through the early ViT layers, the STTS scorer produces importance scores, the bias is injected into layer $l+1$, the tokens are pruned and packed, the remaining ViT layers process the packed batch, the connector maps to LLM token space, and the LLM generates a probability distribution over answer tokens. $\mathcal{L}_{\text{task}}$ measures the cross-entropy between the predicted distribution and the ground-truth answer tokens. Simultaneously, the features at layer $l$ are used to compute the pairwise cosine similarities between adjacent frames, and $\mathcal{L}_{\text{sim}}$ penalizes the scorer for poorly calibrated temporal importance predictions. Both losses are backpropagated jointly, updating all parameters: the LLM, the ViT, the connector, and the STTS scorer.

Training configuration and hyperparameters. The paper provides specific details in Section 4.1:

• Base model: Molmo2 with SigLIP 2 So400M/14 384px Image ViT and Qwen3-4B LLM, starting from the same pretrained video captioner checkpoint as Molmo2.
• Training duration: 6,250 steps with effective batch size 128 (batch size of 64 with sequence packing averaging 2 samples per batch).
• Learning rates: differential rates across components—$1 \times 10^{-5}$ for the LLM, $5 \times 10^{-6}$ for the ViT and projector (connector), and $1 \times 10^{-4}$ for the STTS module. The higher learning rate for STTS makes sense because it is randomly initialized while the other components are pretrained, requiring more aggressive updates to converge.
• Schedule: cosine learning rate decay with 200 warmup steps.
• STTS insertion layer: always $l = 3$, meaning the scorer is inserted after the 3rd ViT layer and the bias is injected into layer 4, with hard pruning occurring after layer 4.
• Spatial pooling width: $w = 3$, matching Molmo2's existing 3×3 spatial pooling between the ViT and LLM.
• Video preprocessing: attempt to sample at 2 FPS; if this yields more than 64 frames, fall back to uniformly sampling 64 frames across the entire video. The final frame of the video is always included.
• LLM attention: bidirectional attention across all vision tokens in the LLM (meaning vision tokens can attend to all other vision tokens, not just causally).
• Sequence packing: Molmo2's standard configuration that concatenates multiple samples into one longer sequence before the LLM, averaging 2 samples per batch, yielding an effective batch size of 128.

Integration with the Molmo2 backbone. STTS is designed to be a drop-in module that requires no architectural modifications to the base VLM. The insertion point is between ViT layers 3 and 4 (after layer 3, the scorer operates; before layer 4, the bias is injected; after layer 4, pruning occurs). The remaining layers (4 through the final ViT layer, typically 27 layers for SigLIP 2 So400M) operate on the packed, pruned tokens. The connector and LLM are completely unchanged—they receive fewer visual tokens (because tokens were removed in the ViT) and process them identically to the baseline. The first frame of every video is always kept intact (its scores from STTS are ignored during pruning), ensuring that the model always has at least one complete reference frame.

Why this design over alternatives:

• Why insert at layer $l = 3$ and not earlier or later? The paper ablated this choice (Section 5.2, Figure 6) and found that performance improves monotonically with depth up to $l = 3$. Inserting at $l = 0$ (before any ViT processing) hurts significantly (~1% average drop) because the ViT hasn't had a chance to build contextualized representations—the scorer is operating on nearly raw patch embeddings that lack the semantic content needed to distinguish important from unimportant regions. Inserting at $l = 2$ is slightly worse than $l = 3$ (~0.2% drop), suggesting that the features at layer 3 contain sufficient information for reliable scoring. Going deeper than $l = 3$ would reduce the computational benefits because fewer ViT layers would benefit from the pruning.

• Why a 3-layer MLP and not something more complex? The scorer needs to be lightweight because it processes every frame in every video. A 3-layer MLP with self-attention pooling is sufficient because the scoring task—predicting a single importance value per pooled region based on current and previous frame features—is a relatively simple regression/ranking problem. More complex architectures (e.g., a full transformer decoder, cross-attention with text) would add significant overhead that could eat into the efficiency gains from pruning.

• Why hard pruning instead of soft weighting? An alternative would be to use the scores as continuous attention weights without actually removing tokens, only down-weighting their influence. But this provides no computational savings because the tokens still participate in matrix multiplications. Hard pruning physically removes tokens, reducing the dimensionality of subsequent operations and yielding genuine speedups. The trade-off is that hard pruning is non-differentiable (the thresholding operation has zero gradient almost everywhere). However, the scores are trained through the attention bias in layer $l+1$ (which IS differentiable) before the hard pruning decision is made, so the scorer receives gradient information about which tokens were useful in that layer. After pruning, there is no gradient path from later layers to the pruned tokens, but this is acceptable because those tokens have been permanently removed—there's nothing to optimize for them.

• Why train with pruned sequences rather than adding pruning post-hoc? The paper's comparison with inference-only baselines (Table 5 in Appendix C) shows that applying pruning only at inference time (without training the model on pruned inputs) performs substantially worse—the "Heuristic [Inference Only]" and "ToMe [Inference Only]" variants score 59.1 and 59.2 average, compared to 62.3 for STTS. This makes intuitive sense: if the ViT and LLM were never trained to handle sparse, pruned visual inputs, they may over-rely on tokens that are suddenly missing at test time. Joint training ensures the entire model adapts to the pruned token distribution.

• Why an additive bias in attention rather than a gating mechanism? The additive bias in the softmax is a minimal intervention: it reweights attention without requiring additional parameters in the attention computation itself. A gating mechanism (multiplying token values by learned scalars before the attention output) would be another option, but additive bias is simpler and directly operates in the space that matters—the attention weights—rather than post-hoc scaling the values.

4. Key Insights and Innovations

Innovation 1: Architecture-Wide Pruning from a Single Insertion Point as a Unified Efficiency Paradigm

The dominant assumption in prior work on video VLM efficiency was that pruning must happen either inside the ViT OR between the ViT and LLM—never both from a single decision point. Pre-/in-ViT methods like ToMe (Bolya et al., 2023), SPViT (Kong et al., 2022), and DToP (Tang et al., 2023) reduce token counts during visual encoding but leave the LLM's input size unchanged from the ViT's perspective. Post-ViT methods like PruneVid (Huang et al., 2024), STTM (Hyun et al., 2025), and Video-XL-Pro (Liu et al., 2025) prune or merge tokens after the ViT but before the LLM, leaving the ViT encoder to process every frame at full resolution. These two paradigms evolved independently because they address different computational bottlenecks—ViT throughput versus LLM sequence length—and combining them would seemingly require separate pruning mechanisms at two different architectural locations, each with its own selection criteria, hyperparameters, and training signals.

STTS upends this assumption with a deceptively simple architectural move: insert a single learned scorer at one early ViT layer and let the pruning decision propagate naturally through the entire downstream computation. There is no separate post-ViT pruning step, no text-conditioned selector between the connector and LLM, and no coordination mechanism between ViT-pruning and LLM-pruning policies. The model makes pruning decisions based on features at layer $l = 3$, permanently removes tokens after layer $l+1 = 4$, and the reduced token count automatically benefits every subsequent ViT layer AND the LLM—because the tokens are physically gone from the sequence, not merely down-weighted or masked.

This is a fundamental reframing of the efficiency problem, not an incremental improvement. Prior work treated ViT pruning and LLM pruning as separate sub-problems requiring separate solutions. STTS recognizes them as a single problem—"which visual tokens does the downstream task actually need?"—with a single solution that happens to reduce compute in both places because of the feedforward architecture. The conceptual elegance is that STTS doesn't need to know about the LLM at all; it only needs to know about temporal redundancy (via auxiliary loss) and spatial saliency (via downstream gradients that naturally flow through the ViT to the scorer). The propagation of efficiency gains to the LLM is an emergent property of the architecture, not an explicitly engineered mechanism.

The significance extends beyond the method itself. It establishes a design principle for future VLM architectures: pruning decisions should be made as early as possible in the pipeline, and they should be permanent (not soft-masked or later-reconsidered) so that ALL downstream components benefit. This principle challenges the prevailing post-ViT pruning paradigm, where the ViT is treated as a fixed, unoptimizable black box whose full output must be computed before any reduction can occur. The paper's evidence that this principle works—50% pruning yields 62% efficiency gains with only 0.7% average accuracy loss (Table 1), and the ViT sees proportional speedups at training time (Table 6, Appendix B)—suggests that post-ViT-only methods are leaving substantial efficiency on the table by not addressing the ViT bottleneck.

A subtle implication: because STTS operates at a fixed ViT depth ($l = 3$) rather than progressively pruning across layers (as ToMe does), it makes a single, irrevocable decision about each token's fate after only four layers of visual processing. This is arguably riskier than progressive pruning, which can revisit earlier decisions. Yet it works, suggesting that the early ViT layers in a pretrained image encoder already extract sufficient information to make reliable keep-or-drop decisions—a finding with implications for how we think about ViT layer functions and the minimal depth needed for visual saliency estimation.

Innovation 2: The Dual-Axis Scoring Design as a Resolution to the "LLM Is Indifferent to Temporal Redundancy" Problem

The paper surfaces a diagnostic finding that would be easy to overlook but has deep implications: the downstream VLM task loss, on its own, provides inadequate supervision for identifying temporally redundant tokens. The evidence is stark: removing the auxiliary temporal loss causes STTS to perform worse than random pruning (Table 2, "STTS (No Aux)": 60.0 average vs. 61.4 for Random), meaning the model trained solely with downstream gradients is not just failing to improve over random—it's actively learning a harmful pruning policy that degrades below an unlearned baseline. The paper's characterization is that "the LLM seemed indifferent to fine-grained temporal redundancy" during preliminary experiments.

This is a genuinely novel diagnostic finding, not just a hyperparameter ablation. It reveals a fundamental misalignment in video VLMs: the LLM's training objective (next-token prediction on answer text) provides gradients that identify spatially important regions (objects, text, faces that matter for answering questions) but fails to identify temporally redundant regions (static backgrounds, motionless objects that repeat across frames). The reason is structural: the LLM sees aggregated visual features after spatial pooling and projection, so its gradient signal about temporal redundancy is coarse and entangled with spatial importance. A patch that is identical across all 64 frames and a patch that contains subtle motion both contribute to the pooled representation; the LLM can reward or penalize the aggregate, but it cannot easily disentangle which individual patches were worth keeping versus which were merely repeated.

The solution—pairing downstream gradients with an explicit auxiliary loss based on neighboring-frame cosine similarity—is elegant but the innovation lies more in identifying and naming the problem than in the specific loss function. Prior work on temporal token pruning (Run-Length Tokenization, FreeVA) used static heuristics for temporal redundancy without questioning whether task-driven learning could suffice. This paper demonstrates empirically that it cannot, at least not with standard VLM training setups. The implication is that any end-to-end learned temporal pruning method for video VLMs likely requires explicit temporal supervision; relying solely on task gradients will produce spatially aware but temporally blind pruning policies.

The dual-axis design also provides a clean conceptual separation that clarifies why certain pruning strategies succeed or fail. The spatial axis (downstream gradients) answers: "does this patch contain information relevant to answering the question?" The temporal axis (auxiliary loss) answers: "is this patch informationally redundant with what we already have from previous frames?" A token should be kept if it is spatially important OR temporally novel (or both), and pruned only if it is spatially irrelevant AND temporally redundant. This separation makes the scoring behavior interpretable—a property the paper exploits in its visualizations (Figures 1, 7) to show that STTS keeps foreground objects even in static scenes (spatially important despite temporal redundancy) and prunes backgrounds that repeat across frames (neither spatially important nor temporally novel).

This is an incremental advance in the sense that cosine-similarity-based temporal pruning heuristics existed before (e.g., Run-Length Tokenization), and end-to-end learned spatial pruning existed before (e.g., VLTP with task-driven selection). The advance is in demonstrating that these two signals must be combined through joint learning rather than applied independently, and in diagnosing why task loss alone fails for temporal pruning—a finding that should inform all future work on learned video token pruning.

Innovation 3: Test-Time Scaling for Pruned Models as a "Token Budget Reallocation" Strategy

The paper's test-time scaling (TTS) results (Section 5.3, Table 3) introduce a conceptual move that is more subtle than it first appears: pruning during training enables a different computational tradeoff at inference—trading spatial resolution for temporal density within a fixed token budget. This is not the standard "scale test-time compute to improve accuracy" narrative; it's a budget reallocation argument.

The standard approach to improving long-video understanding is to sample more frames. But sampling more frames increases the total visual token count, which increases computational cost quadratically under attention. The TTS insight is: if you've trained a model to operate effectively with 50% fewer tokens per frame (by pruning spatially redundant patches), you can double the number of frames at inference time while using the SAME total visual token budget as the unpruned baseline. A 50%-pruned model evaluated on 128 frames uses approximately the same number of visual tokens as an unpruned model on 64 frames. But the 128-frame model captures twice the temporal context—and the paper shows this yields 0.5–1% accuracy improvements on long-video QA benchmarks (Table 3, comparing 50% + TTS vs. 0% baseline on Long avg.: 59.4 vs. 59.0).

This is a reframing of the efficiency-accuracy tradeoff. Prior work presented token pruning as a compromise: accept some accuracy loss in exchange for computational savings. STTS + TTS presents it as a conversion: trade spatial precision (which is often wasted on redundant backgrounds anyway) for temporal coverage (which is genuinely scarce in uniformly-sampled long videos). The finding that 50% pruning + TTS can outperform the unpruned baseline on long video benchmarks (Table 3: 30% + TTS achieves 60.1 Long avg. vs. 59.0 for baseline) is particularly striking because it means the spatial redundancy being pruned was not just "not harmful to remove"—it was actively preventing the model from processing enough frames to capture the temporal dynamics needed for long-video reasoning.

The significance is practical and immediate: for any deployment where long-video understanding matters, the paper provides a recipe to improve accuracy without increasing total compute. Train with aggressive pruning ($k = 30\%$ to $50\%$), then at inference time, increase the frame sampling rate proportionally. The computational cost stays constant (same token count), but the temporal information doubles. This is a zero-cost accuracy improvement for long-video applications—a rare claim in efficiency-focused ML research.

This innovation is incremental in the sense that test-time scaling itself is well-known (simply sampling more frames at inference is standard practice). The conceptual contribution is in coupling test-time scaling to training-time pruning and demonstrating that the two interact synergistically rather than independently—the pruning creates "budget headroom" that test-time scaling can fill with additional frames, converting a spatial efficiency technique into a temporal coverage technique.

A limitation worth noting: the TTS gains are modest (0.5–1%) and are evaluated only on long-video benchmarks where temporal coverage is known to be the primary bottleneck. On short-video benchmarks where 64 frames already capture sufficient temporal context, TTS would likely provide diminishing returns. The paper doesn't evaluate TTS on short-video tasks, so the generalizability of this budget-reallocation strategy across video lengths remains an open question.

Innovation 4: The Packing Algorithm as a Bridge from Theoretical FLOP Reduction to Actual Wall-Clock Speedup

The paper identifies and solves a mundane-sounding but practically critical problem that many token pruning papers sidestep: variable-length pruning produces sparse tensors that standard deep learning frameworks cannot accelerate, meaning theoretical FLOP reductions don't translate to actual speedups. This is not a new observation—masking tokens without physical removal is a well-known inefficiency—but the paper's treatment of it as a first-class design constraint rather than an implementation detail is distinctive.

Most token pruning papers report theoretical FLOP reductions (e.g., "reduces FLOPs by 40%") based on counting the number of surviving tokens, without demonstrating that these reductions produce corresponding wall-clock speedups. The gap between theoretical FLOPs and actual throughput is often substantial because masked-out tokens still occupy memory and participate in matrix multiplications (as zeros). Some methods avoid this by pruning uniform numbers of tokens per frame, but this sacrifices the ability to prune aggressively on static frames while preserving tokens on dynamic frames.

STTS's first-fit descending packing algorithm (Section 3.3, Algorithm 1 in Appendix D) converts the sparse, ragged tensor produced by non-uniform pruning into a dense, compact batch that PyTorch can accelerate efficiently. Crucially, it does so while (a) preserving the original ViT features of surviving tokens (no averaging or merging), (b) maintaining correct attention patterns via a block-diagonal mask (tokens from different source frames don't cross-attend), and (c) adding negligible overhead ($O(T^2)$ in number of frames, where $T = 64$ or $128$, dwarfed by $O(N^2)$ attention cost).

The innovation is not the bin-packing algorithm itself (first-fit descending is a textbook algorithm). It's the insistence on demonstrating real speedups rather than theoretical FLOP counts, and the integration of packing as a core component of the training pipeline rather than a post-hoc optimization. Because STTS trains with packing enabled, the ViT and LLM learn to operate on packed token sequences, and the throughput measurements (Figure 5, Tables 6-7) reflect actual training and inference conditions. The paper reports specific numbers: 1.62× training speedup and 1.61× inference speedup at 128 frames with 50% pruning, scaling to 2.25× and 2.22× at 256 frames. These are genuine wall-clock measurements, not estimated FLOP reductions.

This is significant because it sets a methodological standard for token pruning research. A pruning method that reports only theoretical FLOP reductions hasn't actually demonstrated efficiency; it's demonstrated a proxy for efficiency that may not materialize in practice. STTS's thorough throughput profiling—on a specific hardware configuration (8 H100 GPUs, single node), at specific sequence lengths (128 and 256 frames), for both training and inference—provides a template for how efficiency claims should be validated. The finding that the speedup scales favorably with sequence length (2.25× at 256 frames vs. 1.62× at 128 frames, both at 50% pruning) validates the quadratic attention argument and suggests that STTS's benefits are largest precisely where they're most needed—in the long-context regime where the baseline model approaches VRAM limits.

This contribution is incremental in its technical components but fundamental in its methodological implications for the field. The packing algorithm itself could be replaced with other compaction strategies; the point is that the paper treats it as an essential, non-negotiable component of any pruning system that makes non-uniform per-frame pruning decisions, rather than an implementation detail to be hand-waved away.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on 13 video question-answering benchmarks spanning short and long video understanding. Short-video benchmarks include NextQA test (Xiao et al., 2021), Perception-Test test (Patraucean et al., 2023), MVBench test (Li et al., 2024), Tomato test (Shangguan et al., 2025), MotionBench val (Hong et al., 2025), TempCompass test (Liu et al., 2024), and VideoMME test (Fu et al., 2025). Long-video benchmarks include VideoMME-Sub test (Fu et al., 2025), LongVideo val (Wu et al., 2024), LongVideo-Sub val (Wu et al., 2024), MLVU val MCQ (Zhou et al., 2025), LVBench test (Wang et al., 2025), and VideoEvalPro test (Ma et al., 2025). Results are aggregated into Short average (first 7 benchmarks), Long average (last 6), and overall Average (all 13). These benchmarks collectively cover temporal action recognition, fine-grained motion understanding, long-context temporal reasoning, multi-choice and open-ended QA, and egocentric video understanding. The dataset selection is deliberately broad to test whether STTS's pruning is robust across diverse video understanding tasks rather than tuned to a specific benchmark.

Base model. The backbone is Molmo2 (Clark et al., 2026) using a SigLIP 2 So400M/14 384px Image ViT (Tschannen et al., 2025) connected to a Qwen3-4B LLM (Yang et al., 2025) via a connector module. This architecture was chosen because it represents a recent state-of-the-art open-source VLM with publicly available code and data, enabling reproducible experimentation. Training starts from the same pretrained video captioner checkpoint as the original Molmo2 and fine-tunes only on the video QA subset of Molmo2's data mixture for 6,250 steps with effective batch size 128. The paper explicitly validates that this abbreviated training recipe does not compromise the baseline's competitiveness, reporting that the resulting model outperforms strong baselines like Qwen3-VL-4B (81.4 → 83.9 on NextQA, 68.9 → 72.6 on MVBench) and InternVL3.5-8B (63.0 → 67.0 Short avg.), as shown in Table 1.

Metrics. The primary metric is accuracy (%) on each benchmark, computed as the fraction of test questions for which the model's generated answer matches the ground truth. Benchmarks use their native evaluation protocols—some are multiple-choice (where accuracy is match with correct option), others are open-ended (where exact match or equivalent answer matching is used). The paper aggregates per-benchmark accuracies into Short avg. (mean of 7 short-video benchmarks), Long avg. (mean of 6 long-video benchmarks), and Average (mean of all 13). Sub-category benchmarks (VideoMME-Sub, LongVideo-Sub) are included in the long-video average alongside their parent benchmarks, though this means certain datasets are double-counted in the aggregate. For efficiency measurements, the paper reports throughput in batches per second (training) or iterations per second (inference) measured on a single node with 8 H100 GPUs, and speedup as the ratio of STTS throughput to baseline throughput at the same frame count.

Baselines. The paper compares against several categories of baselines:

• External SoTA models: Qwen3-VL-4B (Yang et al., 2025), PLM-8B (Cho et al., 2025), InternVL3.5-8B (Wang et al., 2025), evaluated on the same benchmarks using their publicly reported numbers. These situate the Molmo2 baseline within the broader video VLM landscape.
• Molmo2 baseline (0% pruning): The same architecture and training recipe as STTS but without the pruning module. This is the primary internal baseline against which all STTS variants are compared.
• Random pruning: For each pruning ratio k%, randomly select k% of vision tokens to drop (uniform across frames), with no learned scoring. This establishes a lower bound for how much performance should degrade when tokens are removed without intelligent selection. Evaluated in Table 2 and the degradation analysis (Table 8, Appendix E).
• Heuristic pruning: Prune the top-k% most similar tokens based on neighboring-frame cosine similarity (the same signal used in STTS's auxiliary loss), applied directly without a learned scorer. This tests whether the cosine similarity signal alone—without learned spatial saliency—is sufficient. Evaluated in Table 2 and Table 5 (Appendix C).
• STTS without auxiliary loss ("No Aux"): STTS trained only with the downstream task loss, removing the temporal auxiliary loss. This ablates the necessity of explicit temporal supervision. Evaluated in Table 2.
• ToMe (Token Merging): Applied as an inference-only baseline (ToMe merging at each ViT layer, no training on merged tokens) and as a fully trained baseline (ToMe applied during training, model learns with merged tokens). Adapted to the Molmo2 architecture by applying ToMe within the ViT and passing merged tokens to the LLM. Evaluated in Table 5 (Appendix C).

Generation budget / compute accounting. Computation is measured in two complementary ways for different analyses:

• For accuracy experiments (Tables 1-3, 8): The "budget" is the pruning ratio k%, which controls the fraction of tokens retained. All models are evaluated with the same number of input frames (64 by default). A fair comparison means comparing STTS at k% pruning against the baseline at the same frame count—STTS uses fewer total visual tokens but processes the same number of temporal samples.
• For efficiency experiments (Figure 5, Tables 6-7): The "budget" is wall-clock throughput measured in batches/second (training) or iterations/second (inference) on 8 H100 GPUs. Throughput is measured at fixed frame counts (128 and 256) and batch sizes (2 and 1 respectively) across different pruning ratios, with all models evaluated in the same hardware environment. This captures actual computational savings including the packing algorithm overhead, not theoretical FLOP reductions.
• For test-time scaling experiments (Table 3): The "budget" is the total number of visual tokens—STTS models with k% pruning are evaluated on proportionally more frames (e.g., 50% pruning on 128 frames ≈ same token count as 0% pruning on 64 frames) to compare at equal computational cost.

Cross-validation / statistical protocol. The paper does not employ cross-validation for model selection or hyperparameter tuning. All models are trained once with the specified hyperparameters and evaluated on the standard test sets of each benchmark. The STTS insertion layer depth (l = 3) was selected based on a one-time ablation (Section 5.2, Figure 6) that compared l ∈ {0, 1, 2, 3} and found l = 3 performed best; no deeper layers were tested to preserve computational benefits. The pruning ratio k was treated as a hyperparameter to sweep (30%, 40%, 50%) rather than being tuned per-dataset. A notable omission is the absence of error bars, standard deviations, or confidence intervals on any reported accuracy or throughput numbers—all results are point estimates from single training runs, making it impossible to assess whether the observed differences (especially the 0.2-0.5% gaps between configurations) are statistically significant or within run-to-run variance.

Main Quantitative Results

Video QA Accuracy Across Pruning Ratios

Headline result. At 50% pruning—discarding half of all visual tokens—STTS achieves an average accuracy of 62.3% across 13 video QA benchmarks, compared to 63.0% for the unpruned baseline, a drop of only 0.7% (Table 1). The efficiency gain for this minimal accuracy loss is 1.62× training throughput and 1.61× inference throughput at 128 frames (Figure 5). This is the paper's central tradeoff claim: 62% more efficiency for less than 1% accuracy degradation.

At 30% pruning, STTS matches or exceeds the baseline on several benchmarks: NextQA goes from 83.9 → 84.1, Perception-Test from 78.7 → 79.0, VideoMME from 62.8 → 63.4, and VideoEvalPro from 47.6 → 47.7. The Short avg. drops by only 0.3 points (67.0 → 66.7), and the Long avg. drops by 0.1 points (59.0 → 58.9). These results demonstrate a "sweet spot" where pruning removes noise rather than signal, actually improving performance on several benchmarks by eliminating distracting background tokens that compete for attention.

Difficulty-dependent behavior (Table 1, Figure 1 right subfigure). The relationship between pruning ratio and accuracy is not monotonic:

• At 30% pruning: Average = 62.8 (baseline: 63.0, −0.2)
• At 40% pruning: Average = 62.1 (baseline: 63.0, −0.9)
• At 50% pruning: Average = 62.3 (baseline: 63.0, −0.7)

The 50% configuration outperforms 40% by 0.2 points on average despite discarding more tokens. The paper attributes this to the scorer's dual objectives creating a "borderline" token problem at intermediate pruning ratios: at 40%, the budget accommodates tokens that are neither clearly temporally redundant (insufficient cosine similarity to trigger pruning) nor clearly spatially important (weak gradient support from the LLM). These tokens act as noise. At the more aggressive 50% setting, the scorer is forced to make harder decisions and learns to identify these non-informative tokens for removal, increasing the signal-to-noise ratio of the retained visual input. This pattern is visible across individual benchmarks: on NextQA, 40% pruning drops to 83.6 while 50% recovers to 83.7; on MVBench, 40% drops to 71.8 while 50% recovers to 72.4; on Tomato, 40% drops to 34.6 while 50% recovers to 35.1.

Comparison to external models (Table 1, top rows). The STTS 50% model at 62.3 average outperforms Qwen3-VL-4B (62.7 average is only 0.4 points higher) and InternVL3.5-8B (60.0 average, STTS leads by 2.3 points), while being substantially more efficient than either. Against PLM-8B (61.3 average), STTS 50% leads by 1.0 point. These comparisons are somewhat apples-to-oranges since the external models have different architectures, training data, and frame sampling strategies, but they establish that STTS's pruned Molmo2 remains competitive with state-of-the-art models while reducing compute by 62%.

Short vs. Long Video Performance Analysis

Short-video benchmarks (7 tasks, Table 1). The baseline achieves 67.0 Short avg. STTS at 30% achieves 66.7 (−0.3), at 40% achieves 66.0 (−1.0), and at 50% achieves 66.1 (−0.9). The degradation is slightly larger than for long videos in absolute terms, though still modest. Specific benchmarks show different sensitivities:

• NextQA: Extremely robust—83.9 baseline, 84.1 at 30% (+0.2), 83.7 at 50% (−0.2). This benchmark tests temporal action recognition, suggesting that the actions themselves (the foreground dynamics) are well-preserved by STTS's spatial saliency scoring.
• MVBench: 72.6 baseline, 72.7 at 30% (+0.1), 72.4 at 50% (−0.2). Similarly robust, indicating comprehensive multi-modal video understanding tasks are not heavily impacted by pruning.
• Perception-Test: 78.7 baseline, 79.0 at 30% (+0.3), 77.7 at 50% (−1.0). This diagnostic benchmark for perception shows slightly larger degradation at aggressive pruning, possibly because it tests fine-grained visual discrimination that relies on tokens STTS may incorrectly prune.
• MotionBench: 61.0 baseline, 59.2 at 30% and 40% (−1.8), 58.2 at 50% (−2.8). This benchmark specifically tests fine-grained motion understanding—the largest degradation among all benchmarks, which is expected because motion cues can be subtle and visually similar across frames (challenging the temporal auxiliary loss to identify them as "not redundant").
• Tomato: 36.5 baseline, 35.6 at 30% (−0.9), 35.1 at 50% (−1.4). Temporal reasoning about event ordering shows mild degradation.

Long-video benchmarks (6 tasks, Table 1). The baseline achieves 59.0 Long avg. STTS at 30% achieves 58.9 (−0.1), at 40% achieves 58.2 (−0.8), and at 50% achieves 58.4 (−0.6). The long-video average degrades marginally less than the short-video average, which is somewhat counterintuitive—one might expect long videos to suffer more from pruning because they have sparser temporal sampling and each frame carries more unique information. The paper does not directly explain this, but the test-time scaling results (discussed below) suggest an interpretation: long videos with uniform frame sampling already have minimal temporal redundancy, so the STTS scorer relies more heavily on spatial saliency signals for pruning, which are well-learned from downstream gradients. The short-video benchmarks, with their denser temporal sampling, may have more "borderline" tokens where the scorer's temporal and spatial signals conflict.

• MLVU val MCQ: 70.3 baseline, 69.5 at 30% (−0.8), 68.4 at 50% (−1.9). Multi-task long video understanding shows the largest absolute drop among long-video benchmarks, possibly because MCQ format relies on fine-grained detail recognition across many minutes of video.
• LVBench: 42.0 baseline, 42.6 at 30% (+0.6), 40.5 at 50% (−1.5). Extreme long video understanding (hour-long videos) shows an interesting pattern: 30% pruning actually improves performance, suggesting that pruning helps the model focus on the most temporally informative frames. The larger drop at 50% indicates a threshold where too much temporal context is lost.
• VideoEvalPro: 47.6 baseline, 47.7 at 30% (+0.1), 46.0 at 50% (−1.6). Long video evaluation shows a similar pattern.
• VideoMME-Sub: 67.6 baseline, 68.5 at 30% (+0.9), 67.2 at 50% (−0.4). Subtitle-aware video understanding benefits noticeably from 30% pruning, likely because subtitles themselves provide strong spatial saliency signals that survive pruning.
• LongVideo and LongVideo-Sub: Both show minimal degradation—LongVideo sub drops from 60.9 to 60.1 at 50% (−0.8), LongVideo drops from 61.5 to 61.0 at 50% (−0.5).

Efficiency Gains: Throughput and Speedup

Training throughput (Figure 5, left panel; Table 6). At 128 frames with batch size 2 and a maximum of 2048 text tokens:

• Baseline (0% pruning): 15,670 tokens per instance, 0.1932 batches/second → 1.00× speedup.
• STTS 30%: 12,560 tokens per instance, 0.2478 batches/second → 1.28× speedup.
• STTS 40%: 11,524 tokens per instance, 0.2786 batches/second → 1.44× speedup.
• STTS 50%: 10,486 tokens per instance, 0.3130 batches/second → 1.62× speedup.

The relationship between token reduction and speedup is sub-linear: 50% fewer tokens (15,670 → 10,486 is a 33% reduction, not 50%, because the text tokens remain constant) yields a 62% throughput improvement. This is because the quadratic attention cost dominates—reducing the sequence length by 33% reduces the $O(N^2)$ attention cost by approximately 55% (since $(0.67)^2 \approx 0.45$ of original), which combined with constant-cost operations (MLP layers, normalization) produces the observed 62% overall speedup.

Scaling with frame count (Table 6). At 256 frames with batch size 1:

• Baseline (0%): 25,307 tokens per instance, 0.0549 batches/second → 1.00×.
• STTS 30%: 19,087 tokens per instance, 0.0811 batches/second → 1.48×.
• STTS 40%: 17,013 tokens per instance, 0.0977 batches/second → 1.88×.
• STTS 50%: 14,939 tokens per instance, 0.1233 batches/second → 2.25×.

The speedup ratios are significantly larger at 256 frames than at 128 frames (2.25× vs. 1.62× at 50% pruning), confirming the quadratic attention argument: longer sequences benefit disproportionately from token reduction. The token reduction ratio is also larger at 256 frames (41% fewer tokens) than at 128 frames (33% fewer) because visual tokens constitute a larger fraction of total sequence length when more frames are sampled.

Inference throughput on MLVU (Table 7). The trends mirror training throughput nearly exactly:

• 128 frames: 1.14× (30%), 1.30× (40%), 1.61× (50%) speedup.
• 256 frames: 1.45× (30%), 1.71× (40%), 2.22× (50%) speedup.

The inference speedups are marginally lower than training speedups at 256 frames (2.22× vs. 2.25× at 50% pruning). The paper attributes this to torch.compile optimization during training: static graph execution with uniform sequence lengths (all examples padded identically) maximizes the relative gains from token reduction. Inference handles dynamic sequence lengths during prefill, which introduces slightly different overhead characteristics.

Scorer Pruning vs. Heuristic and Random Baselines

Comparison of pruning methods at 50% pruning (Table 2). This is a critical ablation isolating the contribution of learned scoring:

• Random: Short avg. 65.3, Long avg. 57.5, Average 61.4. This establishes the lower bound—if you simply drop half the tokens uniformly, performance degrades by 1.6 points from baseline (63.0 → 61.4).
• Heuristic: Short avg. 66.0, Long avg. 57.9, Average 62.0. Using neighboring-frame cosine similarity directly without learning improves over Random by 0.6 points, confirming that temporal redundancy is a useful pruning signal. However, the heuristic is static—it cannot adapt to which similar-looking patches are actually important for answering questions.
• STTS (No Aux): Short avg. 64.4, Long avg. 55.5, Average 60.0. Removing the auxiliary temporal loss causes performance to drop below Random by 1.4 points. This is the paper's most striking negative result: task-driven gradients alone produce a scoring policy that is worse than no policy at all. The model learns a harmful pruning strategy—likely because the LLM's gradients can identify spatially important regions but provide misleading or absent signals about which temporally redundant patches to keep.
• STTS (full): Short avg. 66.1, Long avg. 58.4, Average 62.3. The full model outperforms Heuristic by 0.1 points on Short avg. and 0.5 points on Long avg. (0.3 points overall). The advantage is larger on long videos, where the heuristic's reliance on frame-to-frame similarity is most challenged due to sparse temporal sampling (uniform 64-frame sampling from hour-long videos creates large temporal gaps where cosine similarity between adjacent frames may be uninformative). In this regime, the learned spatial saliency signal compensates.

Comparison to ViT-only pruning baselines (Table 5, Appendix C). This ablation compares STTS against Token Merging (ToMe), a well-established ViT pruning method:

• Heuristic [Inference Only]: 59.1 Average. Applying cosine-similarity-based pruning only at inference (no training with pruned sequences) performs substantially worse than the trained Heuristic variant (62.0), confirming that models must be trained with the pruning distribution.
• ToMe [Inference Only]: 59.2 Average. Similarly, applying ToMe only at inference yields poor performance, barely above the heuristic variant.
• ToMe (fully trained): 61.1 Average. Training with ToMe merging substantially improves performance over inference-only ToMe (59.2 → 61.1, +1.9 points), confirming the necessity of joint training. However, STTS (62.3) still outperforms trained ToMe by 1.2 points. The paper attributes this to ToMe's lack of temporal awareness and its merging operation (which averages tokens rather than selectively discarding them), both of which compromise fine-grained spatio-temporal information needed for video reasoning.
• Broken down: STTS leads trained ToMe by 0.5 on Short avg. (66.1 vs. 65.6) and by 1.8 on Long avg. (58.4 vs. 56.6). The larger gap on long videos suggests that temporal awareness becomes more critical when videos are longer and frames are sampled more sparsely.

Test-Time Scaling for Long Video Benchmarks

Motivation. At 50% pruning, STTS uses approximately half the visual tokens per frame compared to the unpruned baseline. The test-time scaling (TTS) experiments ask: if we double the number of frames at inference time, keeping the total visual token count approximately constant, does accuracy improve? This tests whether the spatial redundancy being pruned can be "converted" into additional temporal coverage.

Headline result (Table 3). All three pruning ratios show consistent TTS improvements over their non-TTS counterparts:

• 30% pruning: TTS (92 frames, roughly proportional to keeping token count equal to 64 unpruned frames) improves Long avg. from 58.9 → 60.1 (+1.2 points), outperforming the unpruned baseline (59.0) by 1.1 points. Individual benchmarks show notable gains: LVBench jumps from 42.6 → 44.9 (+2.3), VideoEvalPro from 47.7 → 49.6 (+1.9), LongVideo from 61.1 → 62.7 (+1.6).
• 40% pruning: TTS (107 frames) improves Long avg. from 58.2 → 59.0 (+0.8 points), matching the unpruned baseline (59.0).
• 50% pruning: TTS (128 frames) improves Long avg. from 58.4 → 59.4 (+1.0 points), surpassing the unpruned baseline by 0.4 points. LVBench shows the largest gain: 40.5 → 44.7 (+4.2), nearly recovering the performance lost from aggressive pruning. VideoEvalPro similarly jumps from 46.0 → 49.3 (+3.3).

Key pattern. The TTS gains are concentrated on benchmarks where temporal coverage is the primary bottleneck: LVBench (extreme long video understanding, hour-long videos), VideoEvalPro (long video evaluation), and LongVideo (long-context interleaved understanding). Benchmarks where the 64-frame baseline already captures sufficient temporal context show smaller gains: VideoMME (62.4 → 62.8 at 50% TTS, +0.4) and VideoMME-Sub (67.2 → 69.0 at 50% TTS, +1.8). This pattern validates the interpretation that TTS works by increasing temporal coverage, not by providing redundant information—the gains are largest where the frame sampling rate is most limiting.

Important nuance: constant token budget, not constant compute. The TTS comparison holds the number of visual tokens approximately constant, but the total computational cost may differ. Processing 128 frames with 50% pruning requires the ViT to process twice as many frames (each with half the tokens), which costs approximately the same in the ViT but may cost more in the connector and LLM because the LLM sees the same total number of visual tokens but distributed across more frames with different positional encodings. The paper does not report throughput numbers for the TTS configurations, so whether the "constant token budget" translates to "constant wall-clock time" is unclear.

Ablation Studies and Robustness Checks

Selecting the pruning layer depth (Section 5.2, Figure 6): The injection layer $l$ (where STTS is inserted) strongly affects performance. Training four separate models with $l \in \{0, 1, 2, 3\}$ reveals a monotonic improvement with depth: $l = 3$ outperforms $l = 0$ by approximately 1% average accuracy, and $l = 2$ by approximately 0.2% (exact values not tabulated, shown only as a bar chart in Figure 6). The paper hypothesizes that pruning too early prevents the ViT from forming robust patch representations—at $l = 0$, the scorer operates on nearly raw patch embeddings that lack the semantic content needed to distinguish foreground from background. The marginal improvement from $l = 2$ to $l = 3$ suggests the features at layer 3 have sufficient contextualization, and the paper chooses not to test deeper layers because doing so would reduce the fraction of ViT layers benefiting from pruning. A missing ablation: the paper does not evaluate whether the optimal $l$ depends on the pruning ratio—it's possible that more aggressive pruning (e.g., 70%) benefits from deeper insertion where features are more reliable.

Scorer architecture ablation (implicit, Section 3.1 vs. Table 2): The paper does not ablate individual architectural choices of the scorer (e.g., number of MLP layers, whether to include the self-attention token pooler, whether to concatenate previous frame features vs. a different temporal context mechanism). However, the comparison between STTS and the Heuristic baseline (Table 2) effectively ablates the entire learned scorer against a non-learned alternative, showing a 0.3-point average improvement. The comparison between STTS and STTS (No Aux) (Table 2) ablates the training objective rather than the architecture, but the 2.3-point gap (62.3 vs. 60.0) demonstrates that the architecture alone is insufficient—the auxiliary loss is necessary for the scorer to produce useful scores regardless of its specific design.

Packing algorithm overhead (implicit, Section 3.3 vs. Figure 5): The paper claims the packing algorithm's overhead is "negligible because $T \ll N$" but does not provide a direct ablation measuring throughput with and without packing (e.g., comparing STTS with packing to a hypothetical STTS variant that uses uniform pruning to avoid packing). The throughput numbers in Figure 5 and Tables 6-7 include packing overhead, so the demonstrated speedups are net of this cost. The fact that speedups closely track token reduction ratios (e.g., 1.62× throughput for ~33% token reduction at 128 frames) suggests packing overhead is indeed small—if packing were expensive, throughput would be substantially lower than the token reduction ratio would predict. But a direct measurement is absent.

Performance degradation under extreme pruning (Section 5.5, Table 8, Appendix E): The paper evaluates STTS at pruning ratios from 50% to 90% (in 10% increments) and compares to Random pruning:

• At 50%: STTS 62.3 vs. Random 61.4 (+0.9)
• At 60%: STTS 61.6 vs. Random 60.1 (+1.5)
• At 70%: STTS 60.7 vs. Random 59.1 (+1.6)
• At 80%: STTS 59.8 vs. Random 57.5 (+2.3)
• At 90%: STTS 56.2 vs. Random 54.8 (+1.4)
• At 100% (text-only, no vision tokens): 44.6 (both methods, since all tokens are removed)

The gap between STTS and Random widens as pruning increases, peaking at 80% (+2.3 points) before narrowing at 90%. This demonstrates that STTS's learned scoring becomes more valuable as the token budget tightens—when you can only keep 20% of tokens, intelligent selection matters more than when you keep 50%. The text-only baseline (44.6) establishes that approximately 45% of questions can be answered using linguistic priors and dataset biases alone, providing a floor for visual reasoning. STTS at 90% pruning (retaining only 10% of vision tokens) achieves 56.2 average, meaning it extracts 11.6 points of visual reasoning from just 10% of the original visual information.

Image-only performance preservation (Appendix A, Table 4): To test whether video-oriented pruning damages image understanding (the base Molmo2 is a general VLM), the paper trains a separate STTS variant on the full Molmo2 data mixture (including both video and image data) and evaluates on 13 image benchmarks. At 50% pruning, the image average is 80.5 compared to 80.6 for the unpruned baseline (−0.1), and the multi-image average is 58.4 compared to 57.7 (+0.7). The image performance is essentially unchanged, while multi-image performance improves, which the paper attributes to transfer learning: video training teaches temporal reasoning that benefits multi-image tasks. This result is important for practical deployment—STTS can be applied to general VLMs without degrading their image capabilities.

Revision model verifier choice (not applicable): This paper does not train a separate verifier or revision model; all pruning decisions are made by the STTS scorer. The closest analog is the comparison between heuristic pruning (which uses cosine similarity directly) and learned STTS (which uses cosine similarity as an auxiliary loss), which serves a similar function to comparing a fixed verifier against a learned one.

Difficulty estimation (not applicable): The paper does not condition pruning on estimated video difficulty—the same pruning ratio is applied uniformly to all videos. This is a notable design simplification compared to adaptive methods that vary pruning strength per example. A missing experiment: evaluating whether certain video categories (e.g., static talking-head videos vs. fast-action sports) benefit differentially from pruning would reveal whether difficulty-conditioned pruning could further improve the accuracy-efficiency tradeoff.

Critical Assessment

Claim: "STTS prunes 50% of vision tokens throughout the entire architecture, resulting in a 62% improvement in efficiency during both training and inference with only a 0.7% drop in average performance across 13 short and long video QA tasks."

Does the evidence support this? Yes, with important qualifications about what "62% improvement" and "0.7% drop" actually measure.

The 62% figure specifically refers to throughput improvement at 128 frames with 50% pruning: 0.1932 → 0.3130 batches/second for training (1.62×, Table 6), and 1.0186 → 1.6439 iterations/second for inference on MLVU (1.61×, Table 7). These are genuine wall-clock measurements on 8 H100 GPUs, not theoretical FLOP reductions. However, the 62% figure is reported as an average of training and inference at one specific frame count (128). At 256 frames, the speedups are substantially larger (2.25× training, 2.22× inference), so 62% is actually a conservative estimate for long-context scenarios. The figure is also specific to the hardware configuration (8 H100 GPUs, single node) and may not generalize to different GPU architectures, batch sizes, or distributed training setups.

The 0.7% drop figure (63.0 → 62.3 Average in Table 1) is the mean across all 13 benchmarks. But this average masks significant variance: MotionBench drops by 2.8 points at 50% (61.0 → 58.2), while NextQA drops by only 0.2 points (83.9 → 83.7). On long-video benchmarks, MLVU drops by 1.9 points, LVBench by 1.5, but VideoMME-Sub drops by only 0.4. The 0.7% average is thus not a uniform property—STTS is substantially more lossy on fine-grained motion understanding and multi-task long-video MCQ than on action recognition or subtitle-aware video understanding. A practitioner choosing STTS for a motion-heavy application (sports analysis, surgical video) should expect larger degradation than the headline number suggests.

Moreover, the paper's aggregate metrics double-count certain benchmarks—VideoMME and VideoMME-Sub are both included in the average, as are LongVideo and LongVideo-Sub. This means the 0.7% figure gives extra weight to benchmarks with subtitle and non-subtitle variants, which may not reflect a practitioner's evaluation priorities. The paper would be strengthened by reporting a simple mean over the 10 unique benchmarks (removing the subtitle variants from the average) to avoid this double-counting.

Claim: "Efficiency gains increase with more sampled frames per video."

Strongly supported. The data in Tables 6 and 7 is unambiguous: 50% pruning yields 1.62× speedup at 128 frames vs. 2.25× at 256 frames for training, and 1.61× vs. 2.22× for inference. This scaling is consistent with the quadratic attention argument and suggests STTS's benefits are largest in the long-video regime where they are most needed. The paper would be strengthened by evaluating at even higher frame counts (512, 1024) to test whether the super-linear scaling continues or plateaus due to constant-cost operations (MLP layers, normalization) becoming relatively more important.

Claim: "Applying test-time scaling for long-video QA further yields performance gains of 0.5-1% compared to the baseline."

Supported, but with a narrower interpretation than the claim suggests. The TTS results (Table 3) show that 30% pruning + TTS (92 frames) achieves 60.1 Long avg., outperforming the unpruned baseline (59.0) by 1.1 points. 50% pruning + TTS (128 frames) achieves 59.4, outperforming the baseline by 0.4 points. These gains are real but come with several caveats:

1. The gains are on long-video benchmarks specifically—the paper does not report TTS on short-video benchmarks (where 64 frames likely already capture sufficient temporal context). The 0.5-1% figure is only demonstrated for the Long avg. metric.
2. The gains are achieved by increasing the number of frames at inference, which requires the video to actually have more frames to sample from. For videos shorter than the TTS frame count, this technique is inapplicable.
3. The TTS gains are computed at "equal visual token count," not equal wall-clock time. Whether processing 128 pruned frames takes the same time as 64 unpruned frames depends on the relative cost of ViT forward passes (more frames, fewer tokens each) vs. connector/LLM costs (same total visual tokens, but different frame count). The paper doesn't measure TTS throughput.
4. The TTS improvements vary widely: 30% + TTS yields +1.1 Long avg., while 50% + TTS yields +0.4. The less aggressive pruning ratio benefits more from TTS, suggesting there is a tradeoff between spatial detail preservation (lower pruning) and temporal coverage (more frames) that the paper does not fully explore.

Claim: "STTS learns that background patches are less important, while the heuristic prunes all tokens equally" (from Figure 1 and Section 5.4).

Qualitatively supported, but the evidence is illustrative rather than systematic. Figures 1 and 7 show two cherry-picked examples where STTS preserves foreground objects (a game character, human faces) while pruning backgrounds, and the heuristic fails to do so. These visualizations are compelling but represent a sample size of two. The paper does not provide:

• Quantitative metrics for pruning quality (e.g., what fraction of pruned tokens are background vs. foreground, measured against human annotations or object detection masks)
• Statistical analysis over many videos (e.g., distribution of foreground token retention rates)
• Failure cases where STTS prunes important tokens that the heuristic correctly preserves

Without such analysis, the claim that STTS "learns to recognize that foreground objects hold greater semantic importance" is a plausible interpretation consistent with the limited evidence, but it is not rigorously demonstrated. The visualizations prove that STTS can behave this way on at least two examples; they do not prove that this behavior is typical or that it explains the quantitative accuracy gains.

Claim: STTS provides "architecture-wide" pruning that reduces computation in both the ViT and LLM.

Indirectly supported, but the relative contribution of ViT vs. LLM savings is not measured. The throughput improvements (Tables 6-7) reflect end-to-end speedups that include savings in both components. However, the paper does not break down what fraction of the speedup comes from faster ViT processing vs. shorter LLM sequences. This matters because the post-ViT pruning methods that STTS is positioned against (PruneVid, Video-XL-Pro, etc.) also reduce LLM cost—they just don't reduce ViT cost. If most of STTS's speedup comes from LLM savings (because the LLM's $O(S^2)$ attention dominates end-to-end latency), then STTS's advantage over post-ViT methods may be smaller than claimed. Conversely, if the ViT savings are substantial (as the strong scaling with frame count suggests), then STTS genuinely offers benefits that post-ViT methods cannot match. A breakdown experiment—measuring ViT-only throughput and LLM-only throughput with and without STTS—would clarify this.

Missing experiments and baselines:

1. No comparison to post-ViT pruning methods in the same architecture. The paper compares STTS to ToMe (a ViT-only method) but not to any post-ViT method (PruneVid, STTM, Video-XL-Pro, etc.) implemented within the Molmo2 backbone. This makes it impossible to quantify how much the "architecture-wide" property matters relative to simply pruning more aggressively post-ViT.

2. No evaluation on models with different ViT depths or LLM sizes. All experiments use SigLIP 2 So400M + Qwen3-4B. The claim that STTS "imposes no architecture-specific constraints" is plausible but unverified—the optimal insertion layer $l = 3$ may be specific to this ViT's depth (27 layers) and behavior. A ViT with different layer count or pretraining might require a different $l$.

3. No investigation of per-video or per-benchmark pruning ratio optimization. The same $k$ is used uniformly across all videos and all benchmarks. Adaptive pruning—where easy videos (static scenes) are pruned more aggressively and hard videos (fast motion) less so—could improve the accuracy-efficiency tradeoff. The paper's own finding that 30% pruning helps some benchmarks (NextQA, VideoMME) while 50% is optimal for the average suggests that benchmark-specific or video-specific pruning ratios could be beneficial.

4. No confidence intervals or statistical significance tests. All results are single-run point estimates. The 0.2% difference between 40% and 50% pruning on Average (62.1 vs. 62.3) could easily be within run-to-run variance. Without error bars, it's impossible to know whether the non-monotonic pruning curve (40% worse than 50%) is a real phenomenon or noise.

5. No evaluation of STTS with different frame sampling strategies. The paper uses uniform sampling for long videos (64 frames across the entire video). STTS's temporal pruning might interact differently with dense sampling (e.g., 2 FPS for shorter videos) where adjacent frames are more similar and temporal redundancy is higher.

6. No comparison to simply reducing frame count. A trivial baseline: if you can drop 50% of visual tokens via STTS with 0.7% accuracy loss, how does that compare to simply sampling 50% fewer frames (32 instead of 64) with the unpruned model? If the accuracy loss from frame reduction is similar, then STTS's spatial scoring adds little beyond what temporal subsampling already provides.

Conditional validity. The paper's claims hold most strongly for:

• The specific Molmo2 architecture (SigLIP 2 ViT + Qwen3-4B LLM)
• Videos with 64 uniformly sampled frames
• The evaluated pruning ratios (30%, 40%, 50%)
• Benchmarks where motion understanding is not the primary task (MotionBench degrades 2.8 points at 50% pruning)

The claims are untested for:

• Architectures with different ViT depths or different LLM scales
• Videos with dense temporal sampling (where temporal redundancy is higher and pruning could be more aggressive)
• Pruning ratios above 50% (the paper evaluates up to 90% but only in the degradation analysis, not on the full benchmark suite)
• Real-time video applications where latency rather than throughput is the binding constraint

Larger methodological concern: the absence of per-benchmark pruning sweet spots undermines the "unified" claim. STTS uses a single global pruning ratio across all benchmarks, but Table 1 shows that different benchmarks have different optimal pruning ratios. NextQA peaks at 30% (84.1 vs. 83.9 baseline), MVBench is essentially flat across all ratios, MotionBench degrades monotonically, and LVBench improves at 30% then degrades. This suggests that the "unified" pruning policy (one $k$ for all tasks) is a compromise—a deployment system could achieve better accuracy-efficiency tradeoffs by selecting $k$ per application. The paper's framing of STTS as a unified solution is technically correct (one trained model works across all benchmarks) but practically suboptimal (a single $k$ does not maximize performance on any specific benchmark).

6. Limitations and Trade-offs

6.1 Difficulty Estimation Is Nonexistent — STTS Applies Uniform Pruning to All Videos Regardless of Content

The assumption or constraint. STTS trains a single global pruning ratio $k$ that is applied identically to every video at inference time, regardless of the video's content, length, motion characteristics, or the specific question being asked. There is no mechanism for adaptive pruning based on per-video or per-frame difficulty. The paper explicitly treats $k$ as a fixed hyperparameter swept across {30%, 40%, 50%}, and all evaluations use the same $k$ for every video in every benchmark. Section 5.1 notes that the heuristic baseline "blindly—and seemingly randomly—prunes redundant tokens based solely on simple inter-frame similarities" and argues that STTS learns to distinguish foreground from background. However, STTS itself does not adapt its pruning strength per video — a static talking-head video and a high-motion sports clip receive the same 50% token removal despite having vastly different amounts of temporal redundancy.

The consequence. This uniform pruning policy is almost certainly suboptimal. The paper's own results demonstrate that optimal pruning ratios vary by benchmark: NextQA performance improves at 30% pruning (83.9 → 84.1, Table 1) but MotionBench degrades monotonically as pruning increases (61.0 → 59.2 → 58.2, Table 1), suggesting that fine-grained motion understanding requires more visual tokens than action recognition. Within a single benchmark, videos likely have heterogeneous pruning needs — a video of someone talking against a static background can be pruned far more aggressively than a video of a fast-moving sports play. By applying a single $k$ to all videos, STTS leaves accuracy on the table for easy-to-prune videos (which could tolerate 70-80% pruning with minimal loss) while simultaneously damaging hard-to-prune videos (which need most of their tokens). The non-monotonic accuracy curve (40% pruning worse than 50% on Average, Table 1) is consistent with the interpretation that the "optimal" $k$ is an average compromise that is too aggressive for some videos and too conservative for others.

What evidence exists in the paper. Table 1 provides per-benchmark accuracy at three pruning ratios and reveals that no single $k$ is optimal across all benchmarks. At 30% pruning, NextQA, Perception-Test, MVBench, VideoMME, VideoMME-Sub, VideoEvalPro, and LVBench all match or exceed baseline — but MotionBench drops by 1.8 points and Tomato drops by 0.9 points. At 50% pruning, MotionBench drops by 2.8 points, Tomato by 1.4, and MLVU by 1.9 — these benchmarks clearly need more tokens. The paper does not report per-video pruning ratio analysis, does not evaluate whether easy-to-prune videos could tolerate higher $k$, and does not propose any difficulty estimation or adaptive allocation mechanism. The degradation analysis (Table 8, Appendix E) sweeps $k$ from 50% to 90% but only reports aggregate averages, not whether certain video categories degrade faster than others.

Mitigation status. Not addressed. The paper does not attempt per-video or per-benchmark pruning ratio selection, does not propose a difficulty estimator, and does not discuss adaptive pruning as future work. The test-time scaling results (Section 5.3, Table 3) demonstrate a form of budget reallocation (prune spatially to sample more frames temporally), but this is still a global policy applied uniformly to all videos. A natural extension — estimating per-video difficulty from early ViT features or the scorer's own confidence and varying $k$ accordingly — is not explored.

6.2 The Auxiliary Temporal Loss Requires Neighboring Frames — Breakdown Under Sparse Temporal Sampling

The assumption or constraint. The temporal auxiliary loss (Section 3.4) computes cosine similarity between corresponding patches in adjacent frames $t$ and $t+1$ to provide supervision for temporal redundancy. This works when frames are densely sampled and adjacent frames contain largely redundant information. However, the paper's own video preprocessing pipeline (Section 4.1) explicitly acknowledges a regime where this assumption breaks down: for long videos, the system "fall[s] back to uniformly sampling 64 frames across the entire video." For an hour-long video in LVBench, 64 uniformly sampled frames means approximately one frame every 56 seconds. Adjacent frames in this uniformly sampled sequence can be nearly a minute apart — they are unlikely to share the temporal redundancy that the auxiliary loss assumes. The cosine similarity between patches in frames sampled 56 seconds apart will be low not because the patches have changed in semantically meaningful ways, but because the temporal gap is simply too large for any patch-level correspondence to be reliable.

The consequence. In the sparse-sampling regime (which applies to all long-video benchmarks in this paper), the temporal auxiliary loss provides a weak or misleading training signal. The cosine similarity target $(1 - \text{CosSim})$ will be consistently high (close to 1) because frames a minute apart rarely have identical ViT features at any spatial location, even for static backgrounds. This pushes the scorer to assign high importance to all tokens — the opposite of pruning — for reasons unrelated to actual temporal redundancy. The paper's Table 2 shows that STTS outperforms Heuristic pruning by only 0.5 points on Long avg. (58.4 vs. 57.9), compared to 0.1 points on Short avg. (66.1 vs. 66.0) — the advantage is larger on long videos, but this may reflect the heuristic's even-worse failure mode rather than STTS's success. The "STTS (No Aux)" variant performs dramatically worse on Long avg. (55.5 vs. 58.4 for full STTS, Table 2), confirming that the auxiliary loss is providing useful signal even in the long-video regime, but the paper does not investigate why or how — whether the signal comes from the few video segments where frames happen to be temporally close, or from some other emergent property.

What evidence exists in the paper. The paper's long-video results are the primary evidence. LVBench and VideoEvalPro (extreme long video benchmarks) show some of the largest TTS gains (LVBench: 40.5 → 44.7 at 50% TTS, +4.2 points, Table 3), consistent with temporal coverage being a primary bottleneck — but also consistent with the auxiliary loss providing suboptimal guidance during training on these benchmarks. The paper does not ablate the auxiliary loss specifically on long-video benchmarks to quantify how much the temporal supervision helps versus hurts in the sparse-sampling regime. The visualizations in Figures 1 and 7 both show videos where adjacent frames are temporally close (a video game and a real-life sequence where consecutive frames are clearly similar), which is representative of the short-video regime but not of the sparse-sampling long-video regime.

Mitigation status. Not addressed. The paper applies the same auxiliary loss formulation to all videos regardless of their temporal sampling density. There is no weighting mechanism that reduces the auxiliary loss's influence when the inter-frame gap is large, and no investigation of alternative temporal supervision signals (e.g., long-range temporal dependencies, scene-level similarity, or learned temporal embeddings) that might work better under sparse sampling. The paper does not discuss this limitation or flag it as an area for future work.

6.3 No Quantification of Packing Overhead, Latency Penalty, or Memory Footprint — Throughput Alone Doesn't Capture Deployment Costs

The assumption or constraint. All efficiency claims are reported as throughput — batches processed per second for training, iterations per second for inference on MLVU (Section 4.3, Figure 5, Tables 6-7). Throughput measures how many videos can be processed per unit time when running a continuous stream of inputs with maximum batch size. This is the right metric for training efficiency and for batch inference pipelines, but it does not capture two critical deployment concerns: (1) latency (wall-clock time to process a single video from input to output) and (2) peak memory usage (VRAM required to hold the packed tensors, attention masks, and intermediate activations).

The packing algorithm (Section 3.3, Algorithm 1 in Appendix D) converts $T$ frames of variable-length token sequences into $T' \leq T$ packed frames. While this reduces the total number of tokens processed, it introduces several potential overheads that are not measured: the first-fit descending bin-packing itself has $O(T^2)$ complexity (acknowledged in Section 3.3), the block-diagonal attention mask adds memory and compute overhead compared to standard dense attention, and the packed tensor may have worse memory locality than the original frame-aligned tensor. Furthermore, for latency-sensitive applications (interactive video assistants, real-time perception), the packing algorithm must run after the scorer produces its output but before subsequent ViT layers can process the pruned tokens — this is a serial bottleneck that all frames must wait for, unlike parallel ViT processing of independent frames.

The paper also does not report VRAM usage for any configuration. The baseline at 256 frames with batch size 1 processes 25,307 tokens per instance and "approaches the hardware's VRAM limits" (Section 4.3). STTS reduces this to 14,939 tokens at 50% pruning (Table 6), but the packed tensor and attention mask may consume additional memory. Whether STTS enables processing larger videos or higher frame counts that were previously impossible due to memory constraints is not evaluated.

The consequence. A practitioner evaluating STTS for deployment cannot determine whether the throughput gains translate to latency improvements or whether they come at the cost of increased per-request latency due to packing overhead. For applications where each video must be processed as quickly as possible (e.g., a user waiting for a response), latency — not throughput — is the binding constraint. If the packing algorithm adds 50ms of latency per request but improves batch throughput by 60%, that's a win for offline batch processing but a loss for interactive applications.

Similarly, without VRAM measurements, it is unclear whether STTS enables processing more frames than the baseline within the same memory budget. The paper claims that "sampling more video frames further increases efficiency gains" (Section 1, contribution 4), but this refers to relative speedup, not to whether the absolute maximum number of frames that can fit in VRAM increases. If the overhead of the packed tensor format and attention mask consumes a significant fraction of the memory saved by token reduction, the frame capacity gain may be smaller than the token reduction ratio suggests.

What evidence exists in the paper. The throughput measurements (Tables 6-7) are the only efficiency metrics reported. The paper acknowledges that "we conduct all profiling on a single node equipped with 8 H100 GPUs" (Section 4.3) and that "the inference loops handle dynamic sequence lengths during prefill, resulting in slightly different overhead characteristics" compared to training with static graph compilation. The paper notes that speedup at inference is marginally lower than at training (2.22× vs. 2.25× at 256 frames, 50% pruning), which is attributed to torch.compile behavior, not to packing overhead specifically. The paper does not report:

• End-to-end latency (milliseconds per video) at any frame count or pruning ratio
• Latency breakdown (time spent in scorer, packing algorithm, remaining ViT layers, connector, LLM)
• VRAM usage (peak or average) for any configuration
• Maximum frame count that fits in VRAM with and without STTS
• Throughput at batch size 1 (for latency-sensitive deployment)

Mitigation status. Not addressed. The paper frames efficiency entirely in throughput terms and does not discuss latency, memory, or the packing algorithm's operational overhead as limitations. The acknowledgment that packing overhead is "negligible because $T \ll N$" (Section 3.3) is asserted but not measured. In a production deployment where $T = 64$ or $128$, an $O(T^2)$ packing step with non-trivial constant factors could be measurable, especially if the ViT's per-layer attention cost per frame is already optimized (e.g., with flash attention). Direct latency measurements and VRAM profiling would be straightforward to add and would substantially strengthen the practical deployment case.

6.4 Single Model Architecture, Single Training Recipe — Unverified Generalizability to Other VLMs and Scales

The assumption or constraint. All experiments use a single VLM architecture: SigLIP 2 So400M/14 384px Image ViT (27 layers) connected to Qwen3-4B LLM via Molmo2's connector module with 3×3 spatial pooling (Section 4.1). The STTS insertion layer $l = 3$ was selected based on a sweep over {0, 1, 2, 3} on this specific architecture (Section 5.2, Figure 6). The training uses exactly 6,250 steps with differential learning rates (1e-5 LLM, 5e-6 ViT, 1e-4 STTS) and the Molmo2 video QA data mixture (Section 4.1). The paper states that STTS "imposes no architecture-specific constraints, requiring only a standard ViT encoder and a token-to-LLM pathway—both ubiquitous in modern VLMs" (Section 3). However, no experiments with any other ViT backbone, LLM scale, connector design, or training recipe are reported.

The consequence. Several design choices in STTS may be architecture-dependent in ways that affect transferability:

• Insertion layer depth $l = 3$. The optimal insertion layer is likely a function of the ViT's total depth and the rate at which features transition from low-level to semantic across layers. SigLIP 2 So400M has 27 layers; a ViT-L with 24 layers or a ViT-B with 12 layers may have different optimal insertion points. STTS inserted at layer 3 in a 12-layer ViT would have a very different position in the feature hierarchy than at layer 3 in a 27-layer ViT.

• Spatial pooling width $w = 3$. STTS's scorer operates on $w \times w$ pooled patches because Molmo2 uses 3×3 spatial pooling between the ViT and LLM. An architecture with 2×2 pooling (e.g., LLaVA-style) or no pooling would require a different scorer granularity. The block-based scoring (assigning the same importance to all $w \times w$ patches in a block) may be more or less appropriate depending on the pooling ratio and the granularity of visual information the LLM expects.

• LLM scale. The paper uses Qwen3-4B (4 billion parameters). The claim that "the LLM seemed indifferent to fine-grained temporal redundancy" (Section 3.4) — motivating the auxiliary temporal loss — might not hold for larger LLMs (8B, 13B, 70B) that have greater capacity to learn to attend to temporally informative patches. A larger LLM might provide stronger downstream gradients for temporal pruning, reducing or eliminating the need for the auxiliary loss.

• Training data volume. The paper trains on only the video QA subset of the Molmo2 data mixture for 6,250 steps (~400,000 video-question pairs with sequence packing). This is a relatively small training budget compared to full VLM training (which can involve millions of examples). The scorer's ability to learn generalizable spatial saliency may depend on training data diversity.

What evidence exists in the paper. None. The paper does not evaluate STTS on any architecture other than Molmo2, does not ablate the ViT backbone or LLM size, and does not experiment with different training data volumes or mixtures. The paper acknowledges that the Molmo2 backbone was chosen because it "is a very recent SoTA model with open source code and data" (Section 4.1), not because the method was tested across multiple architectures. The claim of architecture-agnosticism is purely structural — STTS's scorer, bias injection, and packing algorithm do not depend on Molmo2-specific components — but whether the method works well on other architectures is entirely untested.

Mitigation status. Not addressed experimentally. The paper asserts architectural generality but provides no evidence for it. A minimal robustness check — applying STTS to one other VLM architecture (e.g., LLaVA-Video with a different ViT and LLM) at a single pruning ratio and benchmark — would substantially strengthen the generality claim. The paper does not discuss this as a limitation or flag it for future work.

6.5 Unclear Comparison to Frame Reduction Baselines — Does Spatial Pruning Beat Simply Sampling Fewer Frames?

The assumption or constraint. STTS's core value proposition is that intelligent token pruning — dropping spatially and temporally uninformative patches while keeping informative ones — preserves accuracy better than simply reducing the total amount of visual information uniformly. But the paper never compares STTS against the most natural baseline: sampling fewer frames with the unpruned model. At 50% pruning, the total visual token count is approximately halved. A naive alternative is to sample 32 frames instead of 64 with the unpruned model and compare the accuracy-efficiency tradeoff. If the unpruned model at 32 frames achieves similar accuracy to STTS at 64 frames with 50% pruning, then STTS's spatial scoring adds little beyond what temporal subsampling already provides. If STTS substantially outperforms the frame-reduced baseline (because it preserves important patches from all 64 frames rather than losing entire frames), then the spatial pruning is genuinely valuable.

The consequence. Without this comparison, the paper's headline claim — that learned spatial pruning is necessary and beneficial — is incompletely supported. The mechanism could be simpler than claimed: perhaps most of STTS's benefit comes from the temporal auxiliary loss (which identifies redundant frames/patches) and the spatial saliency learning (via downstream gradients) adds only marginal value. If an unpruned model at 32 frames achieves, say, 61.5 average accuracy, then STTS at 50% pruning (62.3 average, Table 1) offers a 0.8-point improvement over simple frame reduction — a more modest claim than "62% efficiency with only 0.7% accuracy loss" (the 0.7% is relative to 64-frame baseline, not 32-frame baseline). If the 32-frame baseline achieves 62.3 or better, then STTS's spatial pruning is essentially equivalent to frame subsampling and its contribution is efficiency (you can process more frames for the same compute) rather than accuracy preservation.

The test-time scaling results (Section 5.3, Table 3) partially address this by showing that STTS + TTS (more frames, fewer tokens each) outperforms the baseline (fewer frames, more tokens each). But this is a comparison at equal total token count, not at equal compute — and it compares pruned + scaled against unpruned + unscaled, not unpruned + subsampled.

What evidence exists in the paper. None. The paper does not report accuracy for the unpruned model with fewer frames (e.g., 32, 48, or 56 frames). The only frame-count variation is in the TTS experiments (Table 3), where the unpruned baseline is always evaluated at 64 frames and STTS models are evaluated at 64, 92, 107, or 128 frames depending on pruning ratio. The paper does not include the unpruned model at any of these higher frame counts for comparison.

Mitigation status. Not addressed. This is a notable gap because it is a straightforward experiment to run: evaluate the unpruned Molmo2 baseline at frame counts from 16 to 64 on the same benchmarks and plot the accuracy vs. token count curve, overlaid with STTS's accuracy at different pruning ratios. This would directly show whether STTS extracts more information per token than simple frame reduction, or whether the benefits are primarily from the temporal redundancy signal (which uniform frame subsampling partially captures by reducing adjacent-frame similarity). The paper does not discuss this comparison or acknowledge its absence.

6.6 Performance Degradation Concentrated on Fine-Grained Motion Understanding — Not a General-Purpose Efficiency Solution

The assumption or constraint. STTS is evaluated on 13 video QA benchmarks spanning diverse tasks, and the paper reports an average 0.7% accuracy drop at 50% pruning. However, this average masks large per-benchmark variance. The degradation is heavily concentrated on benchmarks that test fine-grained motion understanding and temporal dynamics, while benchmarks focused on action recognition, subtitle reading, and static visual question answering are largely unaffected.

The consequence. The headline 0.7% figure misleads practitioners whose applications are motion-heavy. At 50% pruning, specific benchmarks show:

• MotionBench: 61.0 → 58.2 (−2.8 points, −4.6%)
• MLVU: 70.3 → 68.4 (−1.9 points, −2.7%)
• Tomato: 36.5 → 35.1 (−1.4 points, −3.8%)
• LVBench: 42.0 → 40.5 (−1.5 points, −3.6%)

These are substantial degradations — 2-4× larger than the headline average — and they cluster on tasks requiring understanding of how objects move and change over time. MotionBench specifically tests fine-grained motion understanding (e.g., "is the person accelerating or decelerating?"), and STTS at 30% pruning already degrades it by 1.8 points. The auxiliary temporal loss based on cosine similarity between adjacent frames may be actively harmful for motion understanding: subtle motion (a person slightly shifting posture, an object slowly rotating) produces small feature changes that the cosine similarity may classify as "redundant" (cosine similarity close to 1), causing STTS to prune the very patches that encode motion information.

The other side of this tradeoff — benchmarks where STTS barely degrades or improves — are those where motion is less central: NextQA (−0.2 at 50%), MVBench (−0.2), VideoMME (−0.4), VideoMME-Sub (−0.4). These benchmarks test action recognition from context and scene understanding, where the key visual cues are object identities, spatial relationships, and text (subtitles), all of which STTS's spatial saliency scoring preserves well.

What evidence exists in the paper. Table 1 provides per-benchmark accuracies at 30%, 40%, and 50% pruning, from which these patterns are directly visible. The paper does not analyze per-benchmark variance or discuss motion sensitivity as a limitation. The visualizations in Figures 1 and 7 show examples where STTS preserves foreground objects — but these are static or slow-motion scenarios. No visualization shows a high-motion scenario where STTS's pruning pattern might explain the MotionBench degradation.

Mitigation status. Not addressed. The paper does not:

• Acknowledge that motion-heavy applications are more affected by pruning than static-scene applications
• Analyze whether motion-related tokens are disproportionately pruned (e.g., by measuring token retention rates for patches containing optical flow above a threshold vs. below)
• Propose motion-aware pruning that reduces pruning strength in high-motion regions or video segments
• Evaluate whether a lower pruning ratio specifically for motion-heavy benchmarks closes the gap (the paper only tests {30%, 40%, 50%} globally, and even 30% pruning degrades MotionBench by 1.8 points)

This limitation is particularly consequential because video understanding — unlike image understanding — is defined by motion and temporal change. A video VLM that struggles with motion understanding while excelling at static scene analysis is only partially solving the video problem. STTS's efficiency gains come disproportionately at the expense of the very capability that distinguishes video from images.

7. Implications and Future Directions

How This Work Changes the Landscape

STTS does not introduce a new paradigm for video VLM efficiency, but it does something arguably more valuable: it collapses two previously separate research threads into a single, simpler solution and in doing so reveals that the separation was never necessary. Prior work partitioned the token pruning problem into a ViT-side concern (in-ViT pruning for spatial redundancy, evaluated on unimodal perception) and an LLM-side concern (post-ViT pruning for input length reduction, evaluated on multimodal QA). Researchers working on one rarely engaged with the other because the methods, evaluation protocols, and failure modes were different. STTS demonstrates that a single scorer—inserted at one ViT layer, trained with two complementary signals, and applying hard pruning that propagates naturally through the remaining architecture—solves both problems simultaneously. The ViT gets faster because tokens are physically removed from later layers; the LLM gets faster because fewer tokens enter its sequence; and both benefits come from a single decision point, not two coordinated pruning mechanisms.

This is a reframing, not a paradigm shift. The reframing is: pruning decisions should be made as early as possible in the VLM pipeline, and they should be permanent so that all downstream components benefit automatically. The implication is that post-ViT pruning methods—FreeVA, PruneVid, STTM, HoliTom, Video-XL-Pro, the Matryoshka-based approaches—are solving a problem that should not exist in the first place. If you prune inside the ViT, you do not need a separate post-ViT pruning step; the ViT already produces fewer tokens, and the LLM's sequence length reduction comes for free. Post-ViT methods are not wrong, but they are architecturally inefficient: they leave the ViT bottleneck untouched and then apply additional computation (cross-attention, merging, clustering) to reduce token counts that could have been reduced earlier for less cost.

The paper also diagnoses a specific failure mode that should inform all future work on learned video token pruning: the downstream VLM task loss, on its own, provides inadequate supervision for identifying temporally redundant tokens. The evidence is the "STTS (No Aux)" result in Table 2—a 2.3-point drop below the full STTS, and 1.4 points below random pruning. This is a clean, negative result: task-driven gradients can teach spatial saliency but not temporal redundancy. The implication is that any end-to-end learned video pruning method that relies solely on downstream task loss is likely leaving temporal pruning performance on the table. Future work should either (a) include an explicit temporal supervision signal (as STTS does) or (b) demonstrate that a different training configuration—larger LLM, different loss, different architecture—can provide temporal gradients that the Molmo2 + Qwen3-4B setup could not. The paper does not claim that (b) is impossible, only that it failed in their specific setup, but the negative result is strong enough to shift the burden of proof: a new method that claims to learn temporal pruning from task loss alone must now contend with this evidence.

The resolution of conflicting prior findings is indirect but real. Prior work split into two camps: ViT pruning methods (ToMe, SPViT, DToP) that showed strong efficiency gains on vision tasks but were not evaluated on VLM QA, and post-ViT methods that showed modest gains on VLM QA but did not address ViT cost. The natural question was whether ViT pruning degrades downstream VLM reasoning (the fear being that merged or dropped tokens lose cross-modal information the LLM needs) and whether post-ViT pruning can match the efficiency of ViT-side pruning. STTS answers both: ViT pruning does not inherently harm VLM reasoning—in fact, it can help by removing distracting tokens (Table 1: NextQA improves at 30% pruning, LVBench at 30% pruning, VideoEvalPro at 30% pruning)—and post-ViT pruning is unnecessary if you prune inside the ViT. The prior work was not wrong, but it was addressing a false dichotomy: you do not have to choose between ViT efficiency and LLM efficiency. A single pruning decision achieves both.

The research directions this opens and closes:

• More attractive: Joint training of pruning policies with downstream VLM objectives, explicit temporal supervision signals beyond adjacent-frame similarity, packing-based implementation strategies that turn theoretical FLOP reductions into wall-clock speedups, and test-time scaling as a complement to training-time pruning (trading spatial precision for temporal coverage).
• Less attractive: Purely post-ViT pruning methods that leave the ViT untouched, unless they can demonstrate that their additional complexity (text-conditioned selection, cross-attention modules) provides accuracy benefits that early-ViT pruning cannot match. STTS's simplicity—a 3-layer MLP with attention pooling, no text conditioning, no merging—sets a strong baseline: new methods should justify their added complexity with evidence that STTS's fixed-layer early pruning misses important signals that later, text-aware pruning captures. Similarly, merge-based ViT methods (ToMe, SPViT) that average tokens rather than selectively discarding them face a steeper burden of proof: the paper's Table 5 (Appendix C) shows that trained ToMe underperforms STTS by 1.2 points average, suggesting that token averaging corrupts fine-grained information needed for video reasoning.

The paper also establishes a methodological standard for efficiency claims in token pruning research. The insistence on demonstrating actual wall-clock speedups (not just theoretical FLOP reductions) via the packing algorithm, and the detailed throughput tables at multiple frame counts and for both training and inference, raise the bar for future work. A pruning paper that reports only FLOP reductions without showing that they translate to measured speedups is now implicitly incomplete—the community has a concrete counterexample showing that the gap between theoretical FLOPs and wall-clock throughput can be large (because sparse tensors do not accelerate dense matrix multiplications) and that pack-and-pad strategies can bridge it. Future papers should follow STTS's lead: report throughput at specific hardware configurations, for specific sequence lengths, in both training and inference modes.

Follow-Up Research This Work Enables

Quantifying the relative contribution of spatial vs. temporal scoring to downstream accuracy, and ablating whether the auxiliary loss is necessary for all video types or only some. The paper shows that removing the auxiliary temporal loss causes a 2.3-point accuracy drop (Table 2, STTS vs. STTS No Aux), but this is a single aggregate number across all 13 benchmarks and all video types. A natural follow-up would train STTS variants where the auxiliary loss weight is modulated per video based on temporal sampling density: for densely-sampled short videos (where adjacent frames are a fraction of a second apart and temporal redundancy is high), weight the auxiliary loss heavily; for uniformly-sampled long videos (where adjacent frames can be a minute apart and cosine similarity is unreliable), reduce or zero out the auxiliary loss and rely on downstream spatial gradients alone. The prediction is that the auxiliary loss helps most on short-video benchmarks where adjacent-frame similarity is a meaningful signal, and helps less or even hurts on long-video benchmarks where sparse sampling makes similarity uninformative. The paper's Table 2 already shows that the heuristic (pure cosine similarity, no learning) underperforms STTS more on long videos (+0.5 Long avg. gap) than on short videos (+0.1 Short avg. gap), consistent with the idea that the temporal signal is weaker on long videos—but this is confounded by the heuristic's inability to learn spatial saliency. Disentangling this requires an experiment that varies the auxiliary loss weight per benchmark or per video and measures the interaction.

A systematic comparison of STTS against post-ViT pruning methods implemented within the same Molmo2 backbone, measuring per-module speedup breakdowns. The paper compares STTS against ToMe (a ViT-only method) but not against any post-ViT method. The strongest test of STTS's "architecture-wide" claim would be to implement a representative post-ViT method—say, PruneVid-style spatial + temporal merging or Video-XL-Pro-style text-conditioned selection—within the Molmo2 architecture and compare at equal total token reduction. The key measurements would be: (a) end-to-end accuracy at matched token counts, (b) ViT-only throughput (to quantify the cost of leaving the ViT untouched), (c) LLM-only throughput (to quantify whether post-ViT methods match STTS's LLM speedup), and (d) the additional computational cost of the post-ViT pruning module itself (cross-attention with text tokens for text-conditioned methods, or clustering/merging overhead for merge-based methods). The hypothesis is that post-ViT methods can match STTS's accuracy at the same token count (since both ultimately select which information reaches the LLM), but cannot match end-to-end throughput because the ViT bottleneck is not addressed. Quantifying this gap would directly validate or refute the paper's central architectural argument.

Adaptive per-video pruning ratio selection based on estimated motion content or scorer confidence. The paper shows that the optimal pruning ratio varies by benchmark (Table 1: MotionBench degrades at all pruning levels, NextQA tolerates 50% easily) and that the non-monotonic accuracy curve (40% worse than 50%, Section 4.2) suggests a compromise where the global $k$ is too aggressive for some videos and too conservative for others. A straightforward extension would add a lightweight difficulty estimator: compute a simple motion statistic from the ViT features at layer $l$—e.g., the average cosine similarity across all adjacent-frame patch pairs, which is already computed for the auxiliary loss—and use it to modulate the pruning ratio. High average similarity (static video) → prune more aggressively (e.g., 70% instead of 50%); low average similarity (high-motion video) → prune more conservatively (e.g., 30% instead of 50%). The extension is trivial to implement (the similarity signal is already available during the forward pass) and would test whether the largest accuracy degradations (MotionBench −2.8 at 50%, Table 1) can be mitigated by simply reducing pruning strength on high-motion videos. The follow-up would report per-benchmark accuracy with adaptive pruning and compare to the fixed-$k$ baseline at equal average token reduction across all videos, to ensure the adaptive method is not simply using more tokens on hard videos without saving tokens on easy ones.

Evaluating STTS on other VLM architectures and at larger LLM scales to test the generality of the auxiliary loss requirement. The paper's finding that the downstream task loss cannot drive temporal pruning may be specific to the Qwen3-4B LLM scale. A larger LLM (e.g., Qwen3-8B, 14B, or 72B) might have sufficient capacity to learn to attend to temporally informative patches without explicit supervision, producing downstream gradients that do guide temporal pruning. The follow-up experiment would replicate the "STTS vs. STTS No Aux" ablation (Table 2) on the same Molmo2 architecture but with different LLM sizes, keeping the ViT and training data constant. If the gap between STTS and STTS No Aux narrows as LLM size increases, it suggests that the auxiliary loss is a "scaffolding" mechanism that compensates for limited LLM capacity and becomes unnecessary at scale—which would be an important finding for practitioners training large-scale video VLMs where every auxiliary loss adds engineering complexity. Conversely, if the gap persists or widens with LLM scale, it would strengthen the paper's claim that explicit temporal supervision is fundamentally necessary regardless of model capacity.

Investigating whether STTS's spatial saliency scoring learns to attend to text and subtitles, and whether this can be leveraged for OCR-heavy video tasks. The paper's visualizations (Figures 1 and 7) focus on foreground object preservation, but Table 1 shows that VideoMME-Sub (subtitle-aware video understanding) actually improves at 30% pruning (+0.9 points) and degrades minimally at 50% (−0.4 points). This suggests that the spatial saliency signal from downstream gradients is teaching the scorer to preserve text regions—which makes sense, since many VLM QA tasks require reading on-screen text. A diagnostic follow-up would measure text-region retention rates: on videos with subtitles, what fraction of tokens corresponding to text bounding boxes survive pruning vs. background tokens? This could be done by running an off-the-shelf OCR or text detection model on the input frames and comparing retention rates of text-containing patches against non-text patches. If STTS preserves text at significantly higher rates than random or heuristic pruning, it demonstrates a specific, interpretable capability that the spatial saliency mechanism has learned—and it also explains why subtitle-aware benchmarks benefit from mild pruning (background removal increases the relative attention on text tokens).

Practical Applications and Downstream Use Cases

Training-time acceleration for video VLM research teams with fixed GPU budgets. The paper's training throughput numbers (Table 6) translate directly to faster iteration cycles. At 128 frames with 50% pruning, training throughput increases from 0.1932 to 0.3130 batches/second on 8 H100 GPUs—a 1.62× speedup. For a research team training video VLMs on, say, one million video-question pairs with batch size 128, this reduces training time from approximately 40 hours to 25 hours per epoch. Over the course of a project with dozens of training runs (hyperparameter sweeps, architecture variations, data ablations), the cumulative time savings are substantial. The speedup is larger at higher frame counts (2.25× at 256 frames), making STTS particularly valuable for long-video model development where training on full-length videos is otherwise prohibitively slow. The key practical consideration is that STTS must be trained from the start with pruning enabled—it is not a post-hoc optimization that can be applied to an already-trained model (as demonstrated by the poor performance of inference-only baselines in Table 5). Teams adopting STTS would integrate it into their training pipeline from the beginning, accepting a one-time implementation cost in exchange for ongoing throughput gains.

Long-video batch inference with accuracy parity to unpruned models at lower cost. The test-time scaling results (Table 3) demonstrate a path to improved long-video understanding without increased compute: train with 30% pruning, then at inference time increase the frame count proportionally (92 frames instead of 64) to maintain the same total visual token budget. This achieves 60.1 Long avg., outperforming the unpruned 64-frame baseline (59.0) by 1.1 points while using approximately the same number of visual tokens—and therefore approximately the same compute cost per video (the exact cost parity depends on ViT vs. LLM cost ratios, which the paper does not break down, but the token count parity makes cost parity plausible). For a video understanding API or batch processing pipeline that charges per token or per GPU-hour, this is a direct accuracy improvement at no additional cost: swap the deployed model from unpruned Molmo2 at 64 frames to STTS 30% at 92 frames, and long-video benchmark scores improve without increasing the inference budget. The practical deployment step is simple—only the frame sampling rate at inference needs to change, no model retraining—making this an unusually low-friction accuracy improvement for long-video applications.

Enabling higher frame counts on memory-constrained hardware for extreme-length video understanding. The baseline model at 256 frames with batch size 1 processes 25,307 tokens per instance and "approaches the hardware's VRAM limits" on 8 H100 GPUs (Section 4.3). STTS at 50% pruning reduces this to 14,939 tokens per instance (Table 6)—a 41% reduction. If the VRAM bottleneck is primarily due to the quadratic attention memory, this reduction could enable processing videos with approximately 1.7× more frames on the same hardware (since the token count per frame is halved, the maximum frame count roughly doubles). For extreme-length video understanding tasks—processing hour-long surveillance footage, full-length movies, or continuous video streams for autonomous systems—this shifts the boundary of what is feasible on a given hardware budget. A team currently limited to processing 256-frame video segments could potentially handle 400+ frame segments with STTS, or maintain 256 frames with smaller batch size but higher throughput. The practical limitation is that the paper does not report VRAM measurements, so the actual frame count increase enabled by STTS on a specific GPU configuration would need to be verified empirically.

When to Prefer This Method

The paper does not explicitly articulate a decision rule for choosing STTS over specific named alternatives, and the experimental comparisons to other pruning methods are limited to ToMe (a ViT-only merge-based method) and a cosine-similarity heuristic. The paper's positioning—"unified, architecture-wide pruning" versus partial (ViT-only or LLM-only) pruning—is a structural argument rather than a quantitative tradeoff against, say, PruneVid or Video-XL-Pro implemented in the same backbone. As a result, producing a generic "prefer A when X, prefer B when Y" matrix would require extrapolating beyond the paper's evidence. The empirical guidance the paper does support is narrower and more specific:

• Prefer training STTS jointly with your VLM from the start rather than applying pruning only at inference time: the inference-only baselines (Table 5) underperform trained variants by 1-2 points, and even a trained merge-based method (ToMe) underperforms STTS by 1.2 points, indicating that joint training with hard pruning is essential.
• Prefer higher pruning ratios (50%) over intermediate ratios (40%) if your application is not motion-heavy: the non-monotonic accuracy curve (Table 1: 62.3 at 50% vs. 62.1 at 40% Average) means you get more efficiency for effectively the same average accuracy, and the 50% model's 2.25× training speedup at 256 frames (Table 6) substantially exceeds the 40% model's 1.88×.
• Prefer 30% pruning with test-time scaled frame counts for long-video applications where accuracy is paramount and you are willing to increase inference-time frame sampling: this configuration achieves the highest Long avg. (60.1, +1.1 over unpruned baseline, Table 3) among all evaluated STTS variants.
• Avoid STTS (or use lower pruning ratios, ≤30%) on tasks dominated by fine-grained motion understanding: MotionBench degrades by 1.8 points even at 30% pruning (Table 1) and by 2.8 points at 50%. If your application is sports analysis, surgical video, or physical action assessment, the spatial saliency scorer's tendency to prune subtle motion cues makes aggressive STTS pruning risky; either reduce $k$ or pair STTS with a motion-aware loss term that explicitly preserves high-optical-flow regions.