LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

ArXiv: 2605.17260

🎯 Pitch

Existing Video LLMs try to scale to more frames by compressing tokens after the vision encoder, but LiteFrame reveals this just shifts the latency bottleneck to the vision encoder itself. By distilling a compact encoder to directly output pre-compressed representations, it slashes total latency by 35% while simultaneously processing 8× more frames and boosting accuracy.

---

1. Executive Summary

This paper introduces LiteFrame, a lightweight video encoder backbone for Video LLMs that resolves a newly identified efficiency bottleneck: when post-hoc token reduction alleviates the LLM's quadratic complexity, the vision encoder's per-frame processing becomes the dominant latency cost, preventing effective scaling to long-form video. The authors train LiteFrame via Compressed Token Distillation (CTD) — a framework that teaches a compact student encoder (87M parameters) to directly predict spatio-temporally compressed representations produced by a large teacher model (InternViT-300M, 304M parameters) under a Weighted Average Pooling (WAP) operator — followed by lightweight Language Model Adaptation (LMA) to align the compressed latents with the LLM. Evaluated against InternVL3-8B on Video-MME, MLVU, and LongVideoBench, LiteFrame delivers a 35% reduction in end-to-end latency while processing 8× more frames and improves average video understanding accuracy, establishing a new latency-accuracy Pareto frontier by demonstrating that architectural internalization of token compression via distillation outperforms both standard post-hoc reduction and alternative efficient encoder designs.

2. Context and Motivation

The Core Problem: Video LLMs Don't Scale to Long-Form Video

The fundamental challenge this paper tackles is the prohibitive computational cost of scaling Video Large Language Models to handle long-form video — videos spanning minutes to hours rather than seconds. Modern Video LLMs — including InternVL3, Qwen2.5-VL, and LLaVA-OneVision — follow a near-universal architectural template: an image encoder (typically a Vision Transformer, ViT) processes each video frame independently, an alignment projector maps visual features into the LLM's input space, and the LLM reasons over the interleaved visual and text tokens. The problem is that every additional input frame increases computational cost in both the vision encoder and the LLM, creating a compound scaling challenge that sharply limits how many frames can practically be processed.

This matters enormously for real-world deployment. Long-form video understanding — security footage analysis, lecture comprehension, movie summarization, sports highlight detection, medical procedure review — requires processing hundreds to thousands of frames to capture temporal dynamics, causal relationships, and fine-grained events. Yet as the paper demonstrates in Figure 3 (left), accuracy on benchmarks like Video-MME, MLVU, and LongVideoBench exhibits logarithmic scaling with input frame count: more frames consistently improve performance, but conventional architectures hit a hard wall around 64 frames due to combined ViT and LLM costs. The consequence is that current Video LLMs are fundamentally capped in their ability to benefit from richer temporal information, leaving performance on the table even when the relevant video content exists in the input.

The Post-Hoc Reduction Paradigm — and Its Hidden Floor

The dominant approach to this scaling problem has been what the paper calls the "extract-and-reduce" paradigm: maintain a frozen, pretrained image encoder that extracts dense per-frame features, then apply post-hoc token reduction strategies before feeding tokens to the LLM. The rationale is straightforward — the LLM's self-attention has quadratic complexity in sequence length, so reducing the number of visual tokens directly attacks what appears to be the primary bottleneck.

This family of methods has become crowded with increasingly sophisticated techniques. Spatial reduction methods like LLaVA-PruMerge adaptively prune tokens based on attention scores, VisionZip removes tokens deemed unimportant for the language task, and DyMU dynamically merges and unmerges tokens based on content similarity. Spatio-temporal extensions like FastVID apply dynamic density pruning across both dimensions, HoliTom merges tokens holistically across space and time, and PruneVid specifically targets visual token pruning for video. Query-guided approaches like PyramidDrop and TopV use the LLM's own processing to determine which tokens to retain.

The paper identifies a critical blindspot in this entire paradigm, articulated in Section 3 with the phrase "scaling paradox." When post-hoc reduction is aggressively applied — say, 16× compression reducing 256 tokens per frame to just 16 — the LLM's computational burden drops dramatically. But here's the crucial observation that prior work missed: the vision encoder still processes every frame at full resolution before compression occurs. The ViT's per-frame cost, while seemingly modest in isolation (tens of milliseconds per frame), accumulates linearly with frame count. As the authors demonstrate in Figure 1(b), once post-hoc reduction effectively alleviates the LLM bottleneck, the vision encoder's latency becomes the new dominant cost — and it grows without bound as frame counts increase.

This bottleneck shift is not a minor implementation detail. It establishes a hard latency floor that no amount of post-hoc cleverness can penetrate. You can compress tokens after extraction to 1/16th their original count, but you cannot avoid paying the full ViT forward pass for every frame you want to process. The paper quantifies this concretely in Table 3: when InternVL3-8B processes 8 frames with its 304M-parameter InternViT-300M encoder, vision encoding takes 40.0 ms. With FastVID post-hoc compression at 32 frames (4× more frames), vision encoding jumps to 161.7 ms — a 4× increase that dominates the total latency budget despite aggressive token reduction.

Why Existing Efficient Encoder Approaches Fall Short

The paper does not claim to be the first to notice that vision encoders are expensive. Several lines of work have attempted to build more efficient visual backbones for MLLMs, but each misses a critical requirement for long-form video understanding:

Image-centric efficient encoders like FastVLM and MobileNet-v5 achieve impressive throughput on static images through convolutional efficiency and architectural optimization. FastVLM, for instance, introduces FastViTHD — a hybrid encoder combining convolutions with transformer-based global modeling — and achieves substantially lower per-image latency than standard ViTs. However, these architectures are designed for spatial encoding only. They process each frame independently and do not explicitly exploit the strong temporal redundancy that exists between adjacent frames in video. When applied to video frame-by-frame, they inherit the same linear cost scaling problem: 256 frames means 256 independent forward passes, each paying the full spatial encoding cost regardless of how similar consecutive frames are.

Encoder-free paradigms like Video-Panda take a more radical approach, proposing to bypass a heavy visual backbone entirely. Instead, Video-Panda uses a lightweight Spatio-Temporal Alignment Block to directly map raw video inputs into token sequences for the LLM. This elegantly eliminates the vision encoder bottleneck — with only 45M parameters for the alignment module, the visual processing cost is dramatically reduced. But the paper identifies the flip side of this trade-off (Section 2, Table 4): because Video-Panda doesn't compress the token stream, it exposes the downstream LLM to dense, uncompressed token sequences. At 272 tokens per frame and 32 frames, this means 8,704 visual tokens flooding the LLM, shifting the bottleneck right back to quadratic self-attention. The encoder-free approach solves the vision problem but resurrects the language problem.

Pre-filtering approaches like AutoGaze represent the most recent and directly comparable work. AutoGaze trains a lightweight module to pre-filter visual tokens before they are processed by a heavy VideoViT backbone, using autoregressive "gazing" to selectively attend to promising regions. This is conceptually appealing — reduce the input to the expensive encoder rather than reducing its output. However, the paper's detailed comparison in Figure 7 and Section 5.2.4 reveals a devastating latency cost: the AutoGaze pre-reduction module itself accounts for nearly half of total inference time (3.0 seconds out of 6.1 seconds at 256 frames), because it introduces an autoregressive decoding step and requires a heavy VideoViT backbone that processes 16-frame clips with full spatio-temporal attention. The paper's breakdown in Figure 7 (right) shows that while AutoGaze successfully accelerates its own VideoViT baseline, the overhead of the gazing module plus the VideoViT's inherent cost means the total system is actually slower than a standard ImageViT baseline processing 32 frames (6,106 ms vs. 781 ms) — despite processing 8× more frames.

The Missing Piece: Holistic Encoder + LLM Co-Optimization

What emerges from this analysis is a clear gap: existing efficient encoder approaches optimize either the vision side or the language side in isolation, but none achieve simultaneous, holistic optimization of both. Post-hoc methods optimize the LLM at the cost of ignoring the encoder. Encoder-free methods optimize the encoder at the cost of overwhelming the LLM. Pre-filtering methods add auxiliary overhead that negates their theoretical advantages. Image-centric efficient encoders simply don't address temporal redundancy.

The paper's central insight — articulated in Section 3's "scaling paradox" and formalized in Section 4 — is that the token compression must be internalized within the vision encoder itself. This achieves two things simultaneously: (1) the encoder processes frames more efficiently because it's architecturally lighter and progressively downsamples tokens, reducing per-frame cost; and (2) the LLM receives a drastically compressed token stream (16 tokens per frame vs. 256), eliminating the quadratic complexity bottleneck. Critically, this internalization means the compression is not an additional post-processing step — it's baked into the forward pass, so there's no separate reduction module adding latency overhead.

How the Paper Positions Itself

The paper explicitly frames its contribution as filling the gap between two parallel research threads that have been pursued independently:

Relative to post-hoc reduction methods, LiteFrame is not a competing token reduction technique but rather a replacement for the underlying encoder that makes post-hoc reduction unnecessary. The paper demonstrates in Figure 5 that simply swapping the heavy InternViT-300M for LiteFrame (without any LMA, keeping the LLM frozen) already surpasses all training-free post-hoc methods — including ToMe, LLaVA-PruMerge, and FastVID — in both efficiency and accuracy. The key difference is architectural: post-hoc methods pay the full encoder cost and then try to recover efficiency, while LiteFrame avoids paying that cost in the first place.

Relative to efficient encoder methods, LiteFrame is specifically designed for the video domain's unique challenge: temporal redundancy. Unlike FastVLM (image-centric), LiteFrame incorporates depth-wise temporal convolutions that explicitly model cross-frame dynamics. Unlike Video-Panda (encoder-free), LiteFrame compresses tokens internally so the LLM receives a manageable visual context. Unlike AutoGaze (pre-filtering), LiteFrame incurs no auxiliary module overhead — the compression is a natural consequence of the architecture's progressive downsampling.

The paper positions its Compressed Token Distillation (CTD) framework as the key enabler of this internalization. The challenge is non-trivial: training a compact student (87M parameters, 12 layers, 768D) to match the semantic richness of a large teacher (304M parameters, 24 layers, 1024D) while simultaneously reducing token count by 16×. Standard distillation — having the student mimic the teacher's dense, full-resolution output — would force the student to learn redundant spatial details it cannot effectively represent. CTD solves this by using the teacher's own compressed output (under Weighted Average Pooling) as the supervision target, effectively teaching the student to bypass the redundant computation and go straight to the information-dense compressed representation.

The paper's ultimate claim is not merely that LiteFrame is faster — it's that architectural internalization of compression via distillation redefines the efficiency-accuracy Pareto frontier for video understanding. Figure 2 visualizes this: existing post-hoc methods (color dashed lines) cluster in the high-latency, moderate-accuracy region, unable to break through because the encoder floor constrains them. LiteFrame (red stars) pushes substantially into the low-latency, high-accuracy corner, achieving what the paper characterizes as "a new potential path to unlocking longer-form video understanding under fixed compute budgets."

3. Technical Approach

3.1 Reader Orientation

This paper builds a lightweight video encoder backbone called LiteFrame that replaces the heavy image encoder in a standard Video LLM pipeline, enabling the system to process many more frames within a fixed compute budget. The core insight is that token compression — typically done as a post-hoc step after full-resolution encoding — should instead be internalized into the encoder architecture itself, so the encoder never computes redundant spatio-temporal detail that will be discarded later, thereby eliminating both the LLM's quadratic complexity bottleneck and the vision encoder's linear scaling bottleneck simultaneously.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three major stages, corresponding to the training pipeline and inference deployment:

1. Teacher ViT (InternViT-300M) — a large, pretrained image encoder (304M parameters, 24 layers, 1024D hidden dimension) that processes video frames individually and produces dense, full-resolution feature maps. This serves as the knowledge source for distillation; it is never used at inference time.

2. LiteFrame Student Encoder (ViT-Base + DW Temporal Conv) — a lightweight video encoder (87M parameters, 12 layers, 768D hidden dimension) that processes frames with progressive spatio-temporal downsampling, outputting a drastically compressed token sequence. At inference, this replaces the teacher entirely.

3. Language Model Adaptation (LMA) via LoRA — a lightweight fine-tuning stage where both the LiteFrame encoder and the LLM's LoRA adapters are jointly optimized on video-text pairs, aligning the compressed latent space with the LLM's input expectations and adapting the LLM to handle extended temporal contexts (up to 512 frames).

The training flow proceeds as: (1) Teacher processes dense frames → WAP compresses teacher features into a compact target → Student is trained via MSE to predict this compressed target directly (CTD). (2) The distilled student encoder is then plugged into the full Video LLM pipeline (replacing the teacher), and both encoder and LLM are fine-tuned jointly on video-QA data (LMA). At inference, the lightweight student processes frames with its internal progressive downsampling, producing only 16 tokens per frame, which are then fed to the LLM for reasoning.

3.3 Roadmap for the Deep Dive

• First, the Weighted Average Pooling (WAP) compression primitive, since it serves double duty: it defines the post-hoc baseline that motivates the work, and it generates the supervision targets for the distillation framework.
• Second, the spatio-temporal encoder architecture, covering how progressive downsampling and depth-wise temporal convolutions achieve token compression internally rather than as a separate step.
• Third, Compressed Token Distillation (CTD), the training framework that teaches the student to predict the teacher's WAP-compressed output, including the loss function, initialization strategy, and stability techniques.
• Fourth, Language Model Adaptation (LMA), the fine-tuning stage that bridges the modality gap between the distilled encoder and the LLM, and adapts the LLM to extended frame counts.
• Fifth, the full inference pipeline, showing how these components fit together at test time and how the efficiency gains manifest.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that token compression should be internalized into the vision encoder via a two-stage distillation-then-adaptation training pipeline, rather than applied as a post-hoc step after expensive full-resolution encoding.

---

Weighted Average Pooling (WAP) as the Compression Primitive

Before explaining how LiteFrame internalizes compression, we must understand the compression operator itself — because it serves as both the post-hoc baseline and the supervision target for distillation. The paper proposes Weighted Average Pooling (WAP) as a simple yet effective spatio-temporal compression primitive that outperforms more complex alternatives.

The input is a feature tensor $X \in \mathbb{R}^{T \times H \times W \times C}$, representing $T$ frames, each with spatial dimensions $H \times W$ and $C$ feature channels. The goal is to produce a compressed tensor at a target resolution $(t, h, w)$, where $t < T$, $h < H$, and $w < W$, with a compression ratio $r = \frac{T H W}{t h w}$ (e.g., $r = 16$ for 4× temporal and 4× spatial compression).

The WAP operation partitions the input into non-overlapping spatio-temporal blocks $\Omega_{u,v,s}$ corresponding to the target grid positions. For each block, WAP computes a weighted sum where the weights are determined by an attention mechanism between each token and the class token of its frame:

$Y_{u,v,s} = \sum_{(\tau,i,j) \in \Omega_{u,v,s}} \text{softmax}\left( \frac{x_{\tau,\text{cls}}^\top x_{\tau,i,j}}{\sqrt{C}} \right) x_{\tau,i,j}$

where $x_{\tau,i,j} = X[\tau, i, j, :]$ is the feature vector at position $(i,j)$ in frame $\tau$, $x_{\tau,\text{cls}}$ is the class token embedding of the $\tau$-th frame, and the softmax is computed within each block $\Omega_{u,v,s}$ independently.

What it computes: For each target spatio-temporal position $(u,v,s)$, WAP takes a weighted average of all input tokens within the corresponding block, where the weight of each token is proportional to the exponential of its scaled dot-product similarity with the frame's class token. In plain language: tokens that the ViT's class token considers "more relevant" (higher attention score) contribute more strongly to the pooled output. The $\sqrt{C}$ scaling factor is the standard attention temperature that prevents dot products from growing too large with increasing dimensionality.

Why this form: Standard average pooling gives equal weight to every token in the block, which dilutes important high-activation features with background or redundant tokens. Max pooling selects only the single maximum-activation token, discarding all other potentially useful information. Subsampling (taking every $r$-th token) preserves neither saliency nor structural regularity — it arbitrarily picks tokens based on grid position rather than content importance. WAP occupies a middle ground: it retains continuous structural regularity (unlike attention-based pruning or token merging, which disrupt the spatio-temporal grid), but it incorporates content-aware weighting so that informative tokens dominate the pooled representation. The paper explicitly contrasts this with methods like ToMe (bipartite soft-matching across tokens), PruMerge (attention-based pruning), and FastVID (dynamic density pruning), arguing that these methods "disrupt the continuous spatio-temporal structure required for coherent video understanding" because their attention and matching scores are content-driven rather than position-driven, scrambling the spatial layout that the LLM relies on to understand object locations and movements.

Table 1 provides the empirical validation: under 16× compression (4× spatial, 4× temporal) applied to InternVL3-8B at 64 frames, WAP achieves 62.0% average accuracy across Video-MME (w/o and w/ subtitles), MLVU, and LongVideoBench, compared to 60.2% for average pooling, 60.0% for max pooling, 61.2% for subsampling, 60.0% for ToMe, 59.7% for PruMerge, and 60.6% for FastVID. The gap is particularly pronounced on MLVU (67.4% vs. 62.3% for average pooling), suggesting that content-aware weighting is especially valuable for long-form multi-task understanding.

---

The Spatio-Temporal Token Compressive Encoder Architecture

The LiteFrame student encoder is designed around two architectural principles: (1) it must be substantially more compact than the teacher (87M vs. 304M parameters), and (2) it must progressively reduce token count during the forward pass so that the final output is already compressed, with no separate post-hoc step needed.

Base backbone dimensions. The student uses a ViT-Base configuration: 12 transformer layers with 768-dimensional hidden states, compared to the teacher's ViT-Large: 24 layers with 1024-dimensional hidden states. The parameter counts are 87M and 304M respectively. The student begins with weights clipped from the teacher — the larger teacher's weight matrices are truncated to match the student's smaller dimensions, providing a strong initialization rather than random weights.

Temporal modeling via depth-wise temporal convolutions. To handle video input (multiple frames), the student must model temporal dynamics across frames. The paper explores several options and selects depth-wise 1D temporal convolutions (DWTempConv) interleaved with standard spatial attention layers.

The alternatives and their drawbacks, quantified in Table 2 (measured at 256 input frames on a single NVIDIA A100-80GB):

• Temporal attention (TempAttn): interleaving temporal self-attention blocks between spatial layers. This achieves 348.29 ms latency and 32.77 TFLOPs, but adds 57.5M parameters (143.83M total) because temporal attention requires separate key, query, value projection matrices.
• Full spatio-temporal attention (SpatioTempAttn): replacing the spatial attention with attention over all tokens across space and time simultaneously. This reduces latency to 204.35 ms (because it processes all tokens in one operation rather than sequentially per-frame) and FLOPs to 17.92 TFLOPs, with minimal parameter overhead (87.15M). However, Table 5 shows this underperforms DW temporal convolutions in accuracy (61.9% vs. 62.8% average at 128 frames under CTD only), likely because mixing spatial and temporal attention in a single operation makes it harder for the model to learn distinct spatial and temporal features.
• Standard temporal convolutions (TempConv): 1D convolutions along the temporal axis with full channel mixing. This achieves 202.08 ms latency and 22.44 TFLOPs, but adds 23.2M parameters (109.54M total) because the convolution kernels mix all input channels to all output channels.
• Depth-wise temporal convolutions (DWTempConv, selected): 1D convolutions along the temporal axis where each channel is convolved independently (no cross-channel mixing). This achieves the lowest latency (174.84 ms), lowest FLOPs (17.92 TFLOPs), and adds only 0.84M parameters (87.15M total). The depth-wise design means the temporal mixing is per-channel, which is computationally cheap but still allows the model to capture frame-to-frame dynamics in each feature dimension independently.

The DWTempConv layers are interleaved after every spatial attention layer where the temporal dimension is greater than 1. If the temporal dimension has been downsampled to 1 (all frames merged), no temporal convolution is applied at that depth, and the encoder falls back to pure spatial processing for the remaining layers.

Progressive spatio-temporal downsampling. To achieve internal token compression, the encoder integrates depth-wise strided convolution layers at strategic intervals that downsample the feature maps. Specifically, after the 4th transformer block, a strided convolution with stride $[t, h, w] = [2, 2, 2]$ reduces the temporal dimension by 2× and both spatial dimensions by 2× (8× total token reduction). After the 8th transformer block, another strided convolution with stride $[2, 1, 1]$ reduces only the temporal dimension by an additional 2×. The total compression is therefore $2 \times 2 \times 2 \times 2 = 16\times$: 4× spatial (from the first downsampling) and 4× temporal (cumulative from both downsamplings).

This progressive design means that deeper layers operate on fewer tokens, making them substantially cheaper than the equivalent layers in a standard frame-wise image encoder. In a standard encoder (like the InternViT-300M teacher), every layer processes the full token count for every frame, so the computational cost per layer is constant. In LiteFrame, the cost per layer decreases as the network deepens because the token count shrinks. This is the architectural mechanism that achieves the encoder efficiency gains quantified in Table 3: at 64 frames, LiteFrame vision encoding takes 54.8 ms vs. 40.0 ms for the teacher at just 8 frames — comparable per-frame cost despite processing 8× more frames.

Output format. The final output of the LiteFrame encoder is a tensor of shape $(t, h, w, D)$ where $D = 768$ (the student's hidden dimension), $t = T/4$ (temporal dimension reduced by 4×), and $h = H/2$, $w = W/2$ (spatial dimensions each reduced by 2×). For a standard input of $T$ frames at $448 \times 448$ resolution with patch size 14, the teacher would produce $T \times 32 \times 32 = 1024T$ tokens. LiteFrame produces $(T/4) \times 16 \times 16 = 64T$ tokens total — a 16× reduction. With 64 input frames, this means 4,096 tokens from LiteFrame vs. 65,536 tokens from the teacher — a dramatic reduction that makes the LLM's quadratic complexity manageable.

Why depth-wise temporal convolutions over attention: The paper argues that depth-wise convolution is fundamentally more parameter-efficient for temporal modeling than attention because it applies a fixed, local receptive field rather than learning pair-wise interactions across all temporal positions. For video, adjacent frames are highly redundant — the content changes slowly — so a local convolution (looking at a few neighboring frames) captures the relevant dynamics without the quadratic cost or large parameter count of temporal attention. The depth-wise variant further reduces parameters by treating each channel independently, which is sufficient because temporal dynamics for different visual features (edges, textures, object boundaries) are largely independent and don't require cross-channel mixing in the temporal dimension.

Why progressive downsampling over one-shot compression: Doing all compression in the first layer would throw away spatial detail before the model has a chance to extract meaningful features. By downsampling after the 4th and 8th blocks, the early layers process full-resolution features and can learn low-level patterns (edges, textures, simple shapes), while the later layers operate on compressed features and learn high-level semantics (objects, actions, scene context). The temporal downsampling specifically occurs in two stages (4× total) rather than one, allowing the model to first capture fine-grained frame-to-frame motion (at full temporal resolution) and then aggregate longer-range temporal patterns (at reduced resolution).

---

Compressed Token Distillation (CTD)

CTD is the core training framework that transfers knowledge from the large teacher encoder to the compact student. The central challenge is that the student must learn to produce semantically rich representations at drastically reduced resolution — a task that standard distillation handles poorly because it forces the student to mimic dense, redundant teacher outputs.

The supervision target. Instead of having the student predict the teacher's full-resolution output $T(x) = Z_T \in \mathbb{R}^{N \times D}$ (where $N$ is the number of tokens and $D$ is the feature dimension), CTD applies WAP to the teacher's output to create a compressed target. Let $\mathcal{P}(\cdot)$ be the WAP operator that aggregates the $N$ dense teacher tokens into $N/r$ compressed tokens (where $r$ is the target compression ratio, e.g., 16). The student's output is $S_\theta(x) = Z_S \in \mathbb{R}^{(N/r) \times D}$ — it directly predicts the compressed representation without ever computing the full-resolution intermediate.

The CTD loss function:

$\mathcal{L}_{\text{CTD}}(\theta) = \|S_\theta(x) - \mathcal{P}(T(x))\|_2^2$

where $S_\theta(x)$ is the student's output, $\mathcal{P}(T(x))$ is the teacher's WAP-compressed output, and $\|\cdot\|_2^2$ is the squared Euclidean (MSE) loss summed over all token positions and feature dimensions.

What it computes: For each compressed token position, the MSE loss measures the squared difference between the student's predicted feature vector and the teacher's WAP-aggregated feature vector. The total loss is the sum over all positions and dimensions, yielding a single scalar. Operationally: the teacher processes the input frames, producing dense features; WAP compresses these into a compact target; the student processes the same frames with its internal downsampling, producing features at the same compressed resolution; the MSE penalizes any deviation between student predictions and teacher targets.

Why this form: Standard distillation — $\|S(x) - T(x)\|_2^2$ where both are at the same resolution — would require the student to either (a) output at full resolution (defeating the efficiency goal) or (b) upsample its compressed output to match the teacher's resolution (introducing an upsampling module and forcing the student to learn features that will never be used). More fundamentally, standard distillation forces the student to model redundant spatial detail — two adjacent patches that look nearly identical must both be reconstructed, wasting the student's limited capacity.

CTD solves this by moving the compression into the target construction. The WAP operator $\mathcal{P}$ condenses the teacher's knowledge: it aggregates dense tokens into information-dense summaries, preserving salient features (via attention weighting) while discarding redundant ones. The student then learns to directly predict these summaries, effectively internalizing the WAP computation into its static weights. This means the student never learns to represent the redundant intermediate tokens — it goes straight from input pixels to compressed, saliency-weighted features.

The paper's ablation in Table 5 validates this design choice dramatically. Reconstructive Token Distillation (RTD) — an alternative where the student's compressed output is fed through a lightweight decoder trained to reconstruct the teacher's full dense features (an autoencoding objective) — achieves only 43.8% average accuracy vs. 62.8% for CTD at 128 frames (both without LMA). Even after LMA, RTD reaches only 61.5% vs. 63.4% for CTD+LMA. The authors attribute this gap to the fact that "explicitly aligning the student with the WAP primitive provides a much more robust, task-relevant semantic foundation than a generic reconstruction objective" — the autoencoder must preserve all visual information (including redundant textures, background details, and noise) to reconstruct the full feature map, while WAP naturally filters for the information the teacher's class token considers important, which turns out to be the information most useful for downstream LLM reasoning.

Training configuration for CTD. The paper provides specific hyperparameters in Appendix A:

• Optimizer: AdamW with a cosine learning rate schedule and linear warmup.
• Maximum learning rate: $4 \times 10^{-5}$, reduced for specific variants susceptible to training instability.
• Warmup: 100 epochs.
• Total training duration: 1,800 epochs (approximately 21 days on 8× NVIDIA H100 GPUs; ablation studies use 800 epochs for efficiency).
• Global batch size: 512.
• Input sampling: 4-frame clips with frame rate (FPS) uniformly sampled from $[1, 4]$ — this ensures the student sees videos at varying temporal resolutions during training, promoting robustness to different frame rates at inference.
• Initialization: Student weights are initialized by clipping the teacher's weights to match the student's smaller dimensions. For example, a teacher weight matrix of shape $1024 \times 1024$ is truncated to $768 \times 768$ for the student, and teacher embedding dimensions of 1024 are truncated to 768.
• Stability techniques: MSE outlier clipping (clipping target-prediction differences exceeding 3× the standard deviation of the current batch) and gradient clipping with maximum norm 1.0.

Distillation datasets. The paper uses a diverse mix of video data described in the InternVL2.5 paper, including ShareGPT4Video (video-caption pairs with rich descriptions), LLaVA-Video-178K (video instruction-tuning data), FineVideo (fine-grained video understanding), CLEVRER (synthetic collision events requiring causal reasoning), and NTURGB+D (human activity recognition). This diversity ensures the student learns generalizable video features rather than overfitting to a narrow domain.

Why distillation rather than training from scratch: Training a 87M-parameter video encoder from scratch on this dataset mix would likely converge to weaker representations because the model lacks the inductive biases and feature hierarchies learned by the 304M-parameter teacher during its much larger-scale pretraining (the teacher was pretrained on massive image datasets). Distillation transfers these pretrained representations — the student inherits the teacher's ability to recognize objects, textures, spatial relationships, and motion patterns, then learns to compress this knowledge into a more efficient form factor.

Why WAP target construction rather than learning the compression: The RTD ablation (Table 6) directly tests this. A learnable compression (autoencoder) produces features that preserve all visual information equally, which is suboptimal because the LLM only needs certain types of information (object identities, actions, spatial relationships) and not others (exact texture details, background noise, lighting variations). WAP, by using the class token's attention scores as weights, naturally filters for the most semantically relevant information — the class token is trained during the teacher's pretraining to aggregate information for classification tasks, so its attention pattern highlights diagnostically useful image regions. The student inherits this semantic filtering by mimicking the WAP output.

---

Language Model Adaptation (LMA)

After CTD, the student encoder produces compressed visual features that capture the teacher's salient information. However, these features exist in a latent space that was optimized for matching the teacher's WAP output — not for the LLM's input expectations. The Language Model Adaptation stage bridges this gap.

What LMA does. The distilled student encoder is plugged into the full Video LLM pipeline (replacing the teacher), and both the encoder and the LLM are fine-tuned jointly on video-text pairs. The objective is the standard causal language modeling loss — cross-entropy between predicted and ground-truth text tokens, conditioned on the video input processed by LiteFrame. In plain language: the model watches a video, the encoder compresses it into tokens, and the LLM is trained to generate the correct answer/caption; gradients flow back through the LLM into the encoder, teaching the encoder to produce features that are not just similar to the teacher, but specifically useful for the LLM's language generation task.

LoRA fine-tuning for the LLM. To preserve the LLM's general reasoning capabilities and ensure training efficiency, the paper applies Low-Rank Adaptation (LoRA) to the LLM rather than full fine-tuning. The specific configuration:

• Rank $r = 4$: Low-rank update matrices of rank 4 are added to the LLM's weight matrices. The paper notes that "extensive experiments demonstrate that a lower rank (e.g., 4) performs better than higher ones (e.g., 8 and 16)" — higher ranks may overfit to the relatively small video adaptation dataset, degrading the LLM's general language capabilities.
• Alpha $\alpha = 8$: The LoRA scaling factor, which controls the magnitude of the low-rank update relative to the original weights.
• Dropout $= 0.05$: Moderate dropout for regularization.

Training configuration for LMA:

• Learning rate: $4 \times 10^{-5}$ with a cosine schedule.
• Effective batch size: 128 (achieved via gradient accumulation).
• Training duration: 25,000 steps (completes in a few hours on 8× NVIDIA H100 GPUs).
• Frame sampling: Uniformly sampled from $\{128, 256, 512\}$ frames, with FPS ranging from 1 to 4. This is a deliberate choice: the total visual token volume (frames × tokens per frame) should match that of the teacher's typical input. For the teacher processing 8–32 frames at 256 tokens per frame, the visual token budget is 2,048–8,192. For LiteFrame at 128–512 frames at 16 tokens per frame, the visual token budget is also 2,048–8,192. Matching the token volume ensures the LLM sees a comparable amount of visual information during adaptation as it did during its original pretraining/fine-tuning.
• Datasets: High-quality video-question answering pairs from LLaVA-Video-178K and FineVideo, plus captioning datasets from ShareGPT4Video and OpenVid-1M. These provide both generative (captioning) and discriminative (QA) supervision.

Why LMA is necessary. The paper's ablation in Table 5 shows that CTD alone (without LMA) achieves 62.8% average accuracy at 128 frames. Adding LMA boosts this to 63.4% — a modest but consistent gain. More importantly, LMA provides two benefits beyond raw accuracy:

1. Modality gap bridging: The teacher's WAP-compressed features, while semantically rich, are not necessarily in the optimal format for the LLM's input projector. The alignment projector was originally trained to map the teacher's full-resolution features to the LLM's text embedding space. After CTD, there's a distribution shift — the student's compressed features have different statistics (different spatial resolution, different feature distributions) than what the projector expects. LMA fine-tunes both the encoder output and the projector to close this gap.

2. Long-context adaptation: Standard LLMs in Video LLM pipelines were fine-tuned with a certain range of visual token counts (matching the teacher's typical 8–32 frames at 256 tokens/frame = 2,048–8,192 tokens). LiteFrame drastically changes this: processing 256 frames at 16 tokens/frame still yields 4,096 tokens, within the familiar range, but the temporal structure is different — the LLM must learn to reason over tokens that represent 256 time steps rather than 32 time steps. LMA teaches the LLM to handle this extended temporal context without degradation.

The student encoder is also fine-tuned during LMA. Unlike traditional adapter-based approaches where the encoder is frozen, LiteFrame's encoder continues to receive gradient updates during LMA. This means the encoder's features can shift from "similar to WAP-compressed teacher" to "optimal for LLM text generation" — a refinement that the MSE-based CTD loss cannot provide because it cares only about feature similarity, not downstream task performance.

---

Full Inference Pipeline

At inference time, the system operates as follows:

1. Frame sampling. Videos are sampled using a dense clip strategy rather than uniform frame sampling. Multiple clips are uniformly sampled across the video duration, where each clip consists of a fixed number (4) of frames extracted at a minimum of 1 FPS. All sampled frames are resized to 448×448 pixels before feeding to the encoder. This contrasts with the standard uniform frame sampling used for the teacher baseline — the clip-based approach ensures that the model sees contiguous temporal segments with natural motion, rather than isolated frames that may miss fast actions occurring between samples.

2. Vision encoding. The LiteFrame encoder processes all frames through its 12-layer backbone with interleaved DWTempConv layers and two progressive downsampling stages (after layers 4 and 8). The output is a compressed token grid with 4× temporal reduction and 4× spatial reduction (16× total), yielding 16 tokens per input frame. For $T$ input frames, the LLM receives $16 \times T/4 = 4T$ tokens total — e.g., 256 tokens for 64 frames, 512 tokens for 128 frames, 1,024 tokens for 256 frames.

3. LLM prefilling. The compressed visual tokens are projected into the LLM's embedding space via the alignment projector and concatenated with text tokens (the question/prompt). The LLM performs a single forward pass over all visual and text tokens to compute key-value caches.

4. Text generation. The LLM generates the answer autoregressively, conditioned on the cached visual representations.

Efficiency breakdown (Table 3). The paper provides a detailed latency comparison between InternVL3-8B (teacher encoder) and InternVL3-8B + LiteFrame at three frame scales:

• At ~200 ms total latency budget: InternVL3-8B processes 8 frames (40.0 ms vision + 167.3 ms LLM = 208.4 ms total) with 256 tokens/frame and 59.0% average accuracy. LiteFrame processes 64 frames (54.8 ms vision + 94.9 ms LLM = 150.1 ms total, -28.0%) with 16 tokens/frame and 61.1% accuracy (+2.1 pp).

• At ~400 ms total latency budget: InternVL3-8B processes 16 frames (74.0 ms vision + 329.3 ms LLM = 403.6 ms) with 62.2% accuracy. LiteFrame processes 128 frames (105.3 ms vision + 166.6 ms LLM = 272.6 ms, -32.5%) with 63.7% accuracy (+1.5 pp).

• At ~800 ms total latency budget: InternVL3-8B processes 32 frames (144.5 ms vision + 669.8 ms LLM = 814.5 ms) with 65.3% accuracy. LiteFrame processes 256 frames (204.0 ms vision + 327.4 ms LLM = 532.3 ms, -34.6%) with 65.7% accuracy (+0.4 pp).

The key pattern: as frame counts scale, the teacher's LLM latency explodes (167.3 → 669.8 ms at 4× frames, roughly 4× growth due to quadratic attention), while LiteFrame's LLM latency grows much more slowly (94.9 → 327.4 ms at 4× effective temporal resolution). The vision encoder latency also scales more favorably: 40.0 → 144.5 ms (3.6×) for the teacher vs. 54.8 → 204.0 ms (3.7×) for LiteFrame, but LiteFrame starts from a much lower per-frame cost and processes 8× more frames at each tier.

Zero-shot spatial scaling (Section 5.2.3). Beyond temporal scaling, the token efficiency of LiteFrame enables a different form of scaling: processing higher spatial resolutions within a fixed LLM context length. The paper implements a zero-shot tiling strategy: high-resolution frames are split into 448×448 pixel sub-tiled clips, each processed independently by LiteFrame (producing 16 tokens each), and the concatenated tokens are fed to the LLM. This allows the model to "zoom in" on fine spatial details without sacrificing temporal resolution. The result on HLVid (Figure 6): LiteFrame achieves 54.1% at 2,688px resolution with 48 frames, surpassing AutoGaze's 52.6% at 3,584px with 1,024 frames — better spatial understanding with far fewer tokens and lower resolution, because the encoder's efficiency preserves the token budget for temporal context.

Latency measurement protocol. The paper specifies (Appendix A) that latency is measured as the median over 100 iterations following a 40-iteration warmup (140 total iterations) on a single NVIDIA A100-80GB GPU. Only vision encoding and LLM prefilling are measured — autoregressive text generation is excluded because it depends on output length and is not the bottleneck addressed by this work. This focus on encoding + prefilling is justified because these are the stages that scale with input frame count and visual token volume, while text generation depends only on output length (typically a few hundred tokens for QA).

---

Design Decisions and Their Justifications

Why 16× compression specifically? The paper explores 4×, 8×, and 16× compression in Figure 3 (right) and finds that 16× performs best under a fixed token budget because it allows proportionally more frames to be processed — the accuracy gains from richer temporal context outweigh the information loss from spatial compression. Going beyond 16× is not explored, likely because further spatial compression (e.g., reducing a 32×32 grid to 4×4) would destroy the spatial layout needed for object localization and relationship reasoning.

Why DWTempConv over SpatioTempAttn? Table 2 shows SpatioTempAttn has comparable latency (204.35 ms vs. 174.84 ms) and identical FLOPs (17.92 TFLOPs). However, Table 5 shows DWTempConv achieves 62.8% accuracy vs. 61.9% for SpatioTempAttn under CTD-only at 128 frames. The paper doesn't provide a deep analysis of this gap, but one likely explanation is that separating spatial and temporal processing (spatial attention followed by temporal convolution) creates an inductive bias that helps the model learn: spatial attention captures within-frame patterns (object shapes, textures), and temporal convolution captures cross-frame dynamics (motion, change). Full spatio-temporal attention mixes these, potentially creating a harder optimization landscape where the model must disentangle spatial and temporal signals from a single attention operation.

Why initialize from teacher weights rather than random? The student is architecturally different from the teacher (fewer layers, smaller hidden dimension, added temporal convolutions and strided downsampling layers), so weight initialization is non-trivial. Clipping the teacher's weights provides a starting point that already encodes useful visual features (edge detectors, shape recognizers, object-level semantics), dramatically accelerating convergence. Training from scratch would require the student to rediscover these basic visual primitives, which is wasteful when the teacher has already learned them.

Why two-stage training (CTD then LMA) rather than end-to-end? Training the encoder from scratch jointly with the LLM would be extremely unstable: the LLM's language modeling loss provides a weak and noisy signal for learning low-level visual features. CTD provides a strong, direct supervision signal (MSE to teacher features at every token position) that teaches the encoder the fundamental visual representation. LMA then refines this representation for the specific LLM and task. This two-stage approach is analogous to pretraining then fine-tuning, which is standard practice for LLMs themselves.

4. Key Insights and Innovations

Innovation 1: Identifying the Bottleneck Shift as a First-Class Architectural Problem

The paper's most intellectually distinctive contribution is not a new compression technique, but a diagnostic reframing of the Video LLM efficiency problem. Prior work — spanning ToMe, PruMerge, FastVID, VisionZip, HoliTom, and dozens of other post-hoc token reduction methods — operated under a shared, implicit assumption: the LLM's quadratic self-attention over visual tokens is the bottleneck, and fixing it fixes the system. This assumption was so deeply embedded that the entire subfield organized around an "extract-and-reduce" paradigm where the vision encoder was treated as an immutable black box — always the 304M-parameter InternViT-300M or its equivalent, always processing every frame at full resolution, always assumed to be negligible next to the LLM's costs.

The paper demolishes this assumption with a single, crisp observation formalized in Section 3 as the scaling paradox: when post-hoc reduction is aggressive enough to genuinely alleviate the LLM bottleneck (e.g., 16× compression), the vision encoder's per-frame cost — which scales linearly with frame count — becomes the dominant latency term. This is not a minor correction. It reveals that the entire post-hoc paradigm has a hard latency floor that no amount of token-reduction cleverness can penetrate. You can compress after extraction to 1/16th the token count, but you cannot avoid paying the full ViT forward pass for every frame you want to process. As Figure 1(b) illustrates, the vision encoder latency "explodes" as frame counts increase under post-hoc methods, tracing a line that no post-hoc method can get underneath.

What makes this a genuine innovation rather than an obvious engineering observation is the counterintuitive conclusion it forces. The natural response to "the encoder is too expensive" would be "build a more efficient encoder" — and indeed, prior work like FastVLM and MobileNet-v5 did exactly that. But those efforts were image-centric, optimizing per-frame spatial encoding without addressing the compounding effect of temporal redundancy. The paper argues that the bottleneck shift is not just about per-frame cost — it's about the interaction between encoder cost and frame scaling. A 2× faster per-frame encoder helps, but if you want to process 8× more frames, you still face 4× the absolute encoder latency. The real problem is that post-hoc methods create a structural inefficiency: the encoder computes information that is immediately discarded by the compression step, wasting FLOPs that could have been avoided entirely.

This diagnostic reframing is significant beyond its immediate practical implications. It changes how the field should think about the Video LLM architecture stack — not as a sequence of independent components (encoder, projector, LLM) where each can be optimized in isolation, but as a coupled system where optimizing one component shifts the bottleneck to another. The paper's "scaling paradox" is a specific instance of a more general principle: in multi-stage pipelines with heterogeneous cost scaling (linear for the encoder, quadratic for the LLM), the optimal allocation of resources is not static — it depends on the operating point (frame count, compression ratio). This is a conceptual contribution that echoes Amdahl's Law from computer architecture, translated to the neural network regime.

The evidence for this innovation is primarily in the architecture of the paper's argumentation, not a single table. Figure 1(b) visualizes the bottleneck shift schematically. Table 3 provides the quantitative proof: at comparable total latency (~400 ms), InternVL3-8B + FastVID spends 310.6 ms on vision encoding (77% of total) vs. 95.4 ms on the LLM — the encoder has become the bottleneck that FastVID was designed to solve for the LLM. Figure 2 demonstrates the consequence: all post-hoc methods (dashed colored lines) cluster in a region of the latency-accuracy plane that LiteFrame leaves behind entirely, because they cannot escape the encoder latency floor.

Innovation 2: Internalizing Token Compression via Distillation as a New Training Paradigm

The second major conceptual move is the architectural internalization of compression through distillation. Prior work on efficient video encoders for MLLMs fell into two categories, neither of which solved the full problem. Encoder-free methods like Video-Panda eliminated the vision backbone entirely but produced dense, uncompressed token streams that overwhelmed the LLM — solving the vision problem by resurrecting the language problem. Efficient spatial encoders like FastVLM reduced per-frame cost but didn't exploit temporal redundancy, so they still scaled linearly with frame count. Pre-filtering methods like AutoGaze attempted to reduce tokens before the encoder, but introduced auxiliary modules whose overhead negated the theoretical gains (3.0 seconds out of 6.1 seconds total latency spent in the pre-reduction module, per Figure 7).

LiteFrame's innovation is a unified solution to this coupled problem: make the encoder itself produce compressed outputs, so there is no separate compression step, no auxiliary module, and no dense intermediate representation. The encoder's forward pass is the compression — the progressive strided convolutions don't add a step, they replace what would have been full-resolution processing in deeper layers.

But this architectural idea — a token-compressive encoder — is only half the innovation. The deeper contribution is Compressed Token Distillation (CTD), which solves the training problem that makes such an encoder possible. Training a compact student to be simultaneously (a) semantically rich enough to match a 304M-parameter teacher, and (b) token-compressive enough to output 16× fewer tokens, is a genuinely hard optimization problem. Standard distillation would force the student to either output full-resolution features (defeating the efficiency goal) or upsample its compressed output (adding a module and forcing the student to model redundant detail it will never use).

CTD's key insight is to move the compression into the target construction, not the student's architecture. By applying WAP to the teacher's output before using it as a supervision target, CTD constructs an information-dense, saliency-weighted target that the student can directly predict at its native (compressed) resolution. The student never sees the dense intermediate; it never learns to represent tokens that will be discarded. It goes straight from input pixels to compressed, semantically filtered features.

The significance of this move extends beyond the specific WAP operator. CTD establishes a general template for knowledge distillation under resolution mismatch: when the student must operate at lower resolution than the teacher, don't have the student mimic the teacher's high-resolution output (which would require the student to waste capacity on redundant detail). Instead, use the teacher's own aggregation mechanism (attention pooling, in this case) to construct a lower-resolution target that preserves the teacher's semantic judgments about what information matters. The student then learns to replicate the teacher's judgments (what to attend to, what to compress) rather than the teacher's raw features.

The paper validates this conceptual claim through a stark ablation: Reconstructive Token Distillation (RTD), which uses an autoencoding objective to reconstruct the teacher's full dense features from the student's compressed latents, achieves only 43.8% average accuracy vs. 62.8% for CTD at 128 frames (Table 5, both without LMA). Even after LMA, RTD (61.5%) trails CTD+LMA (63.4%) by nearly 2 percentage points (Table 6). The autoencoder is forced to preserve all visual information — including textures, backgrounds, and noise — to reconstruct the full feature map. WAP, by weighting tokens according to the class token's attention scores, naturally filters for diagnostically useful information, and CTD transfers this filtering into the student's static weights. The nearly 19-percentage-point gap between CTD and RTD before LMA is one of the paper's most striking results, and it demonstrates that what you distill matters far more than the distillation loss function itself.

This is a fundamental shift from the dominant paradigm in efficient vision models, where the focus has been on architectural innovations (depth-wise convolutions, token merging, pruning) and the training objective is typically standard cross-entropy or feature-matching MSE. CTD shows that the training objective design — specifically, the construction of the supervision target — can achieve compression gains that architectural changes alone cannot, by exploiting the teacher's learned attention patterns as a semantic filter.

Innovation 3: The Frame-Count Bottleneck as a Trade-off Between Spatial Fidelity and Temporal Context

A recurring theme in efficient video understanding has been the tension between spatial and temporal resolution. Higher spatial resolution captures fine details (text on signs, small objects, facial expressions); higher temporal resolution captures fast motion, causal sequences, and long-range dependencies. The two compete for a fixed token budget, and prior work has largely treated this as an engineering trade-off to be tuned per application.

The paper reframes this tension through a specific empirical finding that has broader theoretical implications. Figure 3 (right) demonstrates that under a fixed visual token budget, processing more frames with more aggressive compression consistently outperforms processing fewer frames with less compression. The 16× WAP variant (64 frames, 1,024 visual tokens) achieves higher average accuracy than the 8× variant (32 frames, 1,024 tokens), which in turn outperforms the 4× variant (16 frames, 1,024 tokens). This is not an obvious result — one might expect that compressing spatial information by 16× would destroy so much detail that the accuracy loss outweighs the temporal gain. The data says otherwise.

The paper's interpretation — that "aggressive compression effectively trades redundant tokens for richer temporal context" — points to a deeper property of video data: spatial redundancy is much higher than temporal redundancy for the types of understanding that current benchmarks test. Most video frames contain large regions of background, repeated textures, and slowly changing scenes where 256 tokens per frame is massive overkill. The differences between frames — the motion, the events, the causal sequences — are what carry the narrative and semantic content, and these differences require temporal resolution to capture, not spatial resolution.

This finding has implications for how the field should think about video tokenization. The standard approach — treat every frame as an independent image, extract 256+ tokens, and then figure out how to reduce them — implicitly assumes that spatial information is the primary carrier of meaning in video. The paper's results suggest this assumption is wrong for the current generation of benchmarks and likely for many real-world video understanding tasks: temporal context is the scarce resource, and spatial detail is the abundant one. The optimal allocation is therefore to spend the token budget on more time steps, not on finer spatial grids.

The LiteFrame architecture operationalizes this insight by distributing compression across both dimensions (4× spatial, 4× temporal) rather than concentrating it in one. The ablation in Section C.3 demonstrates that spatial-only 16× compression significantly underperforms spatio-temporal 4× + 4× compression — at 128 frames, the spatial-only variant achieves 60.5% average accuracy vs. 62.8% for spatio-temporal (Table 7). The authors attribute this to the "excessive loss of spatial fidelity" when destructively pooling a frame into a 4×4 grid, versus the more balanced approach of moderate spatial pooling (32×32 → 16×16, still retaining object-level spatial structure) combined with temporal pooling that aggregates redundant frame-to-frame content. This is a concrete architectural manifestation of the frame-count insight: the compression budget should be allocated where redundancy is highest, which for video is across both space and time, not just space.

The zero-shot spatial scaling result on HLVid (Figure 6) provides a striking corroboration. LiteFrame achieves 54.1% at 2,688px resolution with only 48 frames, surpassing AutoGaze's 52.6% at 3,584px with 1,024 frames. The model achieves better fine-grained spatial understanding with fewer tokens and lower resolution because its token efficiency allows it to maintain temporal context while scaling spatially — a trade-off that the standard paradigm (where spatial resolution eats directly into the temporal budget) cannot make. This is not just an engineering win; it demonstrates that the frame-count bottleneck insight generalizes beyond the specific compression ratios tested, pointing toward a design principle: build encoders that decouple spatial and temporal token costs, so that scaling one dimension does not force a sacrifice in the other.

Innovation 4: The Weighted Average Pooling Primitive as a Proof of Concept for Attention-Guided Compression

Weighted Average Pooling (WAP) might appear at first glance to be a minor technical tweak — average pooling with attention weights instead of uniform weights. But its role in the paper is more significant: it serves as an existence proof that simple, structurally regular compression operators can outperform complex, content-driven ones when the downstream consumer is a language model that relies on spatial coherence.

The dominant trend in token compression has been toward increasing sophistication: ToMe uses bipartite soft-matching to merge similar tokens, PruMerge uses attention scores to prune unimportant tokens, FastVID uses dynamic density pruning based on token importance scores. These methods all share a common philosophy: the optimal compression is content-dependent, where tokens are merged or pruned based on their semantic similarity or estimated importance, regardless of their spatial positions.

The paper's counter-claim — supported by Table 1 — is that for Video LLMs specifically, structural regularity matters more than content-adaptive sophistication. WAP (62.0% average accuracy) outperforms ToMe (60.0%), PruMerge (59.7%), and FastVID (60.6%), all of which use more complex content-based selection. Why? The paper's explanation is that these methods "disrupt the continuous spatio-temporal structure required for coherent video understanding." When tokens are merged based on feature similarity rather than spatial proximity, the resulting token grid no longer corresponds to a coherent spatial layout — a merged token might combine patches from opposite corners of the frame if they happen to share similar textures. For an LLM that needs to reason about object locations, spatial relationships, and motion trajectories, this scrambled layout is harmful, even if the individual merged tokens carry more "information" in an information-theoretic sense.

This insight has broader implications for the design of visual tokenizers for language models. The vision community has spent years developing sophisticated compression techniques (learned pruning, clustering-based merging, reinforcement-learning-based selection) under the assumption that maximizing information preservation per token is the right objective. The paper's results suggest that for language model consumers, the format of the information matters as much as the quantity — a regular grid of slightly suboptimal tokens may be better than an irregular set of optimally informative tokens, because the LLM's attention mechanism is implicitly structured to process sequences with predictable spatial relationships.

WAP's second role is as a bridge between the post-hoc and internalized paradigms. By demonstrating that a simple, fast compression operator can match or exceed complex alternatives, WAP makes the case that compression doesn't require heavy machinery — which in turn makes the case that it can be internalized into a lightweight encoder. If the best compression required a 304M-parameter model's worth of computation, internalizing it into an 87M-parameter student would be hopeless. But because a simple weighted pooling operation suffices, the student can plausibly learn to replicate it.

The paper doesn't claim WAP is the optimal compression operator — it explicitly calls it a "primitive" and a "proof of concept." The deeper innovation is the methodology: use a strong post-hoc operator as the teacher's target representation, then distill the student to predict that representation directly. This methodology is independent of the specific operator; if future work discovers a better compression primitive (perhaps learned, or incorporating motion-based weights, or using multiple class tokens), the same CTD framework applies. WAP's role is to establish that this methodology works and produces results that surpass both the post-hoc paradigm and alternative efficient encoder designs.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses four widely-adopted video understanding benchmarks: Video-MME (with and without subtitles; Fu et al., 2025), MLVU (Zhou et al., 2025), and LongVideoBench (Wu et al., 2024). These are specifically chosen as "long video benchmarks" where accuracy exhibits logarithmic scaling with input frame count (Figure 3, left), making them the natural stress test for frame-scaling approaches. For zero-shot spatial resolution scaling experiments (Section 5.2.3), the paper uses HLVid (Shi et al., 2026), which requires high-fidelity spatial understanding across video frames. Additional results are reported on short video benchmarks MVBench (Li et al., 2024b) and TVBench (Cores et al., 2024) in Appendix B.1, and on long video benchmarks LVBench (Wang et al., 2024) and MMBench-Video (Fang et al., 2024) in Appendix B.2.

• Base model(s). The primary baseline is InternVL3-8B (Zhu et al., 2025), a state-of-the-art open-source Video LLM that uses InternViT-300M (304M parameters, 1024 hidden dimensions, 24 layers) as its image encoder. This model is chosen because it is "representative of the canonical approach to scaling pretraining compute" (Section 5.1) and its architecture follows the standard paradigm — frame-wise image encoder + alignment projector + LLM — that the paper critiques. For comparisons with AutoGaze, the paper uses NVILA-HD-8B-Video as the corresponding baseline (Appendix D), evaluated both with its standard ImageViT and with the VideoViT backbone that AutoGaze structurally requires.

• Metrics. The primary metric throughout is average accuracy (%) across the four main benchmarks (Video-MME w/o subtitles, Video-MME w/ subtitles, MLVU, and LongVideoBench). Each benchmark has its own evaluation protocol — Video-MME uses multiple-choice question answering, MLVU uses multi-task evaluation across diverse video understanding tasks, and LongVideoBench uses interleaved video-language understanding. Individual benchmark accuracies are reported in tables alongside the average. For efficiency, end-to-end latency (milliseconds) is measured as the sum of vision encoding time and LLM prefilling time, specifically excluding autoregressive text generation (since it depends on output length and is not the bottleneck this work addresses; Appendix A). Latency is reported as the median over 100 iterations following a 40-iteration warmup on a single NVIDIA A100-80GB GPU.

• Baselines. The paper compares against several categories of methods:

  • Post-hoc token reduction methods: ToMe (Bolya et al., 2023) — bipartite soft-matching token merging; LLaVA-PruMerge (Shang et al., 2025) — attention-based adaptive token pruning; FastVID (Shen et al., 2025) — dynamic density pruning for video; these are applied with a 16× compression ratio to InternVL3-8B.
  • Standard pooling baselines: Average Pooling, Max Pooling, Subsampling — all applied post-hoc to InternVL3-8B at 16× compression.
  • Efficient vision encoders for MLLMs: FastVLM (Vasu et al., 2025) — hybrid convolutional-transformer image encoder (125M parameters); Video-Panda (Yi et al., 2025) — encoder-free paradigm with Spatio-Temporal Alignment Block (45M parameters).
  • Pre-filtering approach: AutoGaze (Shi et al., 2026) — autoregressive gazing module that pre-filters tokens before a heavy VideoViT backbone.
  • Internal baselines for ablations: "Distill (No Comp.)" — standard distillation of teacher into ViT-Base-12L without token compression; "Spatial 16×" — spatial-only compression without temporal downsampling; "RTD" — Reconstructive Token Distillation using autoencoding objective instead of WAP targets.

• Generation budget / compute accounting. The paper measures inference-time compute through two tightly coupled quantities: frame count (number of input frames processed) and tokens per frame (determined by the encoder architecture and any compression applied). For fair latency comparisons, methods are matched at comparable total latency budgets — e.g., InternVL3-8B at 8 frames (208.4 ms total) is compared against LiteFrame at 64 frames (150.1 ms total) because both occupy the ~200 ms regime (Table 3). The critical budgeting constraint is that total visual tokens fed to the LLM should not exceed what the LLM can handle given its context length limits and quadratic attention complexity. LiteFrame's 16× compression (16 tokens/frame vs. 256 tokens/frame for the teacher) means it can process 8× more frames while producing fewer total visual tokens — e.g., 64 frames × 16 tokens = 1,024 tokens vs. 8 frames × 256 tokens = 2,048 tokens. Latency for all methods is measured identically: ViT processing + LLM prefilling, median of 100 iterations after 40 warmup, single NVIDIA A100-80GB GPU.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for model selection or hyperparameter tuning in the standard sense, as LiteFrame is not selecting among strategies per-sample. Instead, it reports aggregate accuracy across four benchmarks and uses consistent evaluation protocols (multiple-choice accuracy for Video-MME, standard task-specific metrics for MLVU and LongVideoBench). Latency measurements use median-of-100 with warmup to ensure stable, representative timing values. The paper compares LiteFrame against baselines at multiple operating points (frame counts of 8, 16, 32, 64, 128, 256, 512) to demonstrate Pareto frontier improvements rather than single-point comparisons, which naturally provides robustness to the specific operating point chosen.

Main Quantitative Results

Redefining the Pareto Frontier: LiteFrame vs. InternVL3-8B Teacher

The headline result, presented in Table 3 and visualized in Figure 2, is that LiteFrame establishes a new latency-accuracy Pareto frontier compared to the InternVL3-8B baseline and all tested post-hoc compression methods. At three matched latency tiers:

• ~200 ms total latency regime: InternVL3-8B processes 8 frames and achieves 59.0% average accuracy (40.0 ms vision + 167.3 ms LLM = 208.4 ms total). LiteFrame processes 64 frames — 8× more — and achieves 61.1% average accuracy (54.8 ms vision + 94.9 ms LLM = 150.1 ms total), representing a 28.0% latency reduction with +2.1 percentage point accuracy improvement. The post-hoc baseline (InternVL3-8B + FastVID at 32 frames) achieves only 59.5% accuracy with 224.8 ms latency — a 7.9% latency increase for a marginal +0.5 pp gain.

• ~400 ms total latency regime: InternVL3-8B processes 16 frames and achieves 62.2% accuracy (403.6 ms total). LiteFrame processes 128 frames and achieves 63.7% accuracy (272.6 ms total) — a 32.5% latency reduction with +1.5 pp accuracy improvement. FastVID at 64 frames achieves only 59.5% accuracy at 406.2 ms — a net degradation of -2.7 pp from the 16-frame baseline, despite processing 4× more frames.

• ~800 ms total latency regime: InternVL3-8B processes 32 frames and achieves 65.3% accuracy (814.5 ms). LiteFrame processes 256 frames and achieves 65.7% accuracy (532.3 ms) — a 34.6% latency reduction with +0.4 pp accuracy improvement. FastVID at 128 frames degrades to 62.9% accuracy at 794.9 ms — a -2.4 pp loss.

The key pattern across all three regimes: LiteFrame simultaneously improves accuracy and reduces latency, while the post-hoc baseline (FastVID) either degrades accuracy or provides marginal gains at increased latency. The accuracy gap between LiteFrame and the teacher is largest at the most constrained budget: +2.1 pp at ~200 ms, narrowing to +0.4 pp at ~800 ms. This makes intuitive sense — when compute is severely limited, the ability to process 8× more frames provides the greatest relative advantage; at higher budgets, the teacher catches up somewhat by processing more frames (32 vs. 8), though LiteFrame still maintains an edge.

Figure 2 visualizes this Pareto frontier improvement across a range of post-hoc methods (ToMe, FastVID, PruMerge) and LiteFrame variants (with and without LMA, at different frame counts). All post-hoc methods cluster in the moderate-accuracy, moderate-to-high-latency region, while LiteFrame (red stars) occupies the high-accuracy, low-latency corner. The figure demonstrates that no post-hoc method, regardless of its compression sophistication, can break through the encoder latency floor that LiteFrame eliminates.

Comparison with Post-Hoc Methods Under Frozen LLM

To isolate the contribution of the encoder architecture from the effects of LLM adaptation, the paper evaluates LiteFrame using only CTD (no LMA), keeping the LLM entirely frozen — the same condition as training-free post-hoc methods. Figure 5 presents this comparison:

LiteFrame (CTD-only) with its 87M-parameter video encoder operating at 64 frames achieves higher accuracy than all post-hoc methods (ToMe 16×, FastVID 16×, PruMerge 16×) at comparable or lower latency. The post-hoc methods are all bottlenecked by the teacher's 304M-parameter InternViT-300M, which processes every frame at full resolution before compression. As frame counts increase, the ViT latency "explodes" relative to LiteFrame's internalized compression.

This result demonstrates that the encoder architecture swap alone — even without task-specific LLM fine-tuning — provides an efficiency gain that surpasses the entire post-hoc paradigm. It also validates that CTD successfully transfers the teacher's visual knowledge into the compact student: the frozen LLM, which was originally trained to process the teacher's full-resolution features, can work with LiteFrame's compressed features without catastrophic accuracy loss. In fact, it works better than the teacher + post-hoc compression, because the student is specifically trained to produce features that are optimized for the compressed representation, whereas post-hoc compression applies a lossy operation to features that were optimized for full-resolution use.

Scaling Behavior: Frames, Latency, and Accuracy

Table 3 reveals non-trivial scaling patterns that bear close examination. For LiteFrame, accuracy improves monotonically with frame count: 61.1% at 64 frames → 63.7% at 128 frames → 65.7% at 256 frames. The gains exhibit diminishing returns — +2.6 pp from 64 to 128, +2.0 pp from 128 to 256 — consistent with the logarithmic scaling observed in Figure 3 (left). Latency scales sub-linearly with frame count for LiteFrame: 150.1 ms → 272.6 ms → 532.3 ms, representing increases of 1.82× and 1.95× for 2× frame increases, not the 2× one would expect from linear scaling. This sub-linear scaling is a direct consequence of the progressive downsampling architecture: deeper layers operate on fewer tokens, so the marginal cost of additional frames decreases.

For InternVL3-8B, latency scales approximately linearly with frames in the LLM (167.3 → 329.3 → 669.8 ms, roughly 2× per 2× frames), but super-linearly in the ViT (40.0 → 74.0 → 144.5 ms), likely because ViT processing of additional frames at full 256-token resolution creates memory pressure effects at higher batch sizes. The total latency scaling (208.4 → 403.6 → 814.5 ms, roughly 1.94× and 2.02×) is dominated by the LLM component.

The Post-hoc baseline (FastVID) shows a more complex pattern. At low frames (32), the LLM latency is reduced compared to the teacher (63.0 ms vs. 167.3 ms), showing that the compression successfully alleviates the LLM bottleneck. But the vision encoder latency grows proportionally with frame count (161.7 ms for 32 frames, 310.6 ms for 64, 625.8 ms for 128) — almost perfectly linear (2.00×, 1.92×, 2.01×). This confirms the paper's central diagnostic: post-hoc compression shifts the bottleneck from the LLM to the vision encoder, and the encoder cost scales linearly and unboundedly with frame count.

Comparison with Efficient Vision Encoders

Table 4 compares LiteFrame against state-of-the-art efficient vision encoders for MLLMs, all evaluated with InternVL3-8B's LLM fine-tuned via LoRA on the respective frozen visual encoders:

• FastVLM (125M parameters, 49 tokens/frame, 32 frames): 231.5 ms total latency (98.3 ms vision + 132.9 ms LLM), 47.6% average accuracy. The relatively high token count (49/frame) means the LLM still processes 1,568 tokens — enough for quadratic attention costs to be significant. The accuracy is substantially lower than LiteFrame's 58.0%.

• Video-Panda (45M parameters, 272 tokens/frame, 32 frames): 382.4 ms total latency (36.5 ms vision + 345.7 ms LLM), 49.2% average accuracy. The vision encoder is extremely fast (36.5 ms), but the dense, uncompressed token stream (8,704 tokens total) causes the LLM latency to dominate, illustrating the "shifting the bottleneck back to the LLM" problem the paper identifies in Section 2.

• LiteFrame (87M parameters, 16 tokens/frame, 32 frames): 91.9 ms total latency (30.1 ms vision + 61.5 ms LLM), 58.0% average accuracy. LiteFrame simultaneously achieves the lowest vision latency, the lowest LLM latency, and the highest accuracy — a result of its internalized compression solving both bottlenecks simultaneously.

The latency gap is stark: LiteFrame is 1.2× faster than Video-Panda and 3.3× faster than FastVLM, despite having more vision parameters than Video-Panda (87M vs. 45M). The speed advantage comes entirely from the drastically lower LLM cost (61.5 ms vs. 345.7 ms for Video-Panda, 132.9 ms for FastVLM), which in turn comes from the 16× token compression.

Comparison with AutoGaze

Figure 7 provides a detailed comparison with AutoGaze, the most directly comparable recent work. The left panel shows the Pareto frontier: LiteFrame (red) achieves higher accuracy at lower latency compared to AutoGaze (green) across the tested frame ranges (16–512 frames for LiteFrame, 16–512 frames for AutoGaze).

The right panel provides a detailed latency breakdown at comparable operating points (32 frames baseline, 256 frames scaled):

• NVILA-8B-Video (ImageViT, 32 frames): 780.8 ms total (451.0 ms ViT + 329.9 ms LLM), 63.1% accuracy.
• NVILA-8B-Video + AutoGaze (256 frames): 6,106.5 ms total (2,961.4 ms AutoGaze module + 2,605.6 ms VideoViT + 539.5 ms LLM), 63.1% accuracy. Despite processing 8× more frames, AutoGaze achieves exactly the same accuracy as the 32-frame baseline — the additional frames provide no benefit, likely because the aggressive token filtering loses critical information. The latency increase is staggering: +682.1% total, with the AutoGaze pre-reduction module alone accounting for 48.5% of total inference time (2,961.4 ms out of 6,106.5 ms).
• InternVL3-8B (32 frames): 814.5 ms total (144.5 ms ViT + 669.8 ms LLM), 65.3% accuracy.
• InternVL3-8B + LiteFrame (256 frames): 532.3 ms total (204.0 ms vision + 327.4 ms LLM), 65.7% accuracy. LiteFrame achieves +0.4 pp accuracy improvement with -34.6% latency reduction — a strict improvement in both dimensions. The vision encoder latency increase from 32 to 256 frames is only +41.2% (144.5 → 204.0 ms), demonstrating the effectiveness of progressive downsampling at controlling encoder cost scaling.

The comparison highlights what the paper argues is a fundamental architectural flaw in pre-filtering approaches: adding a module before the encoder shifts the bottleneck to that module rather than eliminating it. AutoGaze's autoregressive gazing module, which sequentially decides which regions to attend to, introduces latency that scales with the number of gazing steps — and at 256 frames, this cost becomes prohibitive.

Zero-Shot Spatial Resolution Scaling

Figure 6 presents results on HLVid, a benchmark requiring high-fidelity spatial understanding. The key finding is that LiteFrame's token efficiency enables spatial resolution scaling that prior methods cannot match:

• InternVL3-8B shows a characteristic pattern: accuracy initially improves with resolution (29.9% at 1 frame 448px → 45.9% at 16 frames 448px → 50.4% at 64 frames 448px), but then stagnates or degrades at higher resolutions as the model is forced to trade temporal resolution for spatial resolution due to the LLM's fixed context length. At 3,584px, accuracy drops to 42.9% with only 64 frames — the model can see fine spatial detail but loses the temporal context needed to understand actions and events.

• AutoGaze achieves 52.6% at 3,584px with 1,024 frames — better than InternVL3-8B at high resolution, but requiring massive frame counts and the highest resolution.

• LiteFrame achieves 54.1% at 2,688px with only 48 frames — the highest score overall, using lower resolution and far fewer frames than AutoGaze. This result is particularly significant because LiteFrame achieves it with zero-shot spatial scaling — the model was never trained on high-resolution inputs; the tiling strategy is applied purely at inference time. The token efficiency (16 tokens per 448px tile) means that processing a higher-resolution image as multiple tiles still leaves room in the LLM's context for sufficient temporal resolution (48 frames).

The practical implication: LiteFrame's architecture naturally decouples spatial and temporal token costs. Increasing spatial resolution does not force a proportional decrease in temporal resolution, because each spatial tile produces only 16 tokens. This decoupling is an emergent property of the internalized compression — the encoder was trained for spatio-temporal compression, but the same architecture works for spatial-only scaling when applied in a tiled manner.

Short and Additional Long Video Benchmarks

Appendix B reports results on benchmarks beyond the main four. For short video benchmarks (Figure 8):

• MVBench: LiteFrame reduces end-to-end latency by 28% while maintaining accuracy comparable to InternVL3-8B across the 1–16 frame range for the baseline and 4–64 frame range for LiteFrame. The accuracy-latency curves largely overlap, with LiteFrame shifted to lower latencies.
• TVBench: LiteFrame reduces latency by 63%, again with comparable accuracy. The larger latency reduction on TVBench (63% vs. 28%) suggests this benchmark is particularly bottlenecked by LLM computation, which LiteFrame's token compression addresses more aggressively than the vision encoder bottleneck.

For additional long video benchmarks (Figure 9):

• LVBench: LiteFrame at 512 frames achieves 43.9% accuracy compared to InternVL3-8B at 64 frames (43.5%) — a +0.4 pp improvement with 38% faster total latency. This is noteworthy because it shows LiteFrame can leverage the extended temporal context on a benchmark designed for extremely long videos (LVBench includes videos up to several hours), rather than merely matching the baseline's accuracy at lower latency.
• MMBench-Video: LiteFrame shows improved efficiency within the low-latency regime (16–128 frames), with the accuracy-latency curve shifted leftward compared to InternVL3-8B (1–16 frames).

Ablation Studies and Robustness Checks

The paper conducts a thorough ablation analysis in Table 5, isolating the contributions of four key design elements: the token-compressive student architecture, depth-wise temporal convolutions, the WAP objective for distillation, and Language Model Adaptation. All ablations in Table 5 use a fixed frame count of 128 and report both the specific benchmark breakdowns (Video-MME w/o and w/ subtitles, MLVU, LongVideoBench) and the average. Additional ablations in Tables 6 and 7 and Section C further expand specific dimensions.

Distillation into a standard ViT-Base-12L backbone without token compression (row: "Distillation (ViT-Base-12L)"): This baseline tests whether simple model distillation — replacing the 304M teacher with an 87M ViT-Base-12L that outputs the same 256 tokens per frame — provides any benefit. At 16 frames, this configuration achieves 362.9 ms total latency (only marginally better than the teacher's 403.6 ms) and 60.3% average accuracy (degraded from 62.2%). The minimal latency improvement despite using a model with 3.5× fewer parameters demonstrates that parameter count reduction alone does not solve the bottleneck — the token count must also be reduced, because the LLM's quadratic attention over 4,096 visual tokens (16 frames × 256 tokens) dominates the cost regardless of how cheaply those tokens were produced. The accuracy degradation (60.3% vs. 62.2%) likely reflects the student's reduced capacity to represent fine visual details when forced to output at the same resolution as the teacher — 768-dimensional features vs. 1024-dimensional, with fewer layers, cannot capture the same level of detail.

Token-compressive architecture with full spatio-temporal attention (row: "CTD (SpatioTempAttn)"): This ablation tests whether the progressive downsampling architecture works with a different temporal modeling mechanism. Using full spatio-temporal attention instead of DW temporal convolutions, with the same CTD objective and no LMA, this variant achieves 102.2 ms latency and 61.9% average accuracy at 128 frames. Compared to the final CTD (DWTempConv) variant (87.4 ms, 62.8%), this reveals that DW temporal convolutions provide both a latency advantage (-14.5%) and a small accuracy advantage (+0.9 pp). The latency advantage is expected from Table 2 (174.84 ms vs. 204.35 ms at 256 frames for pure encoder measurement). The accuracy advantage is more surprising — the paper attributes it to the inductive bias of separating spatial and temporal processing, which may make the optimization landscape easier for the student to navigate during distillation. The 61.9% accuracy is still a dramatic improvement over the "Distillation (ViT-Base-12L)" baseline (60.3% at only 16 frames), confirming that the token-compressive architecture is primarily responsible for the efficiency gains.

Depth-wise temporal convolutions as the temporal modeling mechanism (row: "CTD (DWTempConv)"): This is the CTD-only variant with the full LiteFrame architecture but no LMA. At 128 frames, it achieves 87.4 ms total latency and 62.8% average accuracy. The improvement over SpatioTempAttn (61.9% → 62.8%) is modest but consistent. The real significance is in combination with the latency advantage: DWTempConv achieves better accuracy with lower latency, making it a strict improvement on both dimensions. The per-benchmark breakdown shows the gains are broadly distributed: Video-MME w/o subs (59.7% → 61.9%), Video-MME w/ subs (62.1% → 65.6%), MLVU (65.1% → 65.8%), LongVideoBench (53.9% → 57.7%). The largest jump is in Video-MME with subtitles (+3.5 pp), suggesting that the improved temporal modeling particularly helps when the model must integrate visual information with text overlays that change frame-to-frame.

Reconstructive Token Distillation (RTD) as an alternative objective (row: "RTD"): This is a critical ablation that tests the paper's central claim about the importance of the WAP supervision target. RTD replaces the WAP objective with an autoencoding objective — the student's compressed features are decoded through lightweight auxiliary transformer blocks to reconstruct the teacher's full dense feature map. At 128 frames with the same architecture (DWTempConv, token-compressive), RTD achieves only 43.8% average accuracy — a catastrophic 19.0 pp drop from CTD's 62.8%. The per-benchmark breakdown is devastating: Video-MME w/o subs plummets from 61.9% to 31.7%, Video-MME w/ subs from 65.6% to 35.5%, MLVU from 65.8% to 64.3%, LongVideoBench from 57.7% to 43.8%. MLVU is notably resilient (only -1.5 pp), while Video-MME collapses by ~30 pp.

This ablation is perhaps the paper's most important single result. It demonstrates that how you construct the distillation target matters more than the student architecture or the distillation loss itself. Both CTD and RTD use the same student architecture, the same MSE loss, and the same training data. The only difference is what the student is trained to predict: WAP-compressed teacher features (CTD) vs. full teacher features through an autoencoder bottleneck (RTD). The nearly 20 pp gap shows that the WAP operator — with its attention-weighted aggregation that filters for semantically relevant information — provides a dramatically more effective learning signal than a generic reconstruction objective that must preserve all visual information equally. The autoencoder is forced to allocate its limited capacity to reconstructing redundant textures, background details, and noise alongside the semantically meaningful content, while WAP naturally filters out the redundancy before the student ever sees it.

RTD followed by LMA (row: "RTD + LMA"): Adding Language Model Adaptation partially recovers RTD's catastrophic failure — accuracy jumps from 43.8% to 61.5%, a +17.7 pp improvement. This shows that LMA can compensate for a poor initial representation to a significant degree, but it still falls short of CTD without LMA (61.5% vs. 62.8%) and substantially trails CTD + LMA (61.5% vs. 63.4%). The gap between RTD+LMA and CTD+LMA (1.9 pp) demonstrates that the choice of distillation objective has lasting effects that downstream fine-tuning cannot fully erase. A better initial representation (CTD) provides a stronger foundation that LMA can then refine, while a weaker initial representation (RTD) can be improved by LMA but never fully catches up.

CTD with Language Model Adaptation (row: "CTD + LMA (Ours)"): The full method achieves 87.4 ms latency and 63.4% average accuracy at 128 frames. The gain from adding LMA to CTD is +0.6 pp (62.8% → 63.4%), with per-benchmark gains concentrated in Video-MME w/o subs (61.9% → 63.1%, +1.2 pp) and LongVideoBench (57.7% → 58.6%, +0.9 pp), while Video-MME w/ subs and MLVU see smaller improvements. The modest magnitude of the LMA gain (+0.6 pp) relative to the CTD gain over baselines (+19 pp over RTD, +2.5 pp over Distill No Comp. at matched frames) indicates that the representation learned during CTD is already well-aligned with the LLM's needs, and LMA provides fine-tuning rather than fundamental adaptation. This is a positive result — it means the distillation framework produces features that are immediately useful for downstream language tasks, and the additional adaptation cost is minimal.

Spatial-only vs. spatio-temporal compression (Table 7, Section C.3): This ablation compares the proposed spatio-temporal compression (4× spatial, 4× temporal) against a spatial-only 16× compression variant. The spatial-only variant uses internal pooling solely along spatial dimensions, without temporal convolution layers (i.e., it processes frames as independent images), and is trained to predict the teacher's features under spatial 16× WAP. Across three frame budgets (64, 128, 256 frames):

• At 64 frames: Spatial 16× achieves 59.0% vs. 60.0% for spatio-temporal CTD — a -1.0 pp gap. The spatial-only variant has slightly lower latency (147.4 ms vs. 150.1 ms) because it avoids the temporal convolution cost, but the accuracy gap favors spatio-temporal.
• At 128 frames: Spatial 16× achieves 60.5% vs. 62.8% for spatio-temporal — a -2.3 pp gap that is widening. The latency gap also widens (264.8 ms vs. 272.6 ms), likely because the spatial-only variant must process more independent frames without temporal downsampling, increasing the number of forward passes through the full network.
• At 256 frames: Spatial 16× achieves 60.8% vs. 64.1% for spatio-temporal — a -3.3 pp gap. Critically, the spatial-only variant's accuracy actually degrades slightly from 128 to 256 frames (60.5% → 60.8%, essentially flat), while spatio-temporal continues to scale (62.8% → 64.1%). The latency gap further widens (504.4 ms vs. 532.3 ms).

The paper's interpretation is that spatial 16× compression "destructively pools a frame into a 4×4 grid," losing too much spatial fidelity for tasks like Video-MME that require fine-grained object recognition and spatial reasoning. The spatio-temporal variant distributes compression across both dimensions, maintaining a higher spatial resolution (16×16 grid vs. 4×4) while exploiting temporal redundancy through progressive downsampling. The widening accuracy gap with frame count suggests that temporal redundancy becomes more exploitable as more frames are available — with 256 frames, adjacent frames are very similar, and temporal compression can aggressively reduce redundancy without losing information, while spatial compression is equally destructive regardless of frame count.

Distillation without compression (Table 7, "Distill (No Comp.)"): This baseline tests whether the token-compressive architecture is necessary or whether a standard ViT-Base-12L can be distilled to match the teacher's accuracy when allowed to process the same number of frames. At 16 frames (the teacher's standard operating point), the uncompressed student achieves 60.3% accuracy at 362.9 ms — worse than the teacher's 62.2% at 403.6 ms. The student is faster but less accurate, trading one for the other rather than improving both. At 128 frames (where CTD processes 128 frames to the uncompressed student's 16), the gap widens dramatically: CTD achieves 62.8% vs. uncompressed at 60.3% — and CTD does this at lower latency (272.6 ms vs. 362.9 ms). This confirms that the token compression is the enabling factor: it allows more frames to be processed (128 vs. 16) within a lower latency budget, and the additional temporal context more than compensates for the spatial information lost through compression.

Reconstructive Token Distillation at varying frame counts (Table 6): This ablation in Section C.1 extends the RTD comparison across frame budgets. RTD achieves 43.6% at 64 frames, 43.8% at 128 frames, and 43.9% at 256 frames — essentially flat, showing that RTD cannot leverage additional temporal context. RTD + LMA achieves 59.4% at 64 frames, 61.5% at 128 frames, and 63.1% at 256 frames — it does scale with frames after LMA, but remains below CTD without LMA at every budget (60.0%, 62.8%, 64.1%). CTD + LMA achieves 61.0%, 63.4%, 65.3% — consistently the best. The key pattern: the gap between RTD+LMA and CTD+LMA narrows at higher frame counts (1.6 pp at 64f, 1.9 pp at 128f, 2.2 pp at 256f), suggesting that RTD's weakness is not simply a matter of needing more frames to compensate — the representation quality gap persists and may even widen slightly as temporal context increases.

Critical Assessment

Claim 1: LiteFrame provides a 35% reduction in end-to-end latency while processing 8× more frames and improving accuracy.

This claim is supported, but the 35% figure is an upper bound from one specific comparison point, not a universal property. The paper's Abstract and Introduction prominently feature "35% reduction in end-to-end latency while processing 8× more frames." In Table 3, the largest latency reduction is 34.6% at the 256-frame LiteFrame vs. 32-frame teacher comparison (532.3 ms vs. 814.5 ms). At lower frame counts, the reductions are 28.0% (64f vs. 8f, 150.1 ms vs. 208.4 ms) and 32.5% (128f vs. 16f, 272.6 ms vs. 403.6 ms). The 35% figure is a reasonable round-up of the 34.6% observation but is not consistently achieved across all operating points.

More importantly, the "8× more frames" comparison is uneven — it compares LiteFrame at high frame counts (64, 128, 256) against the teacher at low frame counts (8, 16, 32) that produce comparable total latency. This is a valid efficiency comparison (what you get for a fixed latency budget), but it conflates two effects: the encoder's efficiency and the benefits of temporal scaling. The accuracy improvement (+2.1 pp at the 64f/8f comparison, +1.5 pp at 128f/16f, +0.4 pp at 256f/32f) comes from both the encoder producing useful features and from the additional temporal context. This is the intended effect — the paper argues these are inseparable because the token compression enables the temporal scaling — but a reader parsing "LiteFrame improves accuracy" should understand that the improvement is relative to a baseline processing far fewer frames, not relative to an identical temporal context.

The +0.4 pp improvement at the highest budget (65.7% vs. 65.3%) is notably small — well within what could be statistical noise on a test suite of 500 questions per benchmark (or less). The paper does not report confidence intervals, making it impossible to assess whether this +0.4 pp is statistically significant. At the 128f/16f comparison (+1.5 pp), the signal is stronger but would benefit from error bars.

Claim 2: LiteFrame redefines the latency-accuracy Pareto frontier.

This claim is strongly supported by Figure 2 and Table 3. The evidence is comprehensive: the paper compares against three post-hoc methods (ToMe, PruMerge, FastVID), standard pooling baselines, two efficient encoder methods (FastVLM, Video-Panda), and AutoGaze, consistently showing LiteFrame in the low-latency, high-accuracy corner that no other method reaches. The comparison against post-hoc methods includes multiple compression ratios and frame counts. The Pareto frontier visualization in Figure 2 shows LiteFrame points (red stars) clearly separated from all other methods (colored dashed lines).

A caveat: the post-hoc methods are all evaluated with 16× compression. It's possible that some post-hoc method with a different compression ratio (e.g., 8× spatial only, or a different temporal subsampling scheme) could achieve a different point on the frontier. The paper sweeps 4×, 8×, and 16× WAP in Figure 3 (right) and finds 16× is best under a fixed token budget, but does not sweep compression ratios for ToMe, PruMerge, or FastVID. This is a minor gap — the paper's argument that post-hoc methods are fundamentally limited by the encoder latency floor would hold regardless of compression ratio, because the encoder cost is invariant to post-hoc compression.

Claim 3: Compressed Token Distillation (CTD) is essential for achieving the strong performance of LiteFrame, and the WAP target construction outperforms alternative objectives.

This claim is strongly supported by the RTD ablation, which is the most convincing individual experiment in the paper. The 19 pp gap between CTD (62.8%) and RTD (43.8%) at 128 frames without LMA (Table 5) is large enough to be robust to any reasonable estimate of statistical noise. The fact that RTD + LMA partially recovers (61.5%) but still trails CTD without LMA (62.8%) and CTD + LMA (63.4%) demonstrates that the WAP target provides a fundamentally better training signal.

However, the paper's interpretation of why WAP works better — that it filters for semantically relevant information via the class token's attention scores — is plausible but not directly tested. An ablation that used standard average pooling (without attention weights) as the CTD target would help distinguish whether the benefit comes from attention-based weighting specifically, or from any compression that reduces the token count while preserving structural regularity. The paper has this data indirectly: WAP (62.0% as a post-hoc method in Table 1) outperforms average pooling (60.2%), and CTD (which distills WAP) outperforms RTD (which reconstructs full features). But the chain of inference has a gap: does CTD with average pooling as the target achieve intermediate performance? This ablation is missing.

Additionally, the RTD result has an alternative interpretation: the autoencoding objective may simply be harder to optimize than direct prediction, regardless of the target representation. The decoder adds parameters and a reconstruction loss that may compete with the compression objective, leading to suboptimal convergence. The paper does not report whether RTD was given commensurate training budget (it uses 800 epochs for ablations vs. 1,800 for full CTD, Appendix A) or whether hyperparameters were tuned for RTD specifically. It's possible that RTD with more careful optimization could narrow the gap, though the 19 pp margin makes it unlikely that optimization alone explains the entire difference.

Claim 4: Internalizing token compression within the encoder architecture eliminates the encoder latency bottleneck that limits post-hoc methods.

This claim is supported, but the evidence is more nuanced than the headline suggests. Table 3 shows that LiteFrame's vision encoding latency at 64 frames (54.8 ms) is indeed much lower than the encoder latency under post-hoc methods at comparable total latency (161.7 ms for FastVID at 32 frames, which is actually fewer frames). Figure 5 shows that LiteFrame (CTD-only, frozen LLM) achieves better accuracy-latency trade-offs than all post-hoc methods.

However, the claim that the bottleneck is "eliminated" rather than "reduced" deserves scrutiny. LiteFrame's vision encoding latency still scales with frame count: 54.8 ms at 64 frames → 105.3 ms at 128 frames → 204.0 ms at 256 frames. This is approximately linear (1.92× per 2× frames), the same scaling pattern as the teacher (40.0 → 74.0 → 144.5 ms, ~1.85× per 2× frames). The difference is in the constant factor: LiteFrame's per-frame cost is dramatically lower because its encoder is architecturally lighter (87M vs. 304M parameters, 768D vs. 1024D, 12 layers vs. 24) and because progressive downsampling reduces the cost of deeper layers. But the vision encoder still scales linearly with frame count — it has not been "eliminated" as a scaling concern, merely reduced by a constant factor. If one wanted to process 1,024 frames, LiteFrame's vision encoding would take approximately 800 ms, at which point the encoder is once again the bottleneck.

This is not a weakness of the paper per se — eliminating linear scaling would require a sublinear-cost encoder architecture, which may not be possible for dense video processing. But the framing "resolves both inefficiencies" (Figure 1c caption) overstates the case. LiteFrame reduces the encoder bottleneck substantially, enough to push the Pareto frontier forward, but it does not fundamentally change the scaling law — it improves the constant factor.

Claim 5: LiteFrame enables zero-shot spatial resolution scaling that surpasses prior state-of-the-art.

This claim is supported by Figure 6 but is based on a single benchmark (HLVid) and a single comparison point. The 54.1% at 2,688px with 48 frames vs. AutoGaze's 52.6% at 3,584px with 1,024 frames is an impressive result, particularly given that LiteFrame achieves it without any high-resolution training. However, the comparison is not entirely fair: AutoGaze is evaluated with its optimal hyperparameters (task_loss_requirement_tile = 0.6, specific gazing ratios) on the NVILA-8B-Video backbone, while LiteFrame uses InternVL3-8B. The base models differ, so the +1.5 pp gap may partly reflect InternVL3-8B being a stronger base model than NVILA-8B-Video for this task, independent of the encoder design.

More critically, the zero-shot claim is supported by the fact that LiteFrame was not trained on high-resolution data, but the paper does not report whether InternVL3-8B was — if InternVL3-8B was trained with some higher-resolution examples during its original pretraining, the comparison is less "zero-shot" than claimed. The paper also does not explore what happens if tiling is applied to the teacher encoder directly (InternVL3-8B with spatial tiling at high resolution, sacrificing frames). The teacher's stagnation at high resolution (Figure 6, blue line) suggests this would not work well because the teacher's 256 tokens per tile would quickly exhaust the LLM's context, but an explicit comparison would strengthen the claim.

General Experimental Strengths

• Comprehensive baseline coverage. The paper compares against post-hoc methods (ToMe, PruMerge, FastVID), pooling baselines, efficient encoder designs (FastVLM, Video-Panda), and pre-filtering (AutoGaze) — essentially every major category of approach to the Video LLM efficiency problem. This makes the Pareto frontier claim robust.

• Multiple operating points. Rather than reporting a single accuracy-latency point, the paper evaluates at three latency tiers (~200 ms, ~400 ms, ~800 ms) and across frame counts, demonstrating that the advantage holds across a range of compute budgets.

• Well-designed ablations. The ablation sequence in Table 5 cleanly isolates each design choice: token compression (Distill No Comp. vs. CTD), temporal modeling mechanism (SpatioTempAttn vs. DWTempConv), distillation objective (RTD vs. CTD), and adaptation (with/without LMA). Each row changes exactly one variable from the full method or the previous ablation, making the contribution of each component unambiguous.

• Transparent latency measurement. The paper specifies the measurement protocol (median of 100 after 40 warmup, single A100-80GB), what is included (ViT + LLM prefilling, excluding autoregressive decoding), and reports all latency figures consistently across methods. This level of detail is essential for reproducibility and is often missing from efficiency papers.

General Experimental Weaknesses

• No error bars or confidence intervals. All results report point estimates (e.g., 65.7% accuracy, 532.3 ms latency) without any measure of variance. Accuracy is a statistic computed over a test set of 500 questions (Video-MME), with smaller splits for MLVU and LongVideoBench. A ±1 pp confidence interval could change whether the +0.4 pp improvement at 256 frames is meaningful or noise. Latency is measured as a median, which is robust to outliers, but the variance across videos (some may have more complex content requiring different compute paths) is not reported.

• Single GPU, single precision setup. All latency measurements are on a single NVIDIA A100-80GB. Results may differ on different hardware (H100 with flash attention, edge devices, TPUs) or with different precision (FP16 vs. INT8). The paper does not discuss how the relative efficiency gains might change with hardware or quantization.

• Limited exploration of compression ratio trade-offs. The paper fixes 16× compression (4× spatial, 4× temporal) as the design point based on Figure 3 (right), but does not explore whether a different ratio (e.g., 8× spatial, 2× temporal) would better balance the spatial fidelity vs. temporal context trade-off for specific tasks. The spatial-only 16× ablation (Table 7) suggests spatial compression alone is insufficient, but what about 8× spatial + 2× temporal?

• No real-time or streaming video evaluation. All benchmarks use pre-segmented video clips. The paper does not evaluate LiteFrame in a streaming setting where frames arrive sequentially and the model must process them online. This is relevant because LiteFrame's progressive downsampling operates on batches of frames (clips of 4), which may introduce latency in a streaming context. The clip-based sampling strategy (4-frame clips at 1 FPS minimum) may also behave differently with live video than with pre-recorded benchmarks.

• The student architecture is fixed at ViT-Base scale. The paper notes in Appendix E that "efforts to scale down to even smaller student models were constrained by training instabilities, such as loss explosions," but does not explore this further. It's unclear whether the CTD framework breaks down below a certain student capacity threshold, or whether the instabilities are solvable with better hyperparameters or training techniques. This limits the generalizability of the approach — if 87M parameters is the practical minimum, then LiteFrame cannot target extremely resource-constrained deployments (e.g., on-device video understanding).

• LMA dataset overlap with evaluation. The LMA stage uses LLaVA-Video-178K and FineVideo, which may have some overlap with the training data of InternVL3-8B's original LLM. The paper does not discuss potential data contamination between the LMA training set and the evaluation benchmarks. If some Video-MME or MLVU examples (or near-duplicates) appear in LLaVA-Video-178K, the LMA gains could partly reflect memorization rather than genuine adaptation. This is a standard concern in LLM evaluation that the paper should at minimum acknowledge, even if the risk is low given the scale of the training data.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Renders the Headline 4× Efficiency Gain Partially Theoretical

The paper's compute-optimal test-time scaling framework conditions strategy selection on estimated prompt difficulty. The difficulty estimation method — generating 2048 samples per question and scoring them with the process reward model — is extraordinarily expensive relative to the inference budgets being optimized. The authors acknowledge this directly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence is that the headline claim of 4× efficiency improvement over best-of-N (Figures 4 and 8) is computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, total compute would be difficulty estimation cost + strategy execution cost, and the former — generating and scoring 2048 samples — could dominate the latter. For a prompt where the optimal strategy uses 64 generations, the difficulty estimation alone costs 32× more than the actual problem-solving budget. The paper's predicted difficulty variant (using PRM scores instead of ground-truth correctness) does not reduce the sampling cost — it still requires 2048 samples — it only removes the need for ground-truth labels.

The paper provides extensive evidence that the compute-optimal curves with predicted difficulty largely overlap with oracle difficulty curves (Figures 4, 8), demonstrating that the PRM score distribution is a sufficient proxy for ground-truth difficulty. However, the paper never reports the total FLOPs or wall-clock time including difficulty estimation. All efficiency comparisons (e.g., "4× less test-time compute") exclude the estimation overhead.

Mitigation status: The paper flags this as "a key avenue for future work" (Section 3.2) and suggests pretraining or fine-tuning models to predict difficulty directly from the question text, but no such model is developed or evaluated. The adaptive difficulty estimation approach — using a small initial sample of generations to estimate difficulty before allocating the remaining budget — is mentioned conceptually but not implemented. Until this gap is closed, the 4× figure should be understood as an upper bound on achievable efficiency in a deployment where difficulty is unknown, not a realized gain.

---

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The paper's entire empirical analysis — search strategies, revision model behavior, difficulty-dependent effects, compute-optimal allocation, and FLOPs-matched pretraining comparisons — is conducted on exactly one dataset (MATH benchmark, 500 test questions) with one base model family (PaLM 2-S*). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is untested.

The consequence is that we cannot know which findings are general properties of test-time compute scaling and which are artifacts of MATH's specific characteristics or PaLM 2-S*'s specific output distribution. Several aspects are particularly vulnerable to domain shift. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s calibration properties and error patterns — a model with different token distributions might produce Monte Carlo rollouts with different reliability, changing the PRM's training quality and thus the entire difficulty-dependent scaling pattern. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capacity, which varies substantially across model families. MATH consists of competition-level math problems requiring symbolic multi-step reasoning; it is unclear whether the central finding — that beam search helps on medium problems but over-optimizes on easy ones, that sequential revisions help on easy problems — transfers to code generation, logical reasoning, scientific QA, or any domain requiring factual knowledge rather than pure inference.

The difficulty quintile analysis, which underlies the entire compute-optimal framework, splits 500 test questions into bins of ~100 each. With two-fold cross-validation, strategy selection is based on ~50 questions per fold per bin — a sample size where the variance of the selected optimal strategy could be high. The paper reports no confidence intervals on the compute-optimal scaling curves (Figures 4, 8), nor on the per-difficulty-bin accuracy numbers (Figure 3 right, Figure 7 right). The critical finding that beam search degrades easy-problem performance at high budgets (Figure 3 right, bin 1) — the primary evidence for verifier over-optimization — is based on ~100 questions in bin 1, split across folds, evaluated at multiple budget levels. Without confidence intervals, we cannot assess whether the observed non-monotonicity (accuracy declining from ~78% to ~77% as budget increases) is a real effect or sampling noise.

Mitigation status: The authors acknowledge the limitation implicitly by calling their model "representative" rather than making universal claims, but do not discuss cross-domain or cross-model generalization as a limitation. No replication is attempted on other benchmarks (GSM8K, MMLU reasoning subsets) or other model families (Gemini, LLaMA, Qwen).

---

The 14× Larger Model Baseline in the FLOPs-Matched Comparison Is Not Compute-Optimally Trained

The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, explicitly following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal paradigm (Hoffmann et al., 2022) where both parameters and training data are scaled equally with total FLOPs. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence is that the pretraining baseline is likely weaker than it should be for a fair FLOPs-matched comparison. A Chinchilla-optimal model trained with 14× more total FLOPs would scale both parameters and data — achieving better performance per FLOP than a parameter-only-scaled model, which may be undertrained relative to its parameter count. This means the reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on medium questions at R ≪ 1 (Figure 9) — may shrink or reverse against a properly compute-optimally trained larger model.

Furthermore, the 14× larger model is evaluated only with greedy decoding — no majority voting, no best-of-N, no search, no revisions. This creates an asymmetric comparison: the smaller model receives the full benefit of sophisticated test-time compute strategies, while the larger model receives none. A fairer comparison would give the larger model a modest test-time compute budget (e.g., best-of-8 or a short revision chain), which would still cost less in absolute FLOPs than the smaller model's larger test-time budget, and would test whether test-time compute provides additional gains beyond what a larger model can also achieve with basic inference-time strategies.

The paper's finding that test-time compute cannot help on the hardest problems (difficulty bin 5, near-zero accuracy regardless of budget) is convincing regardless of the baseline weakness, but the crossover points — where test-time compute becomes preferable to pretraining — depend directly on the strength of the pretraining baseline. With a stronger baseline, the R value at which pretraining becomes preferable would shift lower, reducing the regime where the paper's recommendation (test-time compute over pretraining) applies.

Mitigation status: The limitation is acknowledged but not addressed. The authors explicitly defer the Chinchilla-optimal pretraining comparison to future work. The asymmetry of inference strategies between the two models (greedy vs. compute-optimal) is not discussed as a limitation.

---

Verifier Over-Optimization Is a Hard Ceiling That the Compute-Optimal Policy Mitigates but Does Not Solve

The paper identifies verifier over-optimization as the central limiting factor preventing unbounded improvements from test-time compute, and documents it extensively: beam search degrades easy-problem performance at high budgets (Figure 3 right, bin 1), lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3 left), and qualitative examples show degenerate outputs scoring highly under the PRM (Appendix M). The compute-optimal policy mitigates this by routing easy problems to best-of-N rather than beam search, but it does not fundamentally solve the underlying verifier robustness problem.

The consequence is that performance under compute-optimal scaling is fundamentally bounded by verifier quality at all difficulty levels. On easy problems, best-of-N weighted scaling continues to improve (Figure 3 right, bin 1: ~68% to ~88% from 4 to 256 generations), but this is effectively an upper bound — more aggressive search would push into the over-optimization regime. On medium problems where the policy deploys beam search, the beam search curve itself flattens (Figure 3 right, bin 3: beam search plateaus at ~34% by 256 generations). The compute-optimal policy selects the best available strategy, but all available strategies hit ceilings determined by the PRM's reliability. Improving the PRM — through better training data, adversarial robustness, calibration, or ensemble methods — would likely shift these ceilings upward and change which strategies are optimal at each difficulty level.

The paper's results are therefore specific to the verifier quality achievable with the Monte Carlo rollout training procedure described in Appendix D. If a practitioner trains a PRM with a different procedure and achieves different quality, the entire difficulty-dependent strategy map could shift — easy problems might tolerate beam search, medium problems might benefit from lookahead, and the compute-optimal policy would look different. The paper does not study how the difficulty-dependent scaling behavior changes as a function of verifier quality, so we cannot predict how much better a stronger verifier would make the system.

Mitigation status: The paper identifies PRM over-optimization as a key finding and discusses it in Sections 5.3 and 8, but treats it as an empirical observation rather than something to be systematically varied. Future work on robust verifiers is suggested but not explored.

---

The Revision Model Has a Fundamental Correct-to-Incorrect Reversion Problem with No Architectural Solution

Section 6.1 reports that approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step. This is a direct consequence of the training data construction: the revision model is trained only on sequences where all in-context answers are incorrect, followed by a correct target. It therefore has no training signal for what to do when the current answer is already correct — it learns to always change the answer, regardless of whether change is needed.

The consequence is that extending the revision chain does not monotonically improve performance; there is a tension between the benefits of additional refinement opportunities and the risk of corrupting previously correct answers. The paper mitigates this at inference time by applying within-chain selection (majority voting or verifier-based selection) across all revisions, picking the best answer from any point in the chain rather than always taking the final revision. But this is a patch, not a solution: it means the model wastes computation generating revisions that are worse than what it already produced, and it relies on the selection mechanism (which itself is imperfect) to identify the correct answer in the chain.

The severity of this problem is underscored by the ReST^EM experiment (Appendix K, Figure 16): attempting to optimize the revision model with on-policy RL-style training caused sequential revision performance to substantially degrade, with fully sequential accuracy dropping to ~33.5% at 256 generations compared to ~38.5% at the optimal ratio. This suggests that the revision training procedure is fragile — the positive results depend on specific design choices (offline data construction, edit-distance-based incorrect-correct pairing) that may amplify desirable behaviors in the initial training but break under iterative optimization.

Mitigation status: The paper identifies the 38% reversion rate as a "significant practical issue" (Section 6.1) and mitigates it with within-chain selection. A more principled solution — such as training the model to recognize when no revision is needed, or including "no change needed" examples in the training data — is not explored. The ReST^EM failure is reported as a negative result but not deeply analyzed.

---

Difficulty Bins Are Static, Coarse, and Require Uniform Policy Within Each Bin

The compute-optimal framework partitions the test set into five difficulty quintiles using the base model's pass@1 rate, then selects a single strategy (search algorithm, beam width, sequential-to-parallel ratio) for all questions within each bin at each budget level. This coarse discretization has several consequences.

First, there is substantial heterogeneity within a bin: a question at the easy end of bin 3 and one at the hard end of bin 3 might have meaningfully different optimal strategies, but they receive identical treatment. The paper does not report the distribution of accuracies within each bin — only the bin average — so we cannot assess how much performance is left on the table by the within-bin strategy uniformity.

Second, the difficulty bins are computed once per test set and treated as static. There is no mechanism for dynamic difficulty reassessment mid-computation: the system commits to a strategy based on its pre-computed difficulty estimate and does not adapt if initial generations suggest the problem is easier or harder than estimated. For example, if a question is placed in bin 3 (predicted medium) but the first few beam search steps produce very low PRM scores, a dynamic policy might switch to best-of-N or allocate more budget to parallel exploration. This inability to adapt online leaves efficiency gains unexploited — the exploration-exploitation tradeoff the paper flags in Section 3.2 is acknowledged as important but not addressed.

Third, the five-bin discretization means there are only five possible strategies at any budget level. A continuous difficulty estimate with a smooth policy function — or even a finer binning (10 or 20 bins) — could potentially achieve better allocation, but would require a larger test set to reliably estimate per-bin optimal strategies. The paper does not explore sensitivity to the number of bins, so we don't know whether 5 is near-optimal or whether 10 bins would yield meaningful further gains.

Mitigation status: The paper acknowledges that difficulty estimation via 2048 samples is expensive (Section 3.2) but does not discuss the coarseness of the binning as a limitation, nor the potential benefits of dynamic or continuous policies. These are left entirely to future work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reshapes the conversation around Video LLM efficiency by introducing a diagnostic reframing rather than an incremental compression technique. The dominant paradigm in the field — post-hoc token reduction — rests on an assumption so deeply embedded it was rarely stated explicitly: the LLM's quadratic self-attention over visual tokens is the bottleneck, and fixing it fixes the system. The entire subfield of efficient Video LLMs organized around this premise, producing increasingly sophisticated "extract-and-reduce" methods (ToMe, PruMerge, FastVID, HoliTom, VisionZip) that treated the vision encoder as an immutable black box.

The paper's "scaling paradox" — articulated in Section 3 and quantified in Figure 1(b) and Table 3 — reveals that this premise is structurally incomplete. When post-hoc compression is aggressive enough to genuinely alleviate the LLM bottleneck (16× token reduction), the vision encoder's per-frame cost, which scales linearly with frame count, becomes the dominant latency term. This is not a minor correction to existing methods. It establishes a hard latency floor that no amount of post-hoc cleverness can penetrate: you cannot avoid paying the full ViT forward pass for every frame you want to process, regardless of how aggressively you compress its output.

The significance of this reframing extends beyond the specific LiteFrame architecture. It changes how the field should think about the Video LLM architecture stack — not as a sequence of independent, separately-optimizable components (encoder, projector, LLM), but as a coupled system where optimizing one component predictably shifts the bottleneck to another. This echoes Amdahl's Law from computer architecture: the maximum speedup achievable by optimizing one stage is bounded by the fraction of total time that stage consumes. Post-hoc methods drove the LLM's fraction toward zero, exposing the encoder as the new limiter. The field had been optimizing the wrong target, or rather, optimizing a target that was correct at one operating point but became wrong as that operating point shifted.

The paper also resolves a latent tension in the efficient encoder literature. Prior work on lightweight visual backbones for MLLMs fell into two camps, each solving one problem while resurrecting the other. Image-centric encoders like FastVLM reduced per-frame cost but ignored temporal redundancy, so frame scaling still incurred linear cost growth. Encoder-free paradigms like Video-Panda eliminated the vision backbone but exposed the LLM to dense, uncompressed token streams, shifting the bottleneck right back. Pre-filtering methods like AutoGaze attempted to reduce encoder input but introduced auxiliary modules whose overhead consumed the theoretical savings. LiteFrame demonstrates that the resolution is not to choose between these approaches but to internalize the compression within the encoder itself — making the forward pass be the compression, so there is no separate stage to optimize, no auxiliary module to add overhead, and no dense intermediate to pay for.

The paper's finding that a simple, structurally regular compression operator (WAP, achieving 62.0% average accuracy at 16× compression in Table 1) outperforms complex, content-adaptive methods (ToMe at 60.0%, PruMerge at 59.7%, FastVID at 60.6%) has broader implications for visual tokenizer design. The vision community has invested heavily in sophisticated compression — attention-based pruning, bipartite matching, dynamic density estimation — under the implicit assumption that maximizing information preservation per token is the right objective. The paper's results suggest that for language model consumers, the format of the information matters as much as the quantity. A regular grid of slightly suboptimal tokens preserves the spatial layout that LLMs rely on for reasoning about object locations, relationships, and motion trajectories, while content-adaptive merging produces an irregular token set that, however information-dense, disrupts this structure. This is a methodological principle — test downstream task performance, not just reconstruction fidelity — that should inform future tokenizer design.

The RTD ablation (Table 5: 43.8% vs. 62.8% for CTD at 128 frames) makes this point with unusual force: how you construct the distillation target matters more than the student architecture or the loss function. The WAP operator, by weighting tokens according to the class token's attention scores, naturally filters for semantically relevant information before the student ever sees it — a form of "learned semantic compression" transferred from the teacher's pretraining. The autoencoding objective (RTD), which must preserve all visual information to reconstruct the full feature map, forces the student to waste capacity on redundant textures, backgrounds, and noise. The 19-percentage-point gap is large enough to be robust to any reasonable estimate of statistical noise, establishing a clear principle: distillation for downstream tasks should preserve semantic saliency, not visual fidelity.

The paper's position on the broader efficiency landscape is carefully bounded. It does not claim to eliminate the linear scaling of encoder cost with frame count — Table 3 shows LiteFrame's vision latency scales approximately linearly (1.92× per 2× frames, closely matching the teacher's 1.85×). What it improves is the constant factor: the encoder is architecturally lighter (87M vs. 304M parameters), processes compressed tokens in deeper layers, and produces 16× fewer output tokens. For long-form video understanding at modest frame counts (64–512), this constant-factor improvement is sufficient to push the Pareto frontier well beyond what post-hoc methods can reach. For extreme-scale deployment (thousands of frames), the linear scaling would eventually re-emerge as a bottleneck, but the paper provides the conceptual framework — internalized compression — that would be needed to address it.

Finally, the paper establishes a new baseline for what "efficient video understanding" means. Prior work typically evaluated efficiency gains in relative terms (X% faster than a baseline at the same frame count) without controlling for the accuracy implications. LiteFrame's simultaneous latency reduction and accuracy improvement — 35% faster with +0.4 to +2.1 pp accuracy gains depending on the operating point — sets a higher bar. A method that improves latency by 20% but degrades accuracy (FastVID at 32 frames: +7.9% latency, +0.5 pp in Table 3; at 64 frames: +0.6% latency, -2.7 pp) is no longer acceptable when a method exists that improves both dimensions simultaneously. This raises the standard for what constitutes a genuine efficiency contribution in the Video LLM space.

Follow-Up Research This Work Enables

Scaling CTD to smaller student architectures. The paper notes in Appendix E that "efforts to scale down to even smaller student models were constrained by training instabilities, such as loss explosions." This is a critical open problem: can the CTD framework distill a 304M-parameter teacher into a ViT-Tiny (5–10M parameters) or even a fully convolutional architecture? The specific failure mode — loss explosion — suggests the current MSE-based distillation may become unstable when the student's capacity is too small to represent the teacher's compressed features with sufficient fidelity. A productive follow-up would systematically vary student capacity (ViT-Tiny, ViT-Small, ViT-Base, plus convolutional alternatives like MobileNet-v5 variants) while applying stabilization techniques the paper already uses (outlier clipping, gradient clipping, reduced learning rate for sensitive variants). The research question is whether there exists a minimum capacity threshold below which CTD breaks down, and whether that threshold can be lowered through techniques like progressive distillation (intermediate student of intermediate size), attention transfer in addition to feature matching, or adversarial training to improve feature robustness. The practical payoff is substantial: a sub-50M-parameter video encoder with LiteFrame's token compression would enable on-device long-form video understanding on mobile hardware.

Online difficulty estimation and dynamic strategy switching. The paper's compute-optimal framework (in the original work on test-time compute scaling, now inherited by LiteFrame's approach) relies on static difficulty estimation via 2048-sample generation — a cost that can dominate the inference budget. LiteFrame's internalized compression changes this calculus because the per-sample cost is dramatically lower (54.8 ms for 64 frames vs. the teacher's 40.0 ms for just 8 frames). A concrete follow-up would implement adaptive difficulty estimation: generate a small initial set of samples (4–8) using LiteFrame, compute the PRM's average final-answer score or another lightweight confidence metric, and use this to allocate the remaining token budget dynamically — switching between frame counts, compression ratios, or even encoder architectures (LiteFrame vs. the full teacher for genuinely hard clips). The experiment would measure the trade-off between estimation cost and allocation accuracy across a range of initial sample sizes, comparing the adaptive strategy against (a) the static difficulty bin approach from the paper, (b) a uniform allocation strategy, and (c) an oracle that knows ground-truth difficulty. Given LiteFrame's 8× frame budget advantage (Table 3), even a modest number of estimation samples could provide useful difficulty signals without bankrupting the total compute budget.

Combining LiteFrame with post-hoc token reduction for extreme compression. The paper demonstrates that internalized compression (16× via progressive downsampling) outperforms post-hoc compression (16× WAP applied to teacher features), but it does not explore whether further post-hoc compression applied to LiteFrame's already-compressed output provides additional gains. LiteFrame produces 16 tokens per frame; applying an additional 4× WAP would yield 4 tokens per frame, enabling 256-frame processing with only 1,024 total visual tokens — well within the comfort zone of even small LLMs. The experiment would compare: (a) LiteFrame at 64 frames with 16 tokens/frame (1,024 tokens total), (b) LiteFrame + 4× WAP at 256 frames with 4 tokens/frame (1,024 tokens total), (c) LiteFrame at 256 frames with 16 tokens/frame (4,096 tokens total), and (d) the teacher + 16× WAP at 64 frames (1,024 tokens total). The research question is whether LiteFrame's features, which are already optimized for compression, tolerate additional post-hoc compression better than the teacher's full-resolution features, and whether the extended temporal context from 256 frames compensates for the more aggressive per-frame compression. The paper's RTD result (43.8% at 128 frames) suggests that information-preserving compression has limits, but LiteFrame's features may be more compressible than raw teacher features because they have already been filtered for semantic saliency.

Extending CTD to multi-teacher and multi-task distillation for domain-specific video understanding. The paper trains CTD on a diverse mix of general video datasets (ShareGPT4Video, LLaVA-Video-178K, FineVideo, CLEVRER, NTURGB+D), producing a general-purpose video encoder. A natural extension is domain-specific distillation: train separate LiteFrame variants where the teacher's WAP targets are computed with domain-specific attention — for example, a teacher fine-tuned on surgical video understanding would have class tokens that attend to surgical instruments and anatomical structures, while a teacher fine-tuned on sports analysis would attend to players and ball trajectories. CTD would then transfer these domain-specific attention patterns into the student, producing a family of efficient encoders each specialized for a different domain while sharing the same lightweight architecture. The experiment would measure per-domain accuracy and cross-domain generalization, testing whether domain-specific distillation provides benefits beyond what can be achieved by domain-specific LMA on a general encoder. The paper's HLVid result (54.1% zero-shot at 2,688px, Figure 6) suggests LiteFrame has strong generalization, but domain-specific variants might push performance further for high-value applications like medical video analysis or autonomous driving perception.

Stress-testing the WAP primitive against learned compression operators at scale. The paper establishes WAP as a strong baseline (62.0% vs. 60.6% for FastVID at 16× compression, Table 1) and uses it as the CTD target, but WAP's attention weights come from the teacher's class token, which was trained for ImageNet-style classification — not for video understanding or language model alignment. A follow-up could train a learned compression operator specifically optimized for downstream LLM performance: use a lightweight network (perhaps a small transformer or MLP) that takes the teacher's dense features as input and predicts compression weights, trained end-to-end with the LLM's cross-entropy loss via straight-through gradient estimation or reinforcement learning. The comparison would be: (a) CTD with WAP targets (current best), (b) CTD with learned-weight targets (trained jointly with a frozen LLM), and (c) CTD with learned-weight targets trained end-to-end with LMA. The research question is whether WAP's simple class-token attention is near-optimal for this task, or whether a purpose-trained compression operator could extract significantly better features by optimizing directly for downstream performance rather than mimicking a classification-trained attention pattern. The paper's finding that WAP already outperforms complex post-hoc methods suggests the gains from learned compression may be modest, but the experiment is necessary to establish an upper bound.

Cross-model and cross-domain replication of the internalized compression framework. The paper's entire evaluation is on InternVL3-8B as the base model and Video-MME/MLVU/LongVideoBench as the benchmarks. A rigorous follow-up would replicate the core findings with different base models and domains: (a) apply CTD to distill Qwen2.5-VL's or LLaVA-OneVision's vision encoder into a LiteFrame-style student, measuring whether the latency-accuracy Pareto frontier improvement generalizes across model families; (b) evaluate on code generation from video (e.g., video-to-code benchmarks), embodied AI tasks (video of robot manipulation → action prediction), and long-form video summarization, testing whether the frame-count bottleneck insight — that aggressive compression + more frames outperforms mild compression + fewer frames — holds across task types; and (c) evaluate on a real-time video streaming task where frames arrive sequentially and the encoder cannot batch them, measuring whether LiteFrame's clip-based sampling strategy (4-frame clips at 1 FPS minimum) introduces unacceptable latency in a streaming context, or whether the architecture can be adapted for online processing. The paper's strong results on short video benchmarks (MVBench: 28% faster, TVBench: 63% faster, Figure 8) and additional long benchmarks (LVBench: 38% faster, Figure 9) provide preliminary evidence of cross-task generalization, but a systematic cross-model replication would establish the robustness of the internalized compression paradigm.

Practical Applications and Downstream Use Cases

Cost-efficient batch video processing for data annotation and curation. Organizations that process large video corpora — content moderation platforms, video search engines, training data curation pipelines for video generation models — currently face a stark trade-off: use heavy Video LLMs for high-quality understanding at prohibitive cost per video-hour, or use lightweight heuristics that miss nuanced content. LiteFrame's 35% latency reduction while processing 8× more frames (Table 3: InternVL3-8B at 32 frames/814.5 ms vs. LiteFrame at 256 frames/532.3 ms, with higher accuracy) means these pipelines can process both more videos (lower per-video cost) and more frames per video (better temporal coverage) simultaneously. For a pipeline processing 100,000 video-hours, moving from the teacher baseline to LiteFrame would reduce total GPU-hours by roughly 35% while improving the quality of annotations, tags, or summaries — a direct cost saving combined with a quality improvement, the rare case where the efficiency-accuracy trade-off is not a trade-off at all.

On-device video understanding for accessibility and real-time assistance. Applications like real-time sign language translation, navigation assistance for visually impaired users, or wearable camera systems for memory augmentation require continuous video processing on battery-constrained devices. Current approaches either stream video to cloud-based LLMs (introducing latency and privacy concerns) or use lightweight models that sacrifice accuracy. LiteFrame's 87M-parameter encoder, producing only 16 tokens per frame, brings the vision encoding cost within range of mobile GPUs and NPUs. At 64 frames, vision encoding takes 54.8 ms on an A100 — mobile hardware would be slower, but the 16× token compression means the downstream LLM (which could be a smaller on-device language model like Gemma-3n-1B or a quantized LLM) would process only 1,024 tokens for 64 frames, making the full pipeline potentially viable at interactive frame rates. The paper's zero-shot spatial scaling result (54.1% on HLVid at 2,688px, Figure 6) further suggests that tiled high-resolution processing, which is important for reading text or recognizing small objects in assistive applications, works without retraining.

Long-form video question answering in enterprise and education. Use cases like searching within recorded lectures, corporate training videos, or legal depositions require models that can process hour-long videos while answering specific questions about events, statements, or visual details. Current Video LLMs typically cap at 64 frames (the InternVL3-8B baseline in Table 3), which for a one-hour video means sampling roughly one frame per minute — missing any event shorter than a minute. LiteFrame at 512 frames (LVBench result: 43.9% at 512 frames vs. 43.5% for the baseline at 64 frames, Figure 9) can sample roughly one frame every 7 seconds, dramatically improving coverage of short-duration events. The 38% latency reduction means this improved temporal coverage comes at lower computational cost than the baseline's sparse sampling, making it practical for deployment in interactive QA systems where users expect sub-second response times. The paper's strong performance on MLVU (70.7% at 256 frames, Table 3), a benchmark specifically designed for multi-task long video understanding, suggests that the extended temporal context genuinely improves understanding across diverse question types — not just simple event detection but also reasoning about plot, character motivation, and causal relationships.

When to Prefer This Method

The paper does not explicitly position LiteFrame as a method to be selected over named alternatives based on a decision rule. Rather, it demonstrates that LiteFrame produces a new Pareto frontier that dominates existing approaches — it is simultaneously faster and more accurate than post-hoc compression, efficient encoder designs, and pre-filtering methods at all tested operating points. The implication is that for any Video LLM deployment using InternVL3-8B or similar architectures, replacing the vision encoder with LiteFrame (trained via CTD and adapted via LMA) is a strict improvement — there is no regime in the paper's results where the teacher encoder or a post-hoc method outperforms LiteFrame at equal or lower latency. The paper's additional benchmarks (MVBench, TVBench, LVBench, MMBench-Video, HLVid) support this conclusion across short-form, long-form, and high-resolution tasks.

The practical decision facing a practitioner is therefore not "when to prefer LiteFrame over method X" but rather "is the CTD + LMA training pipeline feasible in my setting." The answer depends on whether the practitioner has access to (a) a strong teacher encoder for the target domain (the paper uses InternViT-300M, but any large ViT could serve), (b) sufficient video data for CTD (the paper uses a diverse mix of public datasets, totaling likely millions of video clips), (c) GPU resources for 1,800 epochs of distillation (21 days on 8× H100 GPUs for the full model, or 800 epochs for ablations), and (d) video-text pairs for LMA (LLaVA-Video-178K, FineVideo, ShareGPT4Video, OpenVid-1M). For organizations that already deploy Video LLMs and have in-house training infrastructure, the pipeline is replicable. For resource-constrained settings, the upfront training cost is significant but needs to be paid only once — the resulting 87M-parameter encoder can then be deployed across all inference workloads. The paper does not release pretrained weights at the time of writing (the project page is provided but weights are not discussed), which would be the primary enabler of widespread adoption.