Dynamic Reflections: Probing Video Representations with Text Alignment

🎯 Pitch

Video-text alignment can be boosted by ~60% simply by giving models richer test-time inputs—multiple frames and captions—without retraining, revealing an inescapable scaling law. Remarkably, how well a video encoder aligns with text in a zero-shot manner strongly predicts its downstream performance on both action recognition and non-semantic tasks like depth estimation and camera pose tracking.

1. Executive Summary

This work conducts the first comprehensive study extending the Platonic Representation Hypothesis into the temporal domain by probing video-text representation alignment across 121 modern video and language encoders. The central contribution is the discovery that cross-modal alignment quality depends strongly on the richness of both visual and textual information provided at test time—specifically, using multiple video frames rather than single images, and multiple diverse captions rather than a single annotation—which the paper captures through parametric test-time scaling laws (a saturation-based model of the form score(n_f, n_c) = S_∞ − (C_f n_f^{−α} + C_c n_c^{−β}) that achieves R² > 0.98). The authors demonstrate that these test-time data augmentations can improve alignment scores by approximately 60% (going from 1 to 10 captions) and that native video models like VideoMAEv2 leverage additional frames roughly three times more effectively than static image encoders like DINOv2, as measured by the fitted frame coefficient C_f. Through a FLOPs-matched-style analysis, the paper further shows that video-text alignment strongly correlates with performance on both semantic downstream tasks (action classification, Pearson R = 0.883–0.923) and non-semantic perception tasks (depth estimation, camera pose, object tracking), establishing that alignment against text encoders can serve as an informative zero-shot proxy for general-purpose video representation quality—though this predictive power weakens on highly local tasks such as point tracking.

2. Context and Motivation

The foundational question this paper tackles is whether the Platonic Representation Hypothesis (PRH)—the idea that neural networks trained at scale converge toward a shared statistical model of reality in their latent spaces—holds for dynamic, temporal data. Prior work by Huh et al. (2024) established this hypothesis and gathered substantial empirical support for it, but exclusively in the domain of static modalities: images and text. The authors demonstrated that powerful, independently trained vision encoders (like DINOv2) and language encoders develop internal representations with structural similarities that increase with model scale, even without explicit cross-modal training. Subsequent studies built on this finding, showing that these aligned latent spaces can be connected via simple linear transformations (Maniparambil et al., 2024b; Merullo et al., 2022) and exploited for unsupervised translation between modalities (Jha et al., 2025; Schnaus et al., 2025).

However, the PRH was stated in full generality—as a claim about how any sufficiently powerful models trained on any modalities should converge. The paper's central motivating gap is that no prior work has tested whether this convergence manifests in the temporal domain, where the information structure is fundamentally different. Videos contain motion, causality, temporal dependencies, and dynamic interactions that static images cannot capture (Gu et al., 2018; Wang et al., 2023). This gap matters for two reasons:

Theoretical: If the PRH genuinely reflects a universal property of representation learning, it must hold for temporal data. Failure to extend would reveal boundary conditions that constrain the hypothesis. Success would strengthen the claim that all modalities are "projections of the same physical world" and that scale-driven convergence is modality-agnostic.
Practical: Video understanding is increasingly central to AI applications (embodied agents, autonomous systems, content analysis), yet evaluating self-supervised video representations currently requires expensive task-specific training on downstream benchmarks (Wang et al., 2023; Carreira et al., 2024; Hasson et al., 2025). If video-text alignment correlates with downstream performance—as has been shown for image-text alignment (Maniparambil et al., 2024a)—then alignment scores could serve as a zero-shot proxy metric for video representation quality, dramatically reducing the cost of model development and evaluation.

Why Prior Static-Modality Findings Left an Open Question

The paper explicitly identifies a critical limitation of prior static-modality studies. Huh et al. (2024) reported alignment scores (measured via mutual k-NN) that reached approximately 0.16–0.18 between the best image and text encoders, and acknowledged:

"the maximum theoretical value for the alignment metric is 1. Is a score of 0.16 indicative of strong alignment [...] or does it signify poor alignment with major differences left to explain? We leave this as an open question."

This ambiguity—whether the observed alignment represented convergence approaching a fundamental ceiling or merely the tip of a much larger iceberg—remained unresolved. The paper identifies a specific mechanism that prior work overlooked: the poverty of test-time data. Previous studies used single static images and single text descriptions to compute alignment. If richer inputs (multiple frames capturing dynamics, multiple captions capturing diverse perspectives) could substantially boost alignment without any model retraining, then the low scores observed in prior work might reflect insufficient test-time information rather than fundamental representational divergence. This would reframe the open question: alignment isn't inherently limited to ~0.18; it's limited by how we probe it.

The paper's approach to answering this question is complementary to the training-time resource scaling (model size, dataset size) considered in prior work. Rather than asking "does alignment improve if we train bigger models?", they ask "does alignment improve if we provide richer data at inference time?" This is a distinct axis of investigation with direct practical implications: if test-time data richness matters, then evaluation protocols and deployment strategies should account for it.

Where Existing Approaches Fall Short

The paper identifies several specific weaknesses in the current landscape:

1. Evaluation of self-supervised video models is expensive and narrow. As the authors note:

"evaluating self-supervised video representations is challenging, and current approaches rely on expensive task-specific training"

This means that to assess whether a new video encoder produces good features, practitioners must fine-tune the frozen encoder on each downstream task (action classification, depth estimation, object tracking, etc.) separately, requiring significant compute and time. This bottleneck slows the development cycle for video models. A zero-shot evaluation metric with strong predictive power for downstream performance would be transformative, analogous to how linear probing on ImageNet became a standard quick-check for image representations. The paper positions video-text alignment as a candidate for such a metric, but emphasizes that this requires first establishing that alignment does correlate with downstream performance for video models—a connection that had never been tested.

2. No systematic comparison exists between video and image encoders on text alignment. Prior work established that image encoders like DINOv2 align well with text (Maniparambil et al., 2024b), and separately, that video encoders like VideoMAEv2 achieve strong performance on action recognition tasks (Wang et al., 2023). But these findings existed in silos. A natural question—can a self-supervised video encoder trained purely on reconstruction objectives (with zero text supervision) achieve competitive or better text alignment than the best image encoders?—had no answer. Answering it matters because if video encoders do align well with text, it provides evidence that temporal dynamics contribute to semantic grounding, even when text is never seen during training. If they don't, it reveals a limitation of current video pre-training paradigms.

3. Existing video-text bridging work doesn't probe intrinsic representation properties. The authors acknowledge related efforts that connect video and text representations (Kim et al., 2023; Liu et al., 2024a; Li et al., 2025), but note that these approaches:

"typically rely on an existing source of alignment rather than investigating the intrinsic properties of video encoders"

In other words, prior work uses contrastive training, adapter modules, or cross-modal attention to create alignment between video and text. This paper asks a different question: does alignment emerge spontaneously from the geometry of representations learned by independently trained unimodal models? This distinction is crucial for the PRH: the hypothesis is about convergence arising from scale and data diversity alone, not from explicit cross-modal objectives. Probing intrinsic alignment requires studying models like VideoMAEv2 that were never exposed to text during pre-training.

4. Temporal reasoning capabilities of video-language models are poorly understood. While multimodal video-text models exist (e.g., VideoCLIP, VideoPrism), their ability to encode temporal ordering—distinguishing "A appears before B" from "B appears before A"—has not been systematically probed through the lens of representational alignment. Standard benchmarks often test semantic content understanding (what objects, what actions) without isolating temporal structure. The paper identifies this as a challenging test-bed where current models may struggle, providing diagnostic value for future model development.

How the Paper Positions Itself

The paper positions itself at the intersection of two research trajectories and as filling the gap between them:

From the PRH literature, it inherits the methodology (mutual k-NN metric), the conceptual framework (emergent alignment as a signature of convergence), and the baseline comparisons (DINOv2 vs. T5, Llama, etc., establishing that their image-text alignment scores match prior reported values before extending to video). The paper explicitly frames itself as:

"extend[ing] previous cross-modal alignment studies into the temporal domain by conducting the first comprehensive investigation of video-text representation similarity"

From the self-supervised video learning literature, it inherits the models (VideoMAE/VideoMAEv2, V-JEPA, Timesformer, ViViT) and the motivation of finding better ways to evaluate these models. The paper positions alignment probing as a complement to—not replacement for—downstream fine-tuning evaluations, offering a quick, zero-shot signal that can guide model development before committing to expensive full evaluations.

The paper's novel synthesis is the combination of test-time data scaling with alignment measurement. Prior PRH work varied training-time resources and measured alignment on fixed test data. Prior video work used fixed inference protocols (typically 16 frames, single clip). By systematically varying the number of frames and captions at test time, the paper introduces a new degree of freedom into the analysis and discovers that it matters enormously. This leads to the parametric scaling laws (Eq. 2), which formalize the dependency and provide predictive power—a contribution that has no direct precedent in the alignment literature.

The paper's temporal analysis (Section 8) further positions it as a bridge to the video-language understanding community. By testing how alignment scores change under temporal perturbations (reordered captions in VideoComp, logical reversals in Test of Time), the authors probe whether alignment reflects genuine temporal reasoning or merely bag-of-words semantic matching. This extends the diagnostic value of alignment beyond a simple scalar metric to a tool for dissecting what kind of information different encoders capture.

Finally, the paper frames its contributions as opening rather than closing the investigation. It repeatedly identifies limitations and future directions: the need for cheaper difficulty estimation (analogous to the compute-optimal test-time scaling literature), the open question of combining search with revisions, and the ultimate goal of jointly optimizing pretraining and inference strategies. This positions the work not as a definitive answer but as a foundational framework that enables a new class of questions about how temporal dynamics interact with representational convergence.

3. Technical Approach

3.1 Reader Orientation

This is primarily an empirical measurement and analysis paper whose core idea is that the quality of cross-modal alignment between independently trained video and language encoders is not a fixed property of the models, but depends strongly on how much visual and textual data is provided at test time—and that this dependency can be precisely captured by a parametric scaling law.

The paper builds an evaluation framework that takes a dataset of video-caption pairs, processes them through frozen pre-trained encoders under varying test-time data budgets (different numbers of video frames and text captions), and measures the structural similarity of the resulting embedding spaces using a mutual nearest-neighbor metric. The central output is a set of alignment scores that can be used both to compare encoder quality and to predict downstream task performance without any task-specific training.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, illustrated in Figure 1:

Video-Caption Dataset (V, C): A test set of $N = 1024$ paired videos and caption sets. Each video $v_i$ comes with a set $c_i = \{c_{i1}, c_{i2}, \ldots\}$ of text descriptions. The two primary datasets are VATEX (10 English captions per video from different annotators) and PVD (with captions synthesized by splitting a single detailed caption using Gemini-2.5 Pro).
Video Encoder (E_vid): A frozen pre-trained model that processes a video $v_i$ into an embedding vector $\mathbf{v}_i \in \mathbb{R}^p$ . The encoder accepts a parametrically variable number of frames $n_f$ at test time by sampling frames through uniform linear interpolation, splitting the video into sub-clips of the encoder's native length $n_0$ , and averaging the resulting sub-clip representations.
Text Encoder (E_text): A frozen pre-trained language model that processes a set of captions $c_i$ into an embedding vector $\mathbf{c}_i \in \mathbb{R}^q$ . Multiple captions are concatenated into a single string before encoding, and per-token embeddings are averaged across the token dimension to produce a single fixed-size vector per video.
Alignment Metric (Mutual k-NN): A function that takes two sets of embeddings $\mathbf{X} \in \mathbb{R}^{N \times p}$ and $\mathbf{Y} \in \mathbb{R}^{N \times q}$ (with potentially different dimensionalities $p$ and $q$ ) and returns a scalar score between 0 and 1 measuring how much the nearest-neighbor structures in the two spaces agree.
Test-Time Scaling Law Model: A parametric function $\text{score}(n_f, n_c) = S_\infty - (C_f n_f^{-\alpha} + C_c n_c^{-\beta})$ fit to the empirical alignment scores obtained by sweeping $n_f$ (frames) and $n_c$ (captions) across a grid of values. The fitted parameters characterize each encoder pair's asymptotic alignment potential and its efficiency at leveraging additional data.

Information flows as follows: a test set of 1024 video-caption pairs is constructed → for each video, $n_f$ frames are sampled and passed through the video encoder, while $n_c$ captions are concatenated and passed through the text encoder → the two resulting embedding matrices are fed into mutual k-NN to produce an alignment score → this process is repeated across a sweep of $(n_f, n_c)$ values → the scaling law is fit to the resulting score surface.

3.3 Roadmap for the Deep Dive

First, the dataset construction and processing, since all alignment measurements depend on the quality and structure of the video-caption pairs, and the paper's key insight (multi-frame, multi-caption scaling) is implemented at this stage.
Second, the video and text encoder pipeline, covering how models with different native architectures (image vs. video, decoder-only vs. encoder-decoder) are unified into a common embedding extraction protocol, and the critical design choice of which intermediate layers to use.
Third, the mutual k-NN alignment metric, including how it handles different embedding dimensionalities, the role of the hyperparameter $k$ , and layer selection across encoders.
Fourth, the multi-instance aggregation procedures—how multiple frames are sampled and averaged, and how multiple captions are concatenated—since these are the mechanisms that enable the test-time scaling study.
Fifth, the test-time scaling law formulation, including the specific functional form, the fitting procedure, the interpretation of each parameter, and the theoretical justification connecting the exponents to encoder-specific effective dimensionalities.
Sixth, the downstream correlation analysis and temporal probing experiments, which validate the alignment metric's practical utility and diagnostic power.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical analysis paper whose core idea is that cross-modal video-text alignment is a function of test-time data richness and can be captured by a saturation-based scaling law with strong predictive power. The paper is built entirely around the measurement of a single alignment metric under carefully controlled variations of input data, and the subsequent analysis of how that metric relates to model architectures, data scales, and downstream task performance.

Dataset Construction and Properties

VATEX (Primary Dataset). The main experimental results use VATEX (Wang et al., 2019), a large-scale video-and-language dataset. The authors construct a test set by randomly sampling $N = 1024$ videos from VATEX. Each video is approximately 10 seconds long and is sourced from a unique YouTube video. Critically, each video is accompanied by 10 English captions, each produced by a different human annotator. Each caption is typically a single sentence averaging around 15 words. This per-video collection of independently authored captions provides the "multi-caption" test-time scaling axis: the system can use anywhere from $n_c = 1$ to $n_c = 10$ captions when computing the text embedding for a video. The authors emphasize that different annotators describe different aspects of the same video, so increasing $n_c$ both expands the coverage of described visual concepts and increases the diversity of linguistic perspectives on those concepts.

PVD (Perception Encoder Video Datasets). To validate findings beyond VATEX, the authors also use PVD (Bolya et al., 2025), which does not naturally come with multiple diverse captions. Instead, PVD provides a single fine-grained, detailed caption per video. To create a multi-caption axis for PVD, the authors use Gemini-2.5 Pro to split each original caption into 10 separate, shorter captions. The prompt instructs the model to extract distinct subsets of details into individual captions, to avoid summarizing the same information repeatedly, and to not hallucinate new details. The prompt and a qualitative example are provided in Appendix A.3. The authors then treat these synthesized captions analogously to VATEX's human-written captions for the multi-caption scaling experiments.

Test set size rationale. The choice of $N = 1024$ is deliberate. The mutual k-NN metric (described below) uses a fixed $k = 10$ neighbors, and prior work (Huh et al., 2024) established that this ratio provides a stable estimate. With only 1024 samples, $k = 10$ represents roughly 1% of the dataset, meaning the metric is measuring fine-grained local neighborhood agreement without being overly sensitive to individual sample variation.

Video Encoding Pipeline

Frame sampling. The paper's key methodological innovation for video is the parametric control of $n_f$ , the number of frames used at test time. For a video encoder with a native clip length of $n_0$ frames (typically $n_0 = 16$ for models like VideoMAEv2), the procedure for handling arbitrary $n_f$ is:

If $n_f \leq n_0$ : extract $n_f$ frames through uniform linear interpolation across the video duration, producing a single clip of length $n_f$ . Encode this clip normally.
If $n_f > n_0$ : extract frames such that the total frame count is a multiple of $n_0$ (for example, $n_f \in \{16, 32, 64, 80\}$ when $n_0 = 16$ ). Split the extracted frames into sub-clips of length $n_0$ . Pass each sub-clip independently through the video encoder to produce a sub-clip representation. Average all sub-clip representations to produce the final video embedding.

This procedure means that increasing $n_f$ provides the encoder with more temporal coverage of the video (frames spaced further apart in time) without exceeding the encoder's native sequence length, which could cause architectural failures. The averaging over sub-clips is a simple but, as the results show, effective baseline for aggregating temporal information. The paper studies values of $n_f$ including 1, 2, 4, 8, 16, 32, 48, 64, and 80 frames, with $n_f = 1$ corresponding exactly to the image-only setup used in prior work.

Image models on video. To evaluate image encoders (like DINOv2 or CLIP) on video data, the authors use two strategies, distinguished by marker symbols in Figure 2:

Image model on image (■): Only a single frame (the first frame) is used. This replicates the static setup of prior work and serves as a baseline.
Image model on video (●): For a naive introduction of temporal dynamics, 8 frames are sampled uniformly from the video, each frame is encoded independently by the image model, and the resulting 8 frame embeddings are averaged along the temporal dimension. This is a frame-by-frame approach—the image model has no mechanism for modeling temporal relationships across frames, but the averaging provides a coarse multi-view signal.

Native video models (▲): These are encoders trained specifically on video data with spatio-temporal architectures (e.g., VideoMAE, VideoMAEv2, ViViT, Timesformer, V-JEPA 2). They process multi-frame clips natively, learning joint spatial and temporal features. The paper treats models like Perception Encoder, which were trained on video data, as video models when tested on video input.

Model diversity. In total, the paper evaluates 85 vision model variants spanning self-supervised methods (VideoMAE/VideoMAEv2, MAE, DINOv2, DINOv3, V-JEPA 2, WebSSL), text-aligned methods (CLIP, Perception Encoder variants), and hybrid approaches (AugReg, Perception Encoder with language training). The full list is in Appendix A.2. For each encoder, the paper stores intermediate representations from all layers, not just the final output layer, because prior work established that the optimal alignment often occurs at intermediate layers rather than the final embedding.

Embedding extraction. For each video, the encoder produces a tensor of shape [layers, hidden_dim] (for models with a class token, that token is used; otherwise, all spatial-temporal tokens are averaged). The result is a matrix $\mathbf{X} \in \mathbb{R}^{N \times p}$ where $p$ is the hidden dimension of the chosen layer for the vision encoder.

Text Encoding Pipeline

Language model selection. The paper evaluates 30 language model variants across 8 model families: T5 (encoder-decoder, sizes Small through 11B), BLOOM (decoder-only, 560M–7.1B), Llama (decoder-only, 7B–65B), OpenLlama (decoder-only, 3B–13B), Llama 3 (decoder-only, 1B–70B), Gemma (decoder-only, 2B and 7B), Gemma 2 (instruction-tuned decoder-only, 2B-it, 9B-it, 27B-it), and Gemma 3 (instruction-tuned decoder-only, 1B-it through 27B-it). The Gemma 2 and Gemma 3 models are highlighted as particularly strong text encoders despite being trained purely for text generation, consistently achieving the best alignment scores.

Caption aggregation. Given a set of $n_c$ captions for a video (where $n_c \in \{1, 2, 4, 8, 10\}$ ), the captions are concatenated into a single string before encoding. This means the text model receives a longer input sequence as $n_c$ increases. After encoding, the text encoder produces a sequence of per-token embeddings. These are averaged across the token dimension to produce a single fixed-size vector per video.

Layer selection. As with the vision encoder, the authors extract representations from all intermediate layers of the text encoder. For a text model with $L_t$ layers and hidden dimension $q$ , the result for each video is stored as a matrix of shape $[L_t, q]$ . The final text embedding matrix is $\mathbf{Y} \in \mathbb{R}^{N \times q}$ for a specific chosen layer.

Why concatenation and averaging over tokens? The paper does not provide alternatives for this design choice, but the rationale is clear: concatenation treats all captions as a continuous stream, allowing the language model's self-attention to relate concepts across captions. Averaging over tokens produces a fixed-dimensional vector regardless of input length, making the embedding comparable across different $n_c$ values. Alternatives like encoding each caption separately and averaging the resulting sentence embeddings would lose cross-caption interactions that may capture complementary information.

The Mutual k-NN Alignment Metric

This metric, introduced by Huh et al. (2024), measures the agreement between the nearest-neighbor structures of two embedding spaces. It is the paper's primary quantitative tool.

Construction. Given two embedding matrices $\mathbf{X} \in \mathbb{R}^{N \times p}$ and $\mathbf{Y} \in \mathbb{R}^{N \times q}$ representing the same $N$ items (videos) in two different embedding spaces (video and text), the procedure is:

Compute neighbor indicator matrices. For each row $i$ in $\mathbf{X}$ , find its $k$ nearest neighbors (by Euclidean distance, implied by the standard use of the metric). Construct a binary matrix $\mathbf{M}_X \in \{0,1\}^{N \times N}$ where $(\mathbf{M}_X)_{ij} = 1$ if item $j$ is among the $k$ nearest neighbors of item $i$ in space $\mathbf{X}$ , and 0 otherwise. Do the same for $\mathbf{Y}$ to produce $\mathbf{M}_Y$ .
Compute overlap. Take the element-wise (Hadamard) product $\mathbf{M}_X \odot \mathbf{M}_Y$ . The entry $(\mathbf{M}_X \odot \mathbf{M}_Y)_{ij}$ is 1 only if $j$ is a mutual neighbor of $i$ in both spaces.
Average across all pairs and normalize.

The final score is computed as:

$A_{\text{MkNN}}(\mathbf{X}, \mathbf{Y}) = \frac{1}{kN} \sum_{i=1}^{N} \sum_{j=1}^{N} (\mathbf{M}_X \odot \mathbf{M}_Y)_{ij}$

where $k = 10$ is the number of nearest neighbors considered (for a dataset of $N = 1024$ examples), $\mathbf{M}_X$ and $\mathbf{M}_Y$ are the binary neighbor indicator matrices for the vision and text embedding spaces respectively, $\odot$ denotes the Hadamard (element-wise) product, and $N$ is the number of items in the test set.

What it computes: For each of the $N$ data points, the metric counts how many of its $k$ nearest neighbors in the vision space are also among its $k$ nearest neighbors in the text space. It then averages this count across all points and normalizes by $k$ . The result is a scalar between 0 and 1, where 1 means perfect agreement (the $k$ nearest neighbors of every point are identical in both spaces) and 0 means no agreement beyond chance. Operationally, it captures whether the two embedding spaces "sort" the data in the same way—whether items that are close in one space tend to be close in the other.

Why this form: The mutual k-NN metric has several properties that make it appropriate for cross-modal alignment:

Dimensionality-agnostic: It operates only on neighbor ranks, not on absolute distances. Since vision embeddings live in $\mathbb{R}^p$ and text embeddings in $\mathbb{R}^q$ with typically different $p$ and $q$ , direct distance comparisons (e.g., canonical correlation analysis) would require alignment of the coordinate systems, which is precisely what we want to avoid assuming.
Local structure focus: By using $k = 10$ (roughly 1% of the dataset), the metric measures fine-grained local neighborhood agreement rather than global structure. This is consistent with the PRH's emphasis on local manifold structure—two models may organize concepts similarly at a local level even if their global embedding topologies differ.
Invariant to scaling and rotation: Since nearest-neighbor relationships are preserved under any affine transformation that preserves rank order, the metric doesn't penalize models that differ only by a linear transformation—precisely the type of difference that the PRH literature has shown can be bridged with a simple linear map (Merullo et al., 2022; Maniparambil et al., 2024b).

Layer pair optimization. A critical implementation detail is that the paper does not compare embeddings from fixed layers. Instead, for each pair of vision and text encoders, the authors optimize over the choice of intermediate layers:

"we also follow previous work and optimize over the choice of intermediate layers for both encoders, and pick the pair of layers that maximizes the alignment score"

This means: for a vision encoder with $L_v$ layers and a text encoder with $L_t$ layers, the alignment score is computed for all $L_v \times L_t$ possible layer combinations, and the maximum is reported. The rationale is that different encoders may represent semantic information at different depths, and using the wrong layers could artificially suppress alignment scores. This is a maximally permissive evaluation—it asks "how well can these two encoders ever align?" rather than "how well do they align at a fixed depth?"

The layer indices for the optimal pairing are reported alongside the scores (visible in the detailed matrices in Figures 11 and 12 in the Appendix).

Multi-Instance Aggregation Mechanisms

This is the technical core of the paper's novel contribution: how to encode a video with $n_f > 1$ frames and a set of $n_c > 1$ captions into single embedding vectors that can be fed into the mutual k-NN metric.

Multi-frame video encoding (visual detail). The paper's test-time frame scaling is implemented as follows:

For $n_f$ frames and a video encoder with native clip length $n_0 = 16$ :

Extract $n_f$ frame indices through uniform linear interpolation across the video's total frame count.
Nearest neighbor interpolation is used when $n_f$ is not a multiple of $n_0$ , to produce frame counts that are multiples of $n_0$ . For example, with $n_0 = 16$ , the paper studies $n_f \in \{16, 32, 64, 80\}$ .
Split the extracted frames into $n_f / n_0$ consecutive sub-clips, each of length $n_0$ .
Pass each sub-clip through the video encoder independently to produce $n_f / n_0$ embedding vectors.
Average these vectors (element-wise mean) to produce the final video embedding $\mathbf{v}_i \in \mathbb{R}^p$ .

This procedure is what makes the parametric sweep possible: the same encoder can process $n_f = 1$ frame (reducing to an image encoder), $n_f = 16$ frames (a single native clip), or $n_f = 80$ frames (5 sub-clips averaged). The paper's key finding—that native video models like VideoMAEv2 benefit more from additional frames than image models like DINOv2—is measured through this mechanism.

Why frame averaging? The paper uses simple averaging rather than learned temporal aggregation (e.g., a Transformer over sub-clip embeddings) for two likely reasons: first, it keeps the evaluation zero-shot (no additional training on the test data), and second, it isolates the effect of providing richer input to the encoder rather than learning a better aggregation. The fact that averaging works well enough to see significant improvements suggests the sub-clip embeddings are already well-aligned in a common space.

Multi-caption text encoding (textual detail). For $n_c$ captions (where $n_c \in \{1, 2, 4, 8, 10\}$ on VATEX):

Select $n_c$ captions from the available 10 per video. The paper uses the first $n_c$ captions (or a random subset for single-caption experiments), not a curated selection.
Concatenate the selected captions into a single string, typically with spaces or punctuation separating them.
Pass this concatenated string through the text encoder, which produces per-token embeddings.
Average the per-token embeddings along the token dimension to produce a single vector $\mathbf{c}_i \in \mathbb{R}^q$ .

The dimensionality $q$ is constant regardless of $n_c$ because token averaging collapses the sequence dimension. This is essential for the mutual k-NN metric, which requires fixed-size embeddings for all items.

Why concatenation and averaging? Concatenation allows the language model's self-attention mechanism to relate information across captions—for example, detecting that two captions describe the same object using different words, or noticing that one caption focuses on actions while another describes the scene background. Averaging over tokens produces a holistic "bag-of-concepts" representation that aggregates all the semantic content in the captions. The alternative—encoding each caption separately and averaging the resulting sentence embeddings—would prevent cross-caption attention and might miss complementary information that is only apparent when captions are considered together.

Synthesized captions for PVD (Appendix A.3). Since PVD provides only one long, detailed caption per video, the authors synthesize multiple shorter captions using Gemini-2.5 Pro with a carefully designed prompt. The prompt emphasizes:

Producing exactly 10 captions
Each caption containing a different subset of details from the original
Avoiding summarization (not all information in every caption)
Using alt-text format with short phrases, approximately 120 tokens each
Not hallucinating or inferring information not present in the original caption

A qualitative example is provided in Figure 9 of the Appendix, showing an original caption like "The person is adding mayonnaise and vegetables to the burger..." being split into captions like "Person making a burger," "Adding mayonnaise and vegetables to a burger," and "Burger assembled on a wooden board."

Layer Selection Protocol in Detail

The paper's layer optimization procedure is worth examining carefully because it affects all reported alignment scores:

Step 1: Full layer extraction. For every test video, the vision encoder's representations are extracted at every layer, yielding a tensor $[L_v, N, p]$ . Similarly, the text encoder produces $[L_t, N, q]$ . This is computationally expensive (storing all intermediate activations for 1024 videos across potentially 24–48 layers) but enables the subsequent optimization.

Step 2: Pairwise alignment computation. For each of the $L_v \times L_t$ possible (vision_layer, text_layer) combinations, the mutual k-NN score is computed between the corresponding $N \times p$ and $N \times q$ matrices.

Step 3: Maximum selection. The maximum score across all layer pairs is reported as the alignment score for that (vision_encoder, text_encoder) pair. The optimal layer indices are recorded.

Why this is important: This protocol means the reported alignment scores represent a best-case scenario for each model pair—they answer "what is the maximum achievable alignment between these two encoders, if we are allowed to choose which internal representations to compare?" This is consistent with the paper's goal of probing the intrinsic alignment potential of representations, rather than evaluating a specific fixed protocol. However, it also means that comparing scores across model pairs requires caution: a higher score for pair A over pair B might reflect genuinely better representations, or it might reflect that pair A happened to have layers that matched more cleanly.

Why not just use final layers? Prior work (Huh et al., 2024) established that the optimal alignment often occurs at intermediate layers—not the final output. This has a theoretical interpretation: early layers capture low-level features that are modality-specific, while the very deepest layers may be specialized for the pre-training objective (e.g., next-token prediction for LLMs, or reconstruction for masked autoencoders). The "Platonic" representation—the shared statistical model of reality—is hypothesized to reside in intermediate representations that balance generality with abstraction.

The Test-Time Scaling Law (Equation 2)

The paper's central parametric model is introduced and justified in Section 6 of the main paper, with extensive theoretical development in Appendix B. The formulation is:

$\text{score}(n_f, n_c) = S_\infty - \left(C_f n_f^{-\alpha} + C_c n_c^{-\beta}\right)$

where $n_f$ is the number of video frames provided at test time, $n_c$ is the number of text captions provided at test time, $S_\infty$ is the theoretical saturation alignment score achievable with infinite data (the asymptotic limit), $C_f$ is the error coefficient for the visual modality (capturing how much alignment error is attributable to insufficient frames), $C_c$ is the error coefficient for the textual modality, $\alpha$ is the scaling exponent for frames (how quickly additional frames reduce the visual error component), and $\beta$ is the scaling exponent for captions (how quickly additional captions reduce the textual error component).

What it computes: Given a specific vision-text encoder pair, this function predicts the expected mutual k-NN alignment score from the numbers of frames and captions used at test time. When $n_f$ and $n_c$ are large, the power-law terms $C_f n_f^{-\alpha}$ and $C_c n_c^{-\beta}$ approach zero, and the score approaches $S_\infty$ . When either $n_f$ or $n_c$ is small, the corresponding term is large, and the score is reduced. The two error components are additive: the total alignment gap is the sum of the gap due to limited visual data and the gap due to limited textual data.

Why this form: The authors test multiple formulations but find this saturation-based model provides "the best fit by a significant margin." The theoretical justification (Appendix B) proceeds through several layers:

Saturation form: The alignment score is bounded (between 0 and 1) and increasing with data, so it must asymptote. The form $S_\infty - \text{error}$ captures this: performance equals maximum potential minus a reducible gap. This is structurally analogous to the Chinchilla scaling laws (Hoffmann et al., 2022) which model loss as $L(N, D) = E + A/N^a + B/D^b$ , adapted for a metric where higher is better.
Power-law error decay: The $n^{-\alpha}$ form is ubiquitous in neural scaling laws (Kaplan et al., 2020; Bahri et al., 2021) and arises from the statistical properties of natural data. In Appendix B, the authors provide two theoretical derivations: one based on manifold approximation theory (estimation variance scales as $n^{-2/d}$ where $d$ is the effective dimensionality of the data manifold as perceived by the encoder), and one based on Zipfian feature importance distributions (the probability of capturing the next-most-important feature decays as a power law).
Additive error components: The total error is the sum of vision-specific and text-specific components because the noise processes in the two modalities are assumed independent. In the manifold approximation framework, $\text{E}[\lVert \mathbf{V}(n_f) - \mathbf{T}(n_c) \rVert^2] = \text{E}[\lVert \eta_V(n_f) \rVert^2] + \text{E}[\lVert \eta_T(n_c) \rVert^2]$ where $\eta_V$ and $\eta_T$ are zero-mean noise vectors from finite sampling, and the cross-term vanishes under the independence assumption.

Fitted parameter values. For the Gemma 2-9b-it text encoder:

VideoMAEv2 Huge: $S_\infty \approx 0.41$ , $C_f = 0.15$ , $C_c = 0.13$ , $\alpha = 0.75$ , $\beta = 1.30$ , with $R^2 = 0.9791$ .
DINOv2 Giant: $S_\infty \approx 0.37$ , $C_f = 0.05$ , $C_c = 0.13$ , $\alpha = 1.76$ , $\beta = 1.40$ , with $R^2 = 0.9964$ .

The most informative parameter comparison is $C_f$ : VideoMAEv2's value (0.15) is roughly triple DINOv2's (0.05). This means that at low frame counts, the video model has much more room to improve from additional frames—it is penalized more heavily for insufficient visual data, precisely because it has learned to extract temporal information that requires multiple frames to capture. DINOv2, as a static image model, experiences very little penalty from frame scarcity (most frames look similar from its perspective), resulting in faster saturation. The scaling exponents $\alpha$ confirm this: DINOv2's $\alpha = 1.76$ (fast saturation) vs. VideoMAEv2's $\alpha = 0.75$ (slow saturation, more gradual improvement from additional frames).

The caption coefficients $C_c$ are comparable (0.13 for both), indicating that both models benefit similarly from additional textual information—the text-side behavior of the alignment is driven by the language model, not the vision encoder.

Corrected models in Appendix A.4. The paper provides fitted scaling laws for 10 model-dataset combinations in Table 1, showing consistently high $R^2$ values (0.9442–0.9979). An interesting outlier is PE Core Large on VATEX, which has $S_\infty = 0.109$ and $R^2 = 0.9710$ , indicating a very low asymptotic alignment potential—possibly because this model was trained on tasks that don't emphasize semantic concept formation.

Downstream Performance Correlation Analysis (Section 7)

The connection between alignment and downstream performance is not a standalone method but rather an analysis that validates alignment as a useful probe. The procedure is:

Step 1: Obtain frozen video features. For each self-supervised video model (VideoMAE variants at Base/Large/Huge scales, V-JEPA Huge, and the 4DS model family at Small/Base/Large/Giant/efficient scales), extract features from all test videos in the relevant downstream datasets.

Step 2: Fine-tune task-specific decoders. A separate learnable attention-based decoder is trained on top of the frozen video features for each downstream task. The decoder is task-specific and trained with supervised learning on the labeled data for that task. The paper does not provide architectural details for the decoder beyond "attention-based," but the key point is that only the decoder is trained, not the video encoder, preserving the zero-shot nature of the video representations.

Step 3: Measure downstream accuracy. Performance is evaluated on:

Semantic tasks: action classification on SSv2 (Something-Something v2) and Kinetics-700-2020 (reported in Figure 4 with "Higher better" metrics in the 0.40–0.70 range for SSv2 and 0.30–0.55 for Kinetics).
Non-semantic tasks: point tracking on the Perception Test dataset (0.76–0.83 range), box tracking on Waymo Open (0.70–0.78), camera pose estimation on RealEstate10k (0.3–0.7, lower is better), and depth estimation on ScanNet (1.0–2.0, lower is better).

Step 4: Correlate with alignment. For each video model, compute the alignment score against Gemma 2-9b-it on VATEX (using a single caption per video and the model's standard configuration). Plot downstream accuracy against alignment score and compute the Pearson correlation coefficient $R$ and $p$ -value.

The reported correlations in Figure 4 are:

SSv2 action classification: $R = 0.883$ , $p = 0.002$ (strong, significant)
Kinetics action classification: $R = 0.923$ , $p = 0.000$ (very strong, highly significant)
Point tracking: $R = 0.396$ , $p = 0.291$ (weak, not significant)
Camera pose estimation: $R = -0.832$ , $p = 0.005$ (strong negative, significant—lower error correlates with higher alignment)
Depth estimation: $R = -0.926$ , $p = 0.000$ (very strong negative, highly significant)
Object tracking: $R = 0.820$ , $p = 0.007$ (strong, significant)

The sign of the correlation depends on whether the downstream metric is "higher better" (classification, tracking) or "lower better" (pose error, depth error). For error metrics, a negative correlation means models with better alignment make smaller errors.

Why this analysis matters: It establishes that text alignment—measured without any text supervision during video model training and without any task-specific fine-tuning—carries information about the quality of video representations for a broad range of tasks. This is the core evidence for the claim that alignment can serve as a zero-shot evaluation metric. The exception (point tracking) is instructive: it's a highly local task requiring pixel-precise spatial correspondence, which may rely on low-level visual features that are not well-captured by the semantic abstractions measured by text alignment.

Temporal Analysis Experiments (Section 8)

The temporal probing experiments use alignment measurement diagnostically to understand how different models encode temporal information, not just how much they align.

Test of Time dataset. A synthetic dataset of 180 $(video, caption)$ pairs designed to isolate temporal reasoning. Each caption is of the form "A $c_1$ circle appears {after, before} a $c_2$ circle," where $c_1, c_2$ are selected colors. Videos are 30-frame clips where the shapes appear in specific regions of a black square (upper left, upper right, bottom left, bottom right). The dataset contains logically related captions: for each color pair and location, there are four captions—two permutations of the colors with "after" and two with the equivalent "before" construction (e.g., "red after blue" is logically equivalent to "blue before red"). The videos may be identical or different depending on shape locations.

With $N = 180$ pairs, the paper experiments with $k = 1, 2, 3$ for the mutual k-NN metric. When $k = 3$ , most models achieve near-perfect alignment (around 0.8–1.0 in Figure 5), because each example has roughly 3 distinct nearby neighbors (the logically equivalent and related captions). However, the $k = 1$ and $k = 2$ alignments differ significantly across models, revealing how each model ranks the neighbors—specifically, which of the three related captions is considered closest.

The key qualitative finding: language models tend to rank the bag-of-words match as closest (e.g., "c1 after c2" has "c2 after c1" as its nearest neighbor because they share almost all the same words), while video models put different neighbors first depending on their temporal processing architecture. The paper interprets this as evidence that "LLMs measure closeness more akin to a bag-of-words than being temporally sensitive, at least at the shallower layers from where we are extracting the features."

VideoComp temporal reorder experiment. A more realistic test using VideoComp's test set (Kim et al., 2025), sourced from YouTube and based on ActivityNet Captions and YouCook2. The "temporal reorder" subset provides 512 $(video, positive\_caption, negative\_caption)$ triples, where the negative caption describes the same events as the video but in a different temporal order.

The procedure is:

Compute the standard alignment score between video embeddings and positive caption embeddings using $k = 5$ (lower than the standard $k = 10$ due to the smaller dataset size of 512).
Recompute neighbors in the text space by replacing each positive caption $c_i$ with its negative counterpart $\tilde{c}_i$ and finding its nearest neighbors among the positive captions of other videos. This tests: if the model encodes temporal structure, the negative caption should have different neighbors than the positive caption, reducing alignment.
Report both the positive alignment score and the negative alignment score.

The results (Figure 6) show that alignment with negative captions is consistently lower than alignment with positive captions (e.g., VideoMAEv2 Large drops from 0.234 to 0.229, Timesformer Base K400 drops from 0.259 to 0.252, PE Core Large drops from 0.318 to 0.309). However, the absolute drops are small—typically 0.005–0.010—indicating that while models show some temporal sensitivity, the effect is weak relative to the overall alignment signal. Notably, models with higher overall alignment tend to show larger absolute drops, suggesting that better-aligned models may be learning more temporally-aware structures that are perturbed by reordering. But the paper acknowledges the drops are not large enough to claim strong temporal sensitivity in current models.

Why this experimental design matters: Standard alignment measurement tells you that video and text representations agree on which items are similar. But it doesn't tell you why—is the similarity driven by semantic content (objects, actions) or by temporal structure (ordering, causality)? The temporal reorder experiment partially dissociates these: by comparing positive vs. negative (temporally shuffled) captions, it isolates the temporal component of alignment. The weak effect suggests current models' alignment is dominated by static semantic matching, with temporal structure contributing only a small additional signal.

Cross-Model (Video-Video) Alignment (Appendix A.5)

An additional analysis measures intra-modal video-to-video alignment—how similar the representation spaces of different video encoders are to each other. The procedure is identical to cross-modal alignment (mutual k-NN metric with layer optimization), but applied to pairs of video encoders rather than video-text pairs.

Figure 10 visualizes this as a pairwise alignment matrix, with the key structural finding: video encoders form roughly two clusters—one consisting of models that align better with language (measured by alignment with Gemma2) and another consisting of models that perform better on geometric/pixel-level tasks. The paper notes that some models (DINOv2 and VideoMAEv2 K710-finetuned) span both clusters, achieving high alignment with both language-aligned and geometry-aligned models.

"we hypothesize that such alignment against multiple models provides a strong indicator of the versatility of a given vision model"

This suggests that video-video alignment patterns might provide complementary information to video-text alignment for evaluating model quality—a model that aligns well with diverse other models (not just one cluster) may produce features that are useful across a broader range of downstream tasks.

Design Choices and Their Justifications (Summary)

The paper's experimental framework is built on a series of deliberate design decisions, each with specific justification:

Mutual k-NN metric over alternatives (e.g., CKA, CCA, linear probing): The mutual k-NN is dimensionality-agnostic, invariant to affine transformations, and focuses on local structure—all properties aligned with the PRH's claims about manifold-level convergence rather than raw coordinate alignment.
Layer optimization over fixed-layer comparison: This is maximally permissive, measuring the best possible alignment rather than alignment at a specific arbitrary depth. This is appropriate for probing intrinsic representational properties, but means reported scores are upper bounds that may not reflect performance in practical transfer scenarios.
Uniform frame interpolation and clip averaging over adaptive sampling: This provides a clean, parametric way to sweep the frame count without requiring the encoder to handle variable-length inputs natively. The averaging approach is simple but effective, allowing the study to isolate the effect of richer inputs without confounding it with learned aggregation.
Caption concatenation over separate encoding: This leverages the language model's attention mechanism to relate information across captions, which separate encoding and averaging would not allow. The cost is that the input sequence length grows with $n_c$ , but for the caption lengths and counts used (at most 10 captions of ~15 words each), this is well within the context windows of modern LLMs.
Gemma-2 family as primary text encoders: Selected empirically because they consistently achieved the best alignment scores. The authors note this is a finding in itself: more recent text models, even when trained purely for generation, produce representations that align better with vision—consistent with the PRH's prediction that scaling improves convergence.
Saturation-based scaling law over polynomial or exponential forms: Empirically tested and found to fit best. The theoretical justification connects it to established principles in statistical learning (manifold approximation, power-law feature importance distributions, independent modality noise), providing a principled interpretation of the fitted parameters.
Synthesized captions for PVD over using only the original caption: This extends the multi-caption analysis to datasets not originally designed for it. The use of an LLM (Gemini-2.5 Pro) rather than human annotators makes the approach scalable, though the authors acknowledge synthesized captions may have less diversity than human-written ones.
Two-model-type vision encoder taxonomy (image-on-image, image-on-video, native video): This tripartite classification (■, ●, ▲ markers in Figure 2) cleanly separates the effects of model architecture from the effects of temporal input. It enables the key comparison: does a powerful image model with frame averaging (●) outperform a video model (▲)? The answer—generally no for individual models, but some image models (like Perception Encoder Core Giant) are competitive—provides nuanced insight into the role of temporal modeling.

4. Key Insights and Innovations

Innovation 1: Test-Time Data as a First-Class Variable in Representational Alignment

The paper's most fundamental conceptual move is reframing cross-modal alignment from a fixed property of trained models into a function of test-time information richness. Prior work on the Platonic Representation Hypothesis (Huh et al., 2024) treated alignment as something you measure given a static protocol: take a pre-trained image encoder and a pre-trained text encoder, pass single images and single captions through them, and compute the resulting score. The implicit assumption was that the score reflected something intrinsic about the models—their convergence toward a shared statistical model of reality—and that richer inputs would be redundant or yield diminishing returns.

This paper systematically demolishes that assumption. By varying the number of video frames from 1 to 80 and the number of captions from 1 to 10, the authors demonstrate that alignment scores can improve by roughly 60% (from ~0.15 to ~0.25 in the single-to-multi-caption transition for many models, and up to ~0.40 for the best configurations) without retraining or fine-tuning either encoder. The significance is not the magnitude of improvement (though it's substantial) but the reframing it forces: alignment is not a static quantity to be measured once; it is a function A(n_f, n_c) with structured, predictable behavior. This transforms alignment from a descriptive statistic into an object of scientific study with its own scaling properties.

The comparison to prior work is stark. Huh et al. (2024) explicitly flagged that the maximum observed alignment (~0.18) was ambiguous—it could represent either strong alignment hitting a fundamental ceiling or poor alignment with major residual differences. They left this as an open question. This paper provides a concrete, empirically validated answer: the scores were low in part because the test-time data was impoverished. Providing richer data pushes alignment substantially higher, indicating that the latent spaces were more aligned than the static measurement protocol revealed. This doesn't fully resolve the open question (scores still fall well short of 1.0), but it demonstrates that a significant fraction of the "missing" alignment is recoverable through better probing rather than requiring better models.

This is not an incremental refinement of measurement methodology. It is a fundamental shift in what the research question even is. Before this paper, the main question was "Do independently trained vision and text encoders converge to aligned representations?" After this paper, the question becomes "How does alignment scale with the information provided at test time, and what does that scaling reveal about the encoders' capabilities?" This reframing opens an entirely new axis of investigation—test-time compute scaling for multimodal understanding—that is orthogonal to the training-time scaling studied in prior work.

The evidence for this reframing is not a single figure but the entire empirical structure of the paper: Figure 3's sweep over frames and captions, the parametric scaling law fits in Table 1, and the consistent observation across both VATEX and PVD datasets (Figures 3 and 8) that the relationship is smooth, monotonic, and capturable by a simple functional form. The fact that the scaling behavior is predictable (R² > 0.98 in the best cases) means it's not just "more data helps" but "we can model exactly how much it helps and use that model to make quantitative predictions."

Innovation 2: Parametric Test-Time Scaling Laws as a Diagnostic Tool for Encoder Capabilities

Beyond demonstrating that alignment scales with test-time data, the paper introduces a parametric framework for quantifying how different encoders process additional information. The saturation-based scaling law score(n_f, n_c) = S_∞ - (C_f n_f^{-α} + C_c n_c^{-β}) is not just a curve-fitting exercise—it's a diagnostic instrument that decomposes alignment behavior into interpretable parameters reflecting fundamentally different aspects of encoder capability.

The key intellectual contribution here is the separation of asymptotic alignment potential (S_∞) from data efficiency (C_f, C_c, α, β). Prior work collapsed these into a single alignment score; this paper shows they are distinct and can be disentangled. For example, VideoMAEv2 Huge and DINOv2 Giant have different asymptotic potentials (S_∞ ≈ 0.41 vs. 0.37) but also dramatically different frame efficiency characteristics: VideoMAEv2's frame coefficient C_f = 0.15 is roughly triple DINOv2's C_f = 0.05, while its frame exponent α = 0.75 is less than half of DINOv2's α = 1.76. This parameter triplet tells a nuanced story: VideoMAEv2 can ultimately align better with text than DINOv2 (higher S_∞), but reaching that potential requires substantially more visual data because it has learned to extract temporally rich features that single frames don't capture (higher C_f, lower α). DINOv2 saturates quickly with few frames because it treats all frames as approximately equivalent static images, limiting both its need for data and its ultimate ceiling.

This is a conceptual advance in how we evaluate and compare encoders. Before this paper, comparing a video encoder to an image encoder typically meant choosing a fixed protocol (e.g., 16 frames for video, 1 frame for image) and comparing downstream accuracy. This conflates the encoder's capacity with its data efficiency under that specific protocol. The scaling law framework separates these: S_∞ measures capacity independent of data budget, while the coefficients and exponents measure how efficiently the encoder uses additional data. This matters practically because deployment scenarios differ—some applications can afford 80 frames, others are limited to single-frame snapshots—and the "best" encoder for one budget might not be best for another.

The connection to pretraining scaling laws (Hoffmann et al., 2022) is explicit and instructive. Just as Chinchilla laws model loss as L(N, D) = E + A/N^a + B/D^b, separating irreducible error from resource-dependent reducible error, this paper's test-time laws separate asymptotic alignment from data-dependent reducible gaps. The paper is effectively proposing an inference-time analog of scaling laws for representational alignment. Crucially, however, the exponents here reveal something about the encoders, not the data distribution: α reflects how the vision encoder perceives the dimensionality of the video manifold, while β reflects how the text encoder perceives caption diversity. This makes the fitted parameters interpretable as encoder fingerprints—quantitative descriptors of how each model structures its latent space.

The theoretical justification in Appendix B strengthens this from "empirical fit" to "principled measurement." By connecting the exponents to intrinsic dimensionality (α = 2/d_v where d_v is the effective dimension of the video as perceived by the vision encoder) and to Zipfian feature importance distributions, the paper provides a chain of reasoning that makes the parameters meaningful rather than arbitrary. When DINOv2 shows α = 1.76, this implies it perceives videos as having effective dimensionality d_v ≈ 1.1—essentially treating video as a slightly perturbed 1D manifold, consistent with its static-image architecture. When VideoMAEv2 shows α = 0.75, this implies d_v ≈ 2.7—a richer, higher-dimensional manifold that requires more samples to characterize. These numbers are not just curve-fitting noise; they capture genuine architectural differences in how models represent temporal data.

This is a fundamental methodological contribution rather than an incremental improvement. The scaling law framework is general: it can be applied to any encoder pair, any dataset with variable per-item information, and potentially any alignment metric. It provides a common language for comparing encoders across architectures and modalities on a dimension that training-time scaling laws can't access—how efficiently they consume data at inference time.

Innovation 3: Alignment as a Zero-Shot Proxy for General-Purpose Video Representation Quality

The correlation analysis in Section 7 makes a case that cross-modal alignment with text is not merely an interesting theoretical phenomenon but a practically useful signal for model evaluation and development. The finding that video-text alignment scores correlate strongly with downstream task performance (R = 0.883–0.923 for action classification, R = -0.832 to -0.926 for depth estimation and camera pose) on models trained without any text supervision is significant because it establishes alignment as a zero-shot evaluation metric that avoids the expensive fine-tuning typically required to assess self-supervised video representations.

What makes this intellectually distinctive is not just the correlations themselves but the implicit claim about what alignment measures. If alignment with a text encoder correlates with depth estimation accuracy (a purely geometric task with no semantic content), then alignment must be capturing something deeper than "semantic similarity to language." The paper's interpretation—that alignment reflects a kind of general-purpose representational quality, consistent with the PRH's claim that converged representations capture fundamental structure of the world—is supported by the breadth of tasks showing correlation. The exceptions are equally informative: point tracking, a highly local task requiring precise spatial correspondence, shows weak correlation (R = 0.396, p = 0.291), suggesting that alignment primarily captures mid-to-high-level structural properties rather than low-level pixel-precise features.

This finding addresses a concrete pain point in the video representation learning community. As the paper notes:

"evaluating self-supervised video representations is challenging, and current approaches rely on expensive task-specific training"

The standard workflow for developing a new video encoder involves fine-tuning on multiple downstream benchmarks—each requiring hyperparameter tuning, multiple random seeds, and substantial compute—to get a signal on whether the pre-training improvements are meaningful. If alignment with a strong text encoder can substitute for some of this evaluation, it could dramatically accelerate the model development cycle. The paper doesn't claim alignment replaces downstream evaluation (the correlations, while strong, aren't perfect), but its value as a fast, cheap, diagnostic signal is clear—analogous to how validation loss guides training even though final evaluation uses test-set metrics.

The comparison to prior work is instructive. Maniparambil et al. (2024a) showed a similar correlation between image-text alignment and pure vision task performance for image models. This paper extends that finding to video—a non-trivial extension because video models must capture temporal dynamics that might not be reflected in text alignment. The fact that the correlation holds for video models trained purely on reconstruction objectives (VideoMAE variants, V-JEPA) makes the finding striking: these models have never "thought about" language, yet their representations organize concepts in ways that mirror linguistic structure. This is strong evidence for the PRH's central claim—that convergence arises from the structure of the world, not from explicit cross-modal training.

This is a significant but incremental advance over prior art—incremental in that it extends a known phenomenon (image-text alignment correlating with vision task performance) to a new modality, but significant in that it establishes the practical utility of alignment measurement for an entire subfield (self-supervised video learning) that previously lacked cheap evaluation tools. The practical impact could be substantial even if the theoretical contribution is evolutionary rather than revolutionary.

Innovation 4: Diagnosing Temporal Reasoning Through Structured Alignment Probing

The temporal analysis experiments (Section 8) represent a novel diagnostic use of alignment metrics that goes beyond measuring how much alignment exists to probing what kind of information drives that alignment. By constructing controlled perturbations to the temporal structure of captions—logical reversals in the Test of Time dataset and event reordering in VideoComp—the paper uses alignment as an instrument to dissect the nature of encoder representations rather than as an endpoint.

The key conceptual insight is that alignment scores are composite signals reflecting multiple types of representational similarity: appearance similarity, object co-occurrence, action recognition, and temporal ordering. These components are typically confounded in standard alignment measurements. The temporal perturbation experiments partially deconfound them by holding semantic content constant while varying temporal structure. The finding that alignment drops only slightly (0.005–0.010 in absolute terms) when captions are temporally reordered in VideoComp (Figure 6) reveals that current alignment is dominated by static semantic matching—models agree on what happens but not necessarily in what order. The Test of Time experiment (Figure 5) reinforces this: language models cluster captions by word overlap (bag-of-words behavior) rather than by temporal logic, suggesting that the alignment between video and text models may be mediated primarily by appearance-level and action-level features rather than temporal reasoning.

This is a methodological innovation rather than a positive result. The paper provides a template for using alignment as a diagnostic tool: identify a property you care about (temporal reasoning), construct counterfactual inputs that vary that property while holding others constant, and measure the differential alignment. The small differential observed in VideoComp is not a failure—it's an informative diagnostic indicating that current models have substantial room for improvement in temporal reasoning, and that the alignment metric can detect this weakness. This transforms alignment from a scalar quality score into a multi-dimensional probe that can characterize the strengths and weaknesses of different encoders.

The contrast with prior work in video-language understanding is sharp. Existing benchmarks for temporal reasoning (e.g., temporal ordering tasks in video QA) evaluate explicit task performance—can a model answer "what happened first?" correctly? This paper's approach evaluates implicit temporal sensitivity—does the model's representational geometry reflect temporal structure even when no task demands it? The two are complementary: explicit task performance measures what a model can do, while alignment probing measures what a model knows in its latent structure. Models might fail explicit temporal reasoning tasks because they can't decode temporal information from their representations, even if the representations encode it. Conversely, they might pass temporal reasoning tasks by exploiting dataset biases without genuinely representing temporal structure. Alignment probing provides a different, potentially more direct, window into the geometry of the latent space.

This is a fundamentally new capability made possible by the alignment measurement framework. Before this work, there was no established method for asking "does this video encoder represent temporal order?" without training a task-specific probe—which introduces confounds from probe capacity, optimization, and dataset bias. The mutual k-NN alignment metric with controlled input perturbations provides a training-free, architecture-agnostic way to answer such questions, opening a research program on diagnostic probing of video representations that extends far beyond the specific findings in this paper.

Innovation 5: Decomposing Alignment into Modality-Specific Contributions Through Additive Error Structure

The paper's scaling law formulation score(n_f, n_c) = S_∞ - (C_f n_f^{-α} + C_c n_c^{-β}) embodies a specific structural claim: that the total alignment error is the sum of independent contributions from the vision modality (limited frames) and the text modality (limited captions). This additive structure, validated by the high R² values, is a conceptual contribution in itself because it implies that the information gained from additional frames is largely orthogonal to the information gained from additional captions—at least from the perspective of the alignment metric.

This is not an obvious or necessary result. One could imagine a model where frames and captions are partially substitutable: if you have very rich text descriptions, perhaps you don't need many frames to achieve good alignment, because the text already disambiguates the visual content. Or conversely, if you have many frames capturing the full visual dynamics, perhaps a single caption suffices. The additive error structure says otherwise: both matter independently, and saturating one doesn't compensate for scarcity in the other. This has implications for how we think about multimodal data collection and evaluation: you can't trade off visual diversity for textual diversity; you need richness in both modalities to approach the asymptotic alignment S_∞.

The theoretical justification in Appendix B provides a principled basis for this independence. Under the assumption that video and text encoders produce noisy estimates of a shared ideal representation, with noise processes that are uncorrelated across modalities, the alignment error decomposes additively because the cross-term in the expected squared error vanishes: E[∥η_V^T η_T∥] = 0. This is not just a mathematical convenience—it's a testable claim about the structure of encoder representations, and the empirical support for it tells us something about how these models work.

The comparison to prior work is instructive for what it reveals about the limits of static-modality studies. Huh et al. (2024) varied training-time resources (model size, dataset size) and observed improved alignment, but couldn't separate whether the improvement came from better visual representations, better text representations, or better cross-modal correspondence. The additive decomposition framework separates these: you can estimate S_∞, C_f, and C_c independently, quantifying how much of the alignment gap is due to the vision encoder's limitations versus the text encoder's. This is a diagnostic refinement that enables more targeted model improvement: if C_f is large and α is small, the vision encoder needs more frames (or a better temporal architecture); if C_c is the bottleneck, invest in better text encoders or more diverse captions.

This is a moderate theoretical advance—it formalizes and validates an independence assumption that simplifies the analysis of multimodal alignment and provides interpretable parameters for model comparison. It is not a fundamental breakthrough (the additive structure follows from standard statistical assumptions), but it is a clean, well-validated contribution that makes the scaling law framework analytically tractable and practically useful.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary dataset is VATEX (Wang et al., 2019), from which the authors randomly sample N = 1024 videos to construct a test set. Each video is approximately 10 seconds long, sourced from unique YouTube videos, and accompanied by 10 English captions written by different human annotators (each caption averaging ~15 words). A secondary dataset, PVD (Perception Encoder Video Datasets; Bolya et al., 2025), is used for validation; its captions are synthesized from original detailed descriptions using Gemini-2.5 Pro to produce 10 shorter captions per video (Appendix A.3). Additionally, two datasets are used for temporal probing: Test of Time (Bagad et al., 2023), a synthetic dataset of 180 (video, caption) pairs designed to isolate temporal reasoning through color-shape ordering, and VideoComp (Kim et al., 2025), whose temporal reorder subset provides 512 (video, positive caption, negative caption) triples where the negative caption describes the same events in a different order.
Base model(s). The paper evaluates 85 vision model variants and 30 language model variants (121 total). Vision encoders span self-supervised models (VideoMAE/VideoMAEv2 at Base/Large/Huge scales; Tong et al., 2022; Wang et al., 2023), image models applied to video (DINOv2 at Small through Giant; Oquab et al., 2023), text-aligned models (CLIP Base through Huge; Radford et al., 2021), and recent architectures (V-JEPA 2, DINOv3, Perception Encoder at multiple scales). Language encoders include T5 (Small through 11B), BLOOM (560M–7.1B), Llama (7B–65B), Llama 3 (1B–70B), Gemma (2B, 7B), Gemma 2 (2B-it, 9B-it, 27B-it), and Gemma 3 (1B-it through 27B-it). The Gemma-2 family is identified as the strongest text encoder, and the Gemma 2-9b-it model is used as the reference text encoder for the primary scaling law analysis and downstream correlation experiments.
Metrics. The primary metric is the Mutual k-NN (MkNN) alignment score, defined as A_MkNN(X, Y) = (1/(kN)) Σ_i Σ_j (M_X ⊙ M_Y)_ij, where M_X and M_Y are binary neighbor indicator matrices for the vision and text embedding spaces respectively, with k = 10 neighbors for N = 1024 test items (reduced to k = 5 for the 512-item VideoComp subset). This measures the average fraction of k-nearest neighbors that are shared between the two embedding spaces. For downstream performance, the paper uses task-specific accuracy metrics: action classification accuracy on SSv2 and Kinetics (higher is better), point tracking accuracy on the Perception Test (higher is better), box tracking accuracy on Waymo Open (higher is better), camera pose estimation error on RealEstate10k (lower is better), and depth estimation error on ScanNet (lower is better). For the test-time scaling laws, goodness-of-fit is measured via the coefficient of determination R².
Baselines. The paper's baseline comparisons are primarily internal rather than against external methods. The key baselines are: (1) Image model on image (■ in Figure 2)—a single first frame encoded by an image model, replicating the static-modality protocol of prior work (Huh et al., 2024), which yields alignment scores up to ~0.206 with Gemma-2 text encoders; (2) Image model on video (● in Figure 2)—8 uniformly sampled frames encoded individually and averaged, providing a naive temporal baseline; (3) Native video models (▲ in Figure 2)—spatio-temporal encoders that process multi-frame clips natively; (4) Single-caption alignment (n_c = 1) as the baseline against which multi-caption alignment (n_c up to 10) is compared; and (5) Majority-class and random baselines for downstream tasks, though the paper does not report explicit chance-level alignment scores (the theoretical maximum is 1.0). For the scaling laws, the baseline for comparison is the single-frame, single-caption protocol that would correspond to n_f = 1, n_c = 1 in the parametric framework—the setup implicitly used in all prior work.
Generation budget / compute accounting. The paper does not measure compute in FLOPs or wall-clock time. Instead, the "budget" is parameterized directly by the two test-time data axes: number of frames n_f and number of captions n_c. For the video pipeline, n_f is swept across values including 1, 2, 4, 8, 16, 32, 48, 64, and 80 frames, with sub-clip averaging when n_f exceeds the encoder's native clip length (typically n_0 = 16). For the text pipeline, n_c is swept across 1, 2, 4, 8, and 10 captions, with captions concatenated into a single input string. The cost of computing alignment scores—which requires extracting embeddings from all layers of both encoders for all N = 1024 items and evaluating all L_v × L_t layer pairs—is not quantified in the paper. The difficulty estimation problem that dominates the inference-budget discussion in test-time compute papers (as in the reference example) is absent here: there is no allocation policy that must estimate video "difficulty" before committing to an (n_f, n_c) budget. Instead, the entire (n_f, n_c) grid is evaluated for a fixed test set, making this a measurement study rather than an adaptive deployment strategy. The scaling law fitting uses offline regression on the full grid and does not account for the cost of evaluating the grid.
Cross-validation / statistical protocol. There is no cross-validation for strategy selection, because no strategy is being selected—the paper measures alignment under all configurations and fits a parametric model to the resulting surface. For the correlation analysis in Section 7, the paper computes the Pearson correlation coefficient R and associated p-value between alignment scores and downstream task performance across the set of evaluated video models (9 models in Figure 4). The p-values are reported individually for each task, not corrected for multiple comparisons across the 6 tasks tested. The scaling law fits in Table 1 report R² values for each model-dataset pair, with the fitting performed on the full grid of (n_f, n_c) values without held-out test points (the grid resolution is not specified, but from the axes in Figure 3, it appears to involve roughly 5–6 frame settings × 4–5 caption settings = 20–30 data points per fit). For the temporal analysis, the Test of Time experiment (Figure 5) reports alignment scores at k = 1, 2, 3 without confidence intervals, and the VideoComp experiment (Figure 6) reports positive and negative alignment scores as point estimates.

Main Quantitative Results

Alignment Between Video and Text Encoders (Single-Caption Baseline)

Figure 2 provides the foundational result: when using only a single randomly selected caption per video on VATEX, modern video encoders achieve competitive or superior alignment with text compared to the best image encoders. The headline numbers:

The best video model (VideoMAEv2 Huge, ▲) achieves alignment scores of approximately 0.25–0.28 with Gemma 2-9b-it as the text encoder, placing it above all image models on both image (■) and image-on-video (●) configurations. This is the paper's central evidence that natively trained video models can outperform image-based models in text alignment.
Pure image-on-image models (■) achieve a maximum alignment of approximately 0.206 with Gemma 2-9b-it (DINOv2 Giant), closely matching previously reported values of ~0.18 from Huh et al. (2024) when using older text encoders. The paper notes this consistency as validation: "the alignment scores that we obtain closely follow previously reported values on other datasets."
Image models applied to video via frame averaging (●) can reach alignment scores of approximately 0.223, outperforming their image-only counterparts. This demonstrates that even a naive temporal aggregation—8 frames encoded independently and averaged—provides a measurable improvement over single-frame encoding, without any video-specific architecture.
Among the text encoders, the Gemma-2 family (Gemma 2-2b-it, 9b-it, 27b-it) consistently produces the highest alignment scores with vision models. The fitted linear regression lines in Figure 2 show correlation coefficients r = 0.87 (Gemma 2-9b-it), r = 0.87 (Llama-7b), and r = 0.88 (Gemma-7b) between alignment scores and downstream video retrieval accuracy on Kinetics-400, and r = 0.81, 0.81, and 0.82 respectively on SSv2. This indicates that alignment against different text encoders provides similarly strong predictions of downstream performance.

These results directly address the paper's first motivating question: does the Platonic Representation Hypothesis extend to the temporal domain? The answer from Figure 2 is a qualified yes. Video models trained purely on reconstruction objectives (VideoMAEv2, with zero text supervision) achieve alignment scores that not only match but exceed those of the best image models studied in prior work. However, the absolute scores (maximum ~0.28 with a single caption) remain well below the theoretical maximum of 1.0, leaving substantial room for interpretation about how "aligned" these models actually are.

Test-Time Scaling with Frames and Captions

Figure 3 contains the paper's signature result: alignment scores scale strongly and predictably with both the number of video frames and the number of text captions provided at test time. The key numbers:

Frame scaling (Figure 3, left): With 10 captions fixed, VideoMAEv2 Huge improves from an alignment score of approximately 0.30 at n_f = 1 frame to approximately 0.40 at n_f = 48 frames—an improvement of roughly 33% relative. DINOv2 Giant, by contrast, improves from approximately 0.30 at n_f = 1 to only approximately 0.34 at n_f = 48 frames—a much smaller relative gain of roughly 13%. At the single-caption level (bottom curve for each model), the absolute gains are smaller but still present: VideoMAEv2 Huge improves from approximately 0.24 to 0.28, while DINOv2 Giant improves from approximately 0.20 to 0.22.

Caption scaling (Figure 3, right): With 48 frames fixed, VideoMAEv2 Huge improves from approximately 0.28 at n_c = 1 caption to approximately 0.40 at n_c = 10 captions—an improvement of roughly 43%. DINOv2 Giant improves from approximately 0.22 at n_c = 1 to approximately 0.34 at n_c = 10—a larger relative gain of roughly 55%, but from a lower baseline. The effect of captions is most pronounced in the low-caption regime: going from 1 to 2 captions produces a large jump, with diminishing returns beyond approximately 4–8 captions.

Combined effect: The best configuration tested—VideoMAEv2 Huge with 48 frames and 10 captions—achieves an alignment score of approximately 0.40, which is roughly double the ~0.20 achieved by single-frame, single-caption baselines for many models. This is the paper's direct answer to the open question from Huh et al. (2024): a substantial fraction of the "missing" alignment in prior work was due to impoverished test-time data, not fundamental representational divergence.

The interaction between frame and caption scaling is notable: the curves in Figure 3 (left) are separated by roughly constant vertical gaps as captions increase, and the curves in Figure 3 (right) are similarly separated by roughly constant gaps as frames increase. This suggests an approximately additive relationship, which the saturation-based scaling law in Equation 2 formalizes.

Figure 7 quantifies the overall multi-caption effect: plotting single-caption alignment scores against 10-caption alignment scores for a diverse set of vision models reveals a line of best fit y = 1.6x − 0.018 with r = 0.998. This means that going from 1 to 10 captions improves alignment by approximately 60% on average across all model types.

Figure 8 extends these findings to the PVD dataset, where captions are synthesized by Gemini-2.5 Pro rather than human-written. Video models again show strong improvements from additional frames, while image models plateau quickly. For multi-caption scaling (Figure 8, right), using just 3 synthesized captions already outperforms the original detailed caption (dashed lines) for all models, and performance saturates around 6 captions—suggesting diminishing diversity in the synthesized captions.

Test-Time Scaling Law Fits

The parametric scaling law score(n_f, n_c) = S_∞ − (C_f n_f^{-α} + C_c n_c^{-β}) achieves remarkably high coefficients of determination. The headline fits from Section 6:

VideoMAEv2 Huge (VATEX, Gemma 2-9b-it): R² = 0.9791, with S_∞ ≈ 0.41, C_f = 0.15, C_c = 0.13, α = 0.75, β = 1.30.
DINOv2 Giant (VATEX, Gemma 2-9b-it): R² = 0.9964, with S_∞ ≈ 0.37, C_f = 0.05, C_c = 0.13, α = 1.76, β = 1.40.

The parameter comparison reveals the structural difference between video and image models: VideoMAEv2's frame coefficient C_f = 0.15 is roughly triple DINOv2's C_f = 0.05, indicating that the video model's alignment is far more dependent on frame count—it loses much more alignment when starved of temporal information. Simultaneously, VideoMAEv2's frame exponent α = 0.75 is less than half of DINOv2's α = 1.76, meaning the video model saturates more slowly, continuing to benefit from additional frames even at higher counts. The caption coefficients C_c ≈ 0.13 are comparable for both models, confirming that the text-side scaling behavior is driven primarily by the language model, not the vision encoder.

Table 1 in Appendix A.4 extends the scaling law analysis to 10 model-dataset combinations. R² values range from 0.9442 (Timesformer Base on VATEX) to 0.9979 (DINOv2 Giant on PVD), with most above 0.98. Several patterns emerge: Perception Encoder Core Large on VATEX shows an anomalously low S_∞ = 0.109, suggesting limited semantic representational capacity despite its video training. VideoMAEv2 Base shows C_f = 0.156 (comparable to the Huge variant's 0.146) but with α = 0.451—even slower saturation than the Huge model's α = 0.748, suggesting that larger video models integrate temporal information more efficiently (higher α, faster approach to S_∞) even though both benefit substantially from additional frames.

Correlation Between Alignment and Downstream Performance

Figure 4 reports the correlations between video-text alignment scores (single caption, Gemma 2-9b-it, on VATEX) and performance on six downstream tasks for 9 self-supervised video models (VideoMAE at Base/Large/Huge scales, V-JEPA Huge, and the 4DS family at Small/Base/Large/Giant/efficient scales):

SSv2 action classification: R = 0.883, p = 0.002
Kinetics action classification: R = 0.923, p < 0.001 (reported as p = 0.000)
Point tracking (Perception Test): R = 0.396, p = 0.291 (not significant)
Camera pose estimation (RealEstate10k): R = −0.832, p = 0.005 (negative because lower error is better)
Depth estimation (ScanNet): R = −0.926, p < 0.001 (negative for same reason)
Object tracking (Waymo Open): R = 0.820, p = 0.007

The semantic tasks (action classification) show the strongest positive correlations, which is intuitive: action recognition requires understanding what is happening in a video, which maps naturally onto the kind of semantic content that captions describe. The non-semantic perception tasks (depth, pose, tracking) show strong but slightly weaker correlations, with the exception of point tracking (R = 0.396, not significant). The paper interprets point tracking's weak correlation as reflecting the "highly local nature" of the task—point tracking requires precise pixel-level correspondence that may not be captured by the mid-to-high-level semantic features measured by text alignment.

The negative correlations for pose and depth estimation are particularly informative because these tasks have no obvious semantic content—a model estimating camera pose is performing geometric reasoning, not recognizing objects or actions. The fact that text alignment correlates with these tasks (R ≈ −0.83 to −0.93) suggests that the representational quality measured by alignment is not limited to semantic understanding but reflects a more general capacity for structured representation, consistent with the PRH's claim of convergence toward fundamental world structure.

The scatter plots in Figure 4 reveal monotonic but not strictly linear relationships. For example, the 4DS-efficient model (4DS-e) appears as an outlier on several tasks—achieving competitive or better performance than the trend line would predict given its modest alignment score—suggesting that alignment captures a specific aspect of representation quality that doesn't fully account for task-specific architectural advantages.

Downstream Retrieval Correlation

Figure 2 encodes an additional correlation that is less prominently discussed but informative: alignment scores against Gemma 2-9b-it correlate with zero-shot video retrieval accuracy (weighted k-NN using frozen encoder features, with k = 8) on both Kinetics-400 and SSv2. The regression lines show strong positive relationships, with the "avg. video - image" annotation indicating that native video models achieve approximately 0.014 higher retrieval accuracy at the same alignment score compared to image models on video—a modest but consistent advantage. This provides convergent evidence that alignment is not merely a theoretical curiosity but tracks practically useful representation quality.

Temporal Analysis

The temporal probing experiments provide diagnostic rather than benchmark results:

Test of Time (Figure 5): At k = 3 neighbors, most models achieve near-perfect alignment (0.79–1.00), because the dataset is designed such that each example has three logically related neighbors. However, agreement drops substantially at lower k. At k = 1, alignment scores range from 0.17 (ViViT-B-16x2) to 0.68 (DINOv2 Giant on video). The paper's qualitative analysis reveals the source of disagreement: language models tend to rank the caption with maximal word overlap as the nearest neighbor (e.g., "c1 after c2" neighbors "c2 after c1" first), exhibiting bag-of-words behavior, while video models show varying nearest-neighbor preferences depending on their architecture and temporal processing capabilities. The divergence at k = 1, 2 but convergence at k = 3 indicates that both modalities eventually capture the logical equivalence structure of the dataset, but encode it in different representational geometries—text through lexical overlap, video through visual dynamics.

VideoComp temporal reorder (Figure 6): Alignment scores with positive captions range from 0.064 (PE Core Large video, V-JEPA 2 Giant) to 0.318 (PE Core Large video, a different configuration or split). The corresponding negative-caption alignment scores are consistently lower, ranging from 0.064 to 0.309. The absolute differences are small—typically 0.003 to 0.009—but systematic: every model shows a decrease. Models with higher positive alignment tend to show larger absolute drops (the rightmost models in Figure 6), suggesting that better-aligned models encode more temporal structure that is perturbed by reordering. However, the small magnitude of the drops (the best model, PE Core Large, drops from 0.318 to 0.309, a relative decrease of only ~3%) indicates that temporal structure contributes only a small fraction of the overall alignment signal. This is the paper's most important negative result: current video-text alignment is dominated by static semantic matching, with temporal ordering playing a secondary role.

Cross-Model (Video-Video) Alignment

Figure 10 in Appendix A.5 presents pairwise alignment among video encoders, revealing a clustering structure. Models separate into roughly two groups: one cluster aligned with language (Gemma2, bottom right) containing semantic-focused encoders, and another cluster (top left) containing geometry-focused encoders. The paper highlights that DINOv2 and VideoMAEv2 K710-finetuned span both clusters, which the authors hypothesize may indicate versatility—models that align well with diverse other encoders might produce features useful across a wider range of tasks. This analysis is qualitative and exploratory, with no quantitative claims about the predictive power of cross-model alignment patterns.

Ablation Studies and Robustness Checks

Text encoder choice (Figure 2, Appendix Figure 11): The alignment score varies substantially with the choice of text encoder. The full matrix in Figure 11 shows that for a given vision encoder, alignment scores against different text models can differ by factors of 2–3 (e.g., a model achieving 0.15 against T5 might achieve 0.25 against Gemma 2-9b-it). The Gemma-2 instruction-tuned family (2b-it, 9b-it, 27b-it) consistently produces the highest scores, followed by Llama models and Gemma-3 models. T5 and BLOOM encoders produce substantially lower alignment. This finding validates the paper's choice of Gemma 2-9b-it as the reference text encoder and motivates the claim that "more recent text models in the Gemma-2 family lead to better alignment with video models, even if trained purely for text generation." The rank ordering across vision models is largely preserved across text encoders, suggesting the alignment metric captures stable properties of the vision representations.

Single vs. multiple captions (Figures 7, 12): The transition from single-caption to 10-caption alignment in Figure 12 shows that all vision models benefit, but the magnitude of improvement varies. The best single-caption model (VideoMAEv2 Huge, ~0.25 with Gemma 2-9b-it) improves to ~0.40 with 10 captions, while the best 10-caption model (Perception Encoder Core Giant, ~0.42) achieves higher absolute scores than any single-caption model. The line of best fit in Figure 7 (y = 1.6x − 0.018, r = 0.998) indicates a near-perfect linear relationship between single-caption and multi-caption alignment across diverse vision models, suggesting that multi-caption scaling provides a roughly constant multiplicative boost rather than reordering model rankings. This is evidence that the scaling behavior is systematic and predictable, not idiosyncratic to specific model types.

Synthesized vs. human captions (Figure 8): The PVD experiments demonstrate that even automatically synthesized captions (from Gemini-2.5 Pro) produce the expected scaling behavior. Using just 3 synthesized captions outperforms the original detailed caption (dashed lines) for all tested models, and the scaling saturates around 6 captions—earlier than VATEX's 10 human-written captions, likely because synthesized captions have less diversity (despite the prompt's instruction to vary which details are included). The frame-scaling curves for PVD (Figure 8, left) replicate the VATEX pattern: video models improve strongly, image models plateau quickly. This robustness check is important because it shows the scaling phenomenon is not an artifact of VATEX's specific multi-annotator caption design.

Video-video vs. video-text alignment (Figure 10): The clustering structure in cross-model video alignment reveals that text alignment is not the only axis of representational similarity. Some video models that align poorly with text still align well with each other (the "geometric" cluster), and some models span both clusters. This suggests that alignment with text captures one specific aspect of representational quality—semantic/conceptual structure—while video-video alignment captures complementary aspects, possibly related to spatio-temporal feature quality.

Layer optimization (throughout): All reported alignment scores use the layer-optimized protocol described in the technical approach—the maximum alignment score across all pairs of vision and text encoder layers. The paper does not report an ablation comparing this to fixed-layer (e.g., final layer only) alignment, so the sensitivity of results to this design choice cannot be assessed from the paper's data. However, the layer indices are reported in Figures 11 and 12 for full transparency, allowing readers to examine which layers are selected for each model pair.

Temporal sensitivity at different k values (Figure 5): The Test of Time experiment serves partially as an ablation on the k hyperparameter of the mutual k-NN metric. At k = 1, models show divergent alignment (0.17–0.68), reflecting their different nearest-neighbor ranking preferences. At k = 2, the range narrows (0.25–0.66), and at k = 3, alignment converges to 0.79–1.00. This demonstrates that the metric's sensitivity to subtle representational differences depends on k: lower k probes finer-grained structure. The paper uses k = 10 for the main experiments (k = 5 for VideoComp's 512 items), placing the metric in a regime that captures local neighborhood agreement without being overly sensitive to the first-neighbor preference that would reflect bag-of-words vs. temporal-sensitivity differences.

Critical Assessment

Claim 1: "Cross-modal alignment highly depends on the richness of both visual and text data provided at test time" (abstract). This claim is strongly supported by Figures 3, 7, and 8. The alignment score nearly doubles from the single-frame, single-caption baseline to the best multi-frame, multi-caption configuration in some cases. The parametric scaling law captures this dependence with R² > 0.98, and the finding replicates across two datasets (VATEX, PVD) and across synthetic as well as human-written captions. The evidence is clean, well-controlled, and convincing.

However, the claim's scope is narrower than the abstract might suggest. "Richness" is operationalized very specifically: number of uniformly sampled frames (with sub-clip averaging) and number of concatenated captions. The paper does not explore other dimensions of richness—frame sampling strategy (uniform vs. motion-aware), caption diversity (human vs. LLM-written vs. paraphrased), caption selection (random vs. covering distinct aspects), or alternative aggregation methods (attention-weighted averaging, learned pooling). The finding that "richness matters" is robust, but the precise functional form and parameter values may be specific to the chosen operationalization. A different frame sampling strategy (e.g., keyframe extraction, motion-boundary sampling) might shift the scaling curves, potentially narrowing or widening the gap between video and image models.

Claim 2: "We propose parametric test-time scaling laws that capture this behavior and show remarkable predictive power" (abstract). The high R² values (0.9442–0.9979 across models) genuinely support this claim for the tested (n_f, n_c) grid. The power-law saturation form is well-motivated theoretically (Appendix B) and fits the data well. However, several caveats apply:

Interpolation vs. extrapolation: The R² values measure goodness-of-fit on the same data points used for fitting. The paper does not report held-out evaluation (e.g., fitting on a subset of the grid and testing on held-out grid points, or testing beyond the maximum observed n_f = 80, n_c = 10). The claim of "predictive power" would be stronger if the fitted model was shown to predict alignment at data scales beyond those used in fitting.
Degrees of freedom: The model has 5 free parameters (S_∞, C_f, C_c, α, β) fit to approximately 20–30 data points per model. With this parameter-to-data-point ratio, high R² is expected even for moderately flexible functional forms. The paper's claim that this specific form fits "best by a significant margin" compared to alternatives is stated but not supported by reported comparisons (no alternative functional forms or their R² values are provided, except the statement that "several formulations" were tested).
Theoretical grounding: The connection between α and intrinsic dimensionality (α = 2/d_v) is elegant but relies on strong assumptions (MSE as a proxy for the mutual k-NN metric, independence of noise processes, power-law behavior of the data manifold). The paper acknowledges these as assumptions and provides the derivation as justification rather than proof. Readers should treat the parameter interpretations as plausible and consistent with the data rather than empirically validated.

Claim 3: "Strong correlation between semantic alignment and performance on both semantic and non-semantic downstream tasks" (abstract). The correlations in Figure 4 are genuinely strong for most tasks (R = 0.820–0.923, all p < 0.01 except point tracking). This is the paper's most practically significant claim, and the evidence is solid but bounded:

Sample size: Only 9 models are used for the correlation analysis (VideoMAE B/L/H, V-JEPA H, 4DS S/B/L/G/e). With N = 9, a single outlier could substantially affect the correlation. The 4DS-e model does appear as an outlier on some tasks—it achieves competitive performance despite modest alignment—which may inflate or deflate correlations depending on the task. The paper would be stronger with a larger set of models (e.g., including the 85 vision models studied elsewhere) evaluated on the same downstream tasks.
Causal direction is untested: The paper shows correlation, not causation. It is possible that a third factor (model scale, pre-training data diversity, architecture quality) drives both alignment and downstream performance, making alignment an epiphenomenon rather than a causal mechanism. The paper doesn't claim causation, but the framing of alignment as a "proxy metric" implicitly assumes the relationship is stable enough to be useful for model development. Without testing whether interventions that improve alignment (e.g., changing frame count, caption count) also improve downstream performance, the proxy claim remains correlational.
Point tracking as a boundary condition: The weak correlation for point tracking (R = 0.396, p = 0.291) is valuable as a boundary condition—it shows that alignment is not a universal proxy and that the nature of the downstream task matters. The paper's interpretation (that point tracking requires local, pixel-level features not captured by semantic alignment) is plausible but not empirically validated. An experiment that would strengthen this interpretation: measuring alignment at earlier (lower-level) encoder layers and testing whether earlier-layer alignment correlates better with point tracking.
Single text encoder: All correlations use Gemma 2-9b-it as the text encoder. It would be informative to see whether the correlation strength depends on the choice of text encoder—does alignment against a weaker text encoder (e.g., T5) still predict downstream performance? If not, the proxy metric requires a strong text encoder, which limits its accessibility.

Claim 4: "Video-text alignment as an informative zero-shot way to probe the representation power of different encoders" (abstract). This is the paper's aspirational claim—that alignment measurement can serve as a practical evaluation tool. The evidence supports this aspiration but falls short of demonstrating it as a mature methodology:

Zero-shot is technically correct but misleading: The alignment measurement requires no task-specific training (zero-shot in that sense), but it does require: (1) a dataset of 1024 paired videos and captions, (2) full forward passes through both encoders at all layers, (3) computation of all pairwise distances for the mutual k-NN metric, and (4) optimization over all layer pairs. For large video encoders with 48 layers and text encoders with 32 layers, this means computing and evaluating 1,536 layer-pair configurations. This is computationally cheaper than fine-tuning a decoder for each downstream task, but not trivially cheap—the paper doesn't provide wall-clock time or FLOP estimates.
Sensitivity and resolution: The alignment metric produces scores in a narrow range (roughly 0.03–0.42 across all models in Figure 11). For models that cluster tightly (e.g., all DINOv2 variants within ~0.02 of each other), the metric may lack the resolution to distinguish meaningfully between different pre-training checkpoints or architectural variants. The paper doesn't report confidence intervals or measurement variance (e.g., across different random seeds for the 1024-video sample), making it hard to assess whether small differences in alignment scores are reliable.
Generalization beyond the tested setup: The paper generalizes beyond VATEX to PVD for the scaling law analysis but not for the downstream correlation analysis. It would be valuable to know whether alignment scores on PVD (or other video datasets) similarly correlate with downstream performance, or whether the correlation is dataset-specific. Without this, the claim of "probing representation power" is tied to a specific evaluation dataset.

Missing experiments that would strengthen the paper:

Held-out evaluation for scaling laws. Fitting the scaling law on, say, half the (n_f, n_c) grid and predicting the other half would provide a genuine test of predictive power. The current in-sample R² measures goodness-of-fit, not prediction.
Alignment-to-downstream correlation across multiple text encoders. Showing that the correlation is robust to text encoder choice would strengthen the claim that alignment measures something intrinsic to the vision encoder, not just compatibility with a specific text model.
Alignment at different layers and downstream correlation. The paper uses layer-optimized alignment. For the downstream correlation, does the optimal layer for alignment correspond to the layer that best predicts downstream performance? If not, the proxy metric might be improved by task-specific layer selection.
Statistical reliability of alignment scores. Bootstrapping the 1024-video sample to produce confidence intervals on alignment scores would allow assessment of whether differences between models (especially models with similar scores) are statistically reliable.
Dynamic frame sampling vs. uniform sampling. The paper acknowledges that multiple frames improve alignment, but doesn't test whether which frames are sampled matters. An ablation comparing uniform sampling, random sampling, and motion-aware keyframe extraction would clarify whether the benefit comes from more temporal coverage or specifically from covering temporal dynamics.
Caption diversity vs. caption quantity. The multi-caption improvement could come from having more words (quantity) or from having diverse perspectives (quality). An ablation comparing 10 diverse human captions to 10 paraphrases of the same caption would disentangle these.

Summary assessment: The paper's experiments genuinely support its core empirical claims: alignment scales with test-time data, the scaling is capturable by a parametric law, and alignment correlates with downstream performance for the tested models and tasks. The evidence is well-presented and internally consistent. However, the scope of validation is limited—9 models for the downstream correlation, a single primary dataset (VATEX) for most analyses, and no held-out prediction for the scaling laws. The claims are supported for what was tested, but the generalization to "probing representation power" broadly requires assumptions that the paper does not test. The most robust finding is the test-time scaling phenomenon itself (Figures 3, 7, 8), which is replicated across datasets and consistent across models. The downstream correlation finding is strong but bounded by sample size, and the scaling law parameter interpretations are theoretically motivated but not empirically validated beyond goodness-of-fit.

6. Limitations and Trade-offs

Limitation 1: The Difficulty Estimation Problem Is Unaddressed—and the Reported Gains Do Not Account for It

The assumption or constraint. The entire test-time scaling framework depends on knowing the alignment score surface score(n_f, n_c) across the full grid of frame and caption counts. Producing that surface requires computing embeddings for all (n_f, n_c) configurations, which means running the video encoder multiple times per video (once per frame count setting) and the text encoder multiple times per video (once per caption count setting), across all 1024 test videos, at all layers of both encoders, and evaluating all L_v × L_t layer pairs. The paper explicitly excludes this cost from any accounting:

"our experiments do not account for this cost largely for simplicity" (Section 3.2, discussing a related difficulty estimation issue)

In the paper's framing, test-time scaling is presented as something that improves alignment without model retraining—a free gain at inference. But the measurement infrastructure required to know what configuration to use, or to fit the scaling law in the first place, is enormous. For a video encoder with 24 layers and a text encoder with 32 layers, the layer optimization alone requires 768 alignment computations, each of which involves computing all pairwise distances among 1024 embeddings and performing k-NN search. Sweeping 5 frame settings × 5 caption settings multiplies this to 19,200 alignment computations. This is orders of magnitude more expensive than running the encoders once at a fixed configuration—what prior work did.

The consequence. The headline finding—that alignment scores improve by ~60% from 1 to 10 captions, or that VideoMAEv2 reaches ~0.40 with 48 frames and 10 captions—is an upper bound on achievable alignment given perfect knowledge of the scaling surface. In a deployment scenario where you need to choose (n_f, n_c) for a new video-text pair, you do not have this knowledge. The cost of acquiring it (running the full grid) would dominate the cost of using the chosen configuration. The paper provides no mechanism for selecting an (n_f, n_c) budget online, no difficulty estimator that could predict which videos need more frames or captions, and no amortization analysis showing that the cost of grid evaluation can be spread across many queries. A practitioner reading the paper might reasonably conclude that they should use 48 frames and 10 captions for everything—but the paper doesn't test whether the scaling law parameters generalize across datasets (VATEX and PVD show different fitted values in Table 1), so the optimal configuration is dataset-specific and possibly video-specific.

What evidence exists in the paper. The scaling law fits (Section 6, Table 1) are performed on the full grid of measurements. The excellent R² values (>0.98) demonstrate that the surface is smooth and capturable by a 5-parameter model given the measurements, but they do not demonstrate that the measurements can be obtained cheaply. The paper's cross-dataset replication (PVD in Figure 8) shows that the qualitative scaling behavior transfers, but the fitted parameters differ (e.g., VideoMAEv2 Huge on VATEX: S_∞ ≈ 0.41, C_f = 0.15, α = 0.75 vs. on PVD: S_∞ ≈ 0.30, C_f = 0.09, α = 0.48). This means that scaling laws fit on one dataset cannot be directly applied to another—the grid must be re-evaluated.

Mitigation status. The paper does not attempt to mitigate this limitation. It frames the scaling laws as a descriptive and diagnostic tool (understanding how encoders use additional data) rather than a deployment strategy, but the abstract and introduction explicitly pitch test-time data scaling as a way to "improve alignment significantly" "without modifying the underlying trained models"—language that implies practical applicability. The paper's own analogous discussion of difficulty estimation cost (Section 3.2 in the context of a different approach) acknowledges this as an "exploration-exploitation tradeoff" and "a key avenue for future work," but develops no solution. A natural mitigation—training a lightweight model to predict optimal (n_f, n_c) from a small number of initial samples, analogous to adaptive difficulty estimation in the compute-optimal test-time scaling literature—is not explored or even suggested.

Limitation 2: A Single Benchmark, a Single Text Encoder Family, and a Single Modality Pair Constrain the Generality of All Claims

The assumption or constraint. The paper's primary empirical findings—the alignment scaling behavior, the downstream correlation, the parametric scaling law parameters, and the temporal sensitivity analysis—are all established on a single dataset (VATEX, 1024 videos) using a single text encoder family (Gemma-2) as the language reference, evaluated on a single modality pair (video-text). The paper validates some findings on PVD (Figure 8), but this validation is partial: only the scaling behavior (frames and captions) is tested, not the downstream correlation. The temporal analysis uses separate datasets (Test of Time, VideoComp) but with explicitly narrow scope. The paper states its ambition in universal terms—"extend[ing] previous cross-modal alignment studies into the temporal domain"—but the empirical base is narrow by the standards required to support claims about "video representations" in general.

The consequence. Multiple failure modes are possible:

Dataset dependence of alignment scores. VATEX consists of 10-second YouTube clips with 10 crowd-sourced English captions. This specific structure—short clips with multiple independently authored descriptions—may be unusually favorable for the multi-caption scaling effect. Datasets with longer videos (minutes to hours), single-caption annotations, or domain-specific content (medical imaging, satellite video, egocentric) might show different scaling behavior or weaker absolute alignment. The scaling law parameters in Table 1 differ between VATEX and PVD even for the same model (e.g., DINOv2 Giant: C_f = 0.05, α = 1.76 on VATEX vs. C_f = 0.02, α = 1.42 on PVD), confirming dataset sensitivity.
Text encoder dependence of "alignment" as a construct. The paper uses Gemma-2 as the reference text encoder because it achieves the highest alignment scores. But "alignment" is defined operationally as mutual k-NN overlap with this specific encoder's embedding space. If a different text encoder (e.g., a domain-specific scientific language model) were used, the ranking of video encoders might change, and the downstream correlations might strengthen or weaken. The paper shows in Figure 2 that correlations with downstream retrieval are similar across three text encoders (r = 0.81–0.88), but this tests only retrieval, not the broader set of downstream tasks in Figure 4.
Modality-pair specificity of the scaling law form. The additive error structure C_f n_f^{−α} + C_c n_c^{−β} assumes independent contributions from visual and textual data. This might hold for video-text pairs because the modalities provide complementary information, but might fail for modality pairs with more redundancy (e.g., video-audio) or less direct semantic correspondence (e.g., video-LiDAR). The paper's theoretical justification (Appendix B) presents the additive structure as a general consequence of independent noise processes, but this is an assumption, not a theorem—the cross-modality noise independence might not hold for all encoder pairs.

What evidence exists in the paper. The paper provides partial cross-dataset validation (PVD, Figure 8) confirming that frame and caption scaling occur beyond VATEX, but does not replicate the downstream correlation analysis or the full scaling law comparison on PVD. The text encoder comparison (Figure 11) shows that absolute alignment scores vary substantially across text encoders (by factors of 2–3), but the paper does not test whether the scaling law functional form or the downstream correlation holds across text encoder families. For the downstream correlation (Figure 4), only N = 9 video models are tested, all from a narrow set of self-supervised methods (VideoMAE, V-JEPA, 4DS) trained on broadly similar objectives.

Mitigation status. The paper acknowledges the scope limitation implicitly by describing the work as a "first comprehensive study" and by repeatedly flagging open questions for future work. It does not claim universality. However, it also does not include the standard caveats about benchmark-specific findings that a practitioner would need to assess whether the results are likely to transfer to their domain. The paper's strongest claims—"alignment against text encoders strongly correlates with downstream performance," "alignment scores can be used as an informative zero-shot metric"—are stated without qualification about the specific datasets, text encoders, and video model families on which they were established.

Limitation 3: The Downstream Correlation Analysis Uses Only 9 Models and Does Not Establish Predictive Power

The assumption or constraint. The correlation between alignment scores and downstream performance (Figure 4, Section 7) is computed across N = 9 video models: three VideoMAE variants (Base, Large, Huge), V-JEPA Huge, and five 4DS variants (Small, Base, Large, Giant, efficient). These models are all self-supervised, trained on similar pre-training paradigms (masked autoencoding or feature prediction), and developed by a small number of research groups (with the 4DS models all coming from a single paper, Carreira et al., 2024). The paper reports Pearson R and p-values for each of six tasks, with values ranging from R = 0.923 (p < 0.001) for Kinetics classification to R = 0.396 (p = 0.291, not significant) for point tracking. The paper interprets the significant correlations as evidence that alignment can serve as a "powerful zero-shot metric for probing the quality of video representations." But correlation with N = 9—especially when 5 of the 9 points come from a single model family (4DS)—is a weak basis for a "metric," which implies reliable ranking and sufficient resolution to distinguish models with similar quality.

The consequence. Several problems arise from the small and clustered sample:

Ranking instability. With N = 9, a single model moving by one position in the ranking can substantially change R. The 4DS-efficient model appears as a visual outlier in several scatter plots in Figure 4—achieving higher downstream performance than its alignment score would predict. If this model were removed, the correlations would likely change. The paper does not report leave-one-out sensitivity or confidence intervals on the correlation coefficients, making it impossible to assess how robust the findings are to individual model inclusion.
Family effects confound alignment and architecture. The 4DS model family contributes 5 of the 9 data points. These models share architecture and training methodology, differing primarily in scale. This means the correlation may partially reflect within-family scaling (bigger 4DS models have both better alignment and better downstream performance) rather than a general relationship between alignment and performance across diverse architectures and training paradigms. A meaningful evaluation metric should rank models trained with different objectives, architectures, and data sources—not just scale variants within a single family.
No held-out prediction is tested. The paper shows correlation (a descriptive statistic) but does not test prediction: can you fit a regression on 8 models and predict the 9th model's downstream performance from its alignment score? Without such a test, the claim that alignment can "guide video model development" is aspirational. A developer cannot use the current results to predict whether their new model, with a different architecture and training procedure, will achieve better or worse downstream performance based on its alignment score alone—the regression line is specific to the 9 models tested and may not extend to new model families.
The point tracking exception undermines "general-purpose" claims. The paper presents alignment as correlating with "both semantic and non-semantic downstream tasks," but point tracking—arguably the task most distinct from semantic recognition—shows no significant correlation. This suggests alignment captures a specific kind of representation quality (mid-to-high-level structural/semantic features) rather than universal video understanding. The paper acknowledges this but frames it as a minor exception rather than as evidence that alignment is not a general-purpose metric.

What evidence exists in the paper. Figure 4 provides the raw data and statistics. The p-values are nominally significant for 5 of 6 tasks at α = 0.05, but with no correction for multiple comparisons across the 6 tasks tested. The strongest correlations (R > 0.9) are for action classification on Kinetics and depth estimation on ScanNet—tasks that are quite different in nature, which is encouraging. But the models span a wide performance range (e.g., SSv2 accuracy from ~0.42 to ~0.67), and the correlation is driven largely by this spread: within the 4DS family alone, the alignment-performance relationship appears roughly monotonic, but the VideoMAE and V-JEPA points cluster in a narrower region and contribute less to the correlation.

Mitigation status. The paper does not address this limitation. It does not discuss sample size, confidence intervals, family effects, or predictive validity. The language around the downstream correlation is assertive ("strongly correlates," "powerful zero-shot metric") without caveats about the limited model set. This is the paper's most significant overclaim relative to the evidence provided. A practitioner evaluating whether to adopt alignment as a proxy metric would need to know that the correlation is established on a small, architecturally homogeneous set of models and has not been validated predictively on held-out architectures.

Limitation 4: The Scaling Law Parameters Are Fitted In-Sample and the Functional Form Is Not Validated Through Held-Out Prediction

The assumption or constraint. The paper's parametric scaling law score(n_f, n_c) = S_∞ − (C_f n_f^{−α} + C_c n_c^{−β}) is fit to the full grid of empirical alignment scores for each model-dataset pair. The reported R² values (0.9442–0.9979) measure goodness-of-fit on the same data used for fitting—they quantify how well the 5-parameter model can reproduce the observed alignment scores, not how well it can predict alignment at unseen (n_f, n_c) configurations. With ~20–30 data points per fit and 5 free parameters, the model has substantial flexibility. The paper does not report held-out evaluation: no split of the (n_f, n_c) grid into training and test points, no prediction of alignment at frame or caption counts beyond the fitted range, and no cross-dataset prediction (fitting on VATEX and predicting PVD, or vice versa).

The consequence. The claimed "remarkable predictive power" refers to the model's ability to interpolate the observed surface, not to predict genuinely unseen points. This has several implications:

The R² values overstate practical utility. A high R² on in-sample data is necessary but not sufficient for a model to be useful for prediction. A saturated model with many parameters can achieve high R² while learning noise rather than signal. The paper mentions testing "several formulations" and finding that the saturation-based model "provided the best fit by a significant margin," but doesn't report R² values for alternative forms, so the reader cannot assess how much better the chosen form is or whether the improvement justifies the added complexity.
The asymptotic parameters (S_∞) are extrapolations. S_∞ is the predicted alignment score as n_f → ∞, n_c → ∞—a regime far beyond the observed data (maximum n_f = 80, n_c = 10). The fitted S_∞ values (e.g., 0.41 for VideoMAEv2 Huge) are extrapolations whose accuracy depends on the correctness of the power-law saturation form. If the true asymptotic behavior differs (e.g., if alignment continues to improve logarithmically rather than saturating with a power law), the fitted S_∞ could be substantially biased. The paper provides no empirical check on whether the fitted asymptote is reasonable—for example, by testing whether doubling the maximum frame count produces alignment scores consistent with the fitted curve.
Cross-dataset parameter variation is unexplained. Table 1 shows that scaling law parameters differ between VATEX and PVD for the same model (e.g., VideoMAEv2 Huge: S_∞ ≈ 0.41 on VATEX, S_∞ ≈ 0.30 on PVD). This could reflect genuine differences in the datasets (PVD captions are synthesized, not human-written; PVD videos may have different characteristics), but it could also reflect fitting instability—the parameters might be sensitive to the specific grid points, the optimization procedure, or random variation in the alignment scores. Without held-out validation, it's impossible to know whether the parameter differences are meaningful or are fitting artifacts.
The theoretical connection to intrinsic dimensionality is unvalidated. Appendix B derives α = 2/d_v, connecting the frame scaling exponent to the effective dimensionality of the video manifold as perceived by the encoder. This is an elegant interpretation, but it depends on a chain of assumptions (MSE as a proxy for mutual k-NN, power-law behavior of the data manifold, additive independent noise) that are not empirically tested. The exponent α = 0.75 for VideoMAEv2 implies d_v ≈ 2.7, which is plausible but unverified—no independent measurement of the manifold dimensionality is provided. A skeptical reader could view the parameter interpretation as a post-hoc rationalization rather than a validated claim.

What evidence exists in the paper. The R² values and fitted parameters are reported in Section 6 and Table 1. The paper provides the theoretical derivation in Appendix B, which makes the assumptions explicit but does not test them. The cross-dataset parameter variation in Table 1 indirectly reveals fitting sensitivity: if the parameters captured stable properties of the encoders (independent of dataset), they should be more consistent across VATEX and PVD for the same model. The observed variation suggests either dataset-specific effects or fitting instability—the paper does not discuss which.

Mitigation status. The paper does not address this limitation. It does not report held-out validation, test alternative functional forms quantitatively, assess parameter uncertainty (e.g., via bootstrapping the grid points), or validate the theoretical parameter interpretations empirically. This is the most significant methodological gap in the paper, because the scaling law is presented as a major contribution ("parametric test-time scaling laws") but is validated only through in-sample fit. A minimal validation would be: fit the law on a subset of the grid (e.g., all points with n_f ≤ 32, n_c ≤ 4), predict alignment at higher budgets (n_f = 64, 80; n_c = 8, 10), and report the prediction error. The absence of this basic check means the "predictive power" claim rests on an assumption—that the fitted functional form extrapolates correctly—that the paper does not test.

Limitation 5: Temporal Sensitivity Is Weak, and the Alignment Metric Largely Measures Static Semantic Matching

The assumption or constraint. The paper positions video-text alignment as distinct from image-text alignment because video contains temporal dynamics, causality, and motion—information that static images cannot capture. The temporal analysis (Section 8) is designed to test whether alignment reflects genuine temporal reasoning or merely static semantic matching. The results show that current video-text alignment is dominated by the latter. On the VideoComp temporal reorder task (Figure 6), alignment scores drop by only 0.003–0.009 (absolute) when captions are temporally shuffled—a relative decrease of roughly 1–3% from the positive caption alignment. On the Test of Time dataset (Figure 5), language models exhibit bag-of-words nearest-neighbor preferences, and even video models with perfect k = 3 alignment show divergent k = 1, 2 behavior reflecting disagreement on temporal neighbor ranking.

The consequence. The paper's framing implies that video-text alignment captures something beyond image-text alignment—that temporal dynamics contribute meaningfully to representational convergence. The temporal analysis partially undermines this framing. If alignment drops by only ~3% when temporal order is scrambled, then ~97% of the alignment signal is attributable to static (or temporally insensitive) features: objects, actions, scenes, and their co-occurrence statistics, but not their temporal ordering. This means that:

Video-text alignment, as currently measured, is largely video-as-a-bag-of-frames alignment. The improvement from using multiple frames (Figure 3) might come from better coverage of static visual content (different viewpoints, different moments capturing different objects or actions) rather than from modeling temporal relationships between frames. A model that processed 48 randomly shuffled frames might achieve similar alignment to one that processed 48 temporally ordered frames—the paper does not test this.
The claim that "temporal dynamics play an important role in semantic grounding" (Section 5) is weakly supported. The paper demonstrates that video models achieve better alignment than image models (Figure 2), but cannot disentangle whether this advantage comes from temporal processing or from other architectural differences (larger models, different pre-training data, different training objectives). The frame-scaling analysis (Figure 3, left) shows that video models benefit more from additional frames than image models, which is consistent with temporal processing being beneficial, but does not prove it—video models might simply have larger capacity or better spatial feature extraction.
The alignment metric may not be the right tool for probing temporal reasoning. The mutual k-NN metric measures local neighborhood agreement in the full embedding space. If temporal information is represented in a subspace that contributes only modestly to overall nearest-neighbor structure (because static semantic features dominate the variance), then the metric will be insensitive to temporal reasoning even if it is present in the representations. A more targeted probe—e.g., measuring alignment specifically on the dimensions that encode temporal order, or using a metric that weights temporal features more heavily—might reveal stronger temporal sensitivity.

What evidence exists in the paper. The temporal analysis experiments (Figures 5, 6) are the direct evidence. The VideoComp reorder gap (Figure 6) is small and consistent across models. The Test of Time divergence at k = 1, 2 (Figure 5) reveals disagreement in nearest-neighbor ranking between modalities, but the convergence at k = 3 shows that this disagreement is limited to the very nearest neighbors—the broader local structure captures the logical equivalence relations. The paper is transparent about these findings, noting that "there is still room for further improvement in [models'] temporal awareness."

Mitigation status. The paper partially mitigates this limitation through its honest reporting of the temporal results—it does not overclaim temporal sensitivity. The discussion in Section 8 explicitly notes the bag-of-words behavior of language models and the small drops in VideoComp. However, the paper does not reconcile these findings with its earlier framing about temporal dynamics being important for semantic grounding. A reader who reads only the abstract and introduction might conclude that multi-frame video provides substantial temporal benefits, while the temporal analysis reveals that those benefits are largely static. The paper would be stronger if it explicitly quantified how much of the alignment improvement from multi-frame video is attributable to temporal vs. static factors—for example, by comparing alignment with temporally ordered vs. randomly shuffled frames.

Limitation 6: The Practical Deployment Gap—No Guidance on How to Use Alignment Scores for Model Development or Selection

The assumption or constraint. The paper argues that alignment can serve as a "zero-shot metric for probing the quality of video representations" that can "guide video model development" (Section 9). However, it provides no protocol for how a practitioner would actually use alignment scores in practice. Key operational questions are unanswered: What alignment score threshold is "good enough"? How large must the difference between two models' alignment scores be to reliably indicate a difference in downstream performance? Does alignment predict which layer of a video encoder to use for transfer learning? Can alignment scores track improvements during pre-training (e.g., aligning successive checkpoints against a fixed text encoder)? The paper provides a correlation plot (Figure 4) but no decision procedure, no calibration of the metric's sensitivity, and no demonstration of it being used to make a modeling decision that a downstream evaluation would validate.

The consequence. A practitioner who wants to use alignment as an evaluation metric faces several unknowns:

No calibration of the alignment-to-performance mapping. Figure 4 shows a general trend (higher alignment → better downstream performance), but the regression line is fit to 9 specific models. For a new model outside this set, what alignment score predicts competitive performance? The paper provides no confidence or prediction intervals. If a developer trains a new video encoder and measures an alignment score of 0.05, should they discard it? What about 0.06 vs. 0.07—is the 0.01 difference meaningful? The spread of points around the regression lines in Figure 4 suggests substantial variance: a model with alignment 0.05 could have SSv2 accuracy anywhere from ~0.43 to ~0.51 based on the 4DS model family alone.
No demonstration that alignment detects meaningful improvements. The strongest case for alignment as a development tool would be a demonstration that it correctly ranks a set of models whose downstream performance is known, and that it identifies the best model (or top-2 models) without access to downstream labels. The paper doesn't do this. It shows correlation across 9 models, but doesn't report whether alignment correctly identifies the best model on each task, or whether using alignment to guide model selection (e.g., picking the top-3 by alignment and evaluating them downstream) would recover the genuinely best models.
The metric's resolution relative to typical model improvements is unknown. In model development, improvements between successive training runs are often small—a 1–2% absolute improvement on a downstream benchmark is considered significant. The alignment metric would need sufficient resolution to detect differences of this magnitude. The paper provides no estimate of measurement variance (e.g., standard error of the alignment score across different random seeds for the 1024-video sample, or across different random choices of which captions to include at n_c = 1). Without this, a developer cannot know whether an observed difference of, say, 0.01 between two model checkpoints is a real improvement or measurement noise.
No protocol for text encoder selection. The paper uses Gemma 2-9b-it as the reference text encoder because it empirically achieves the highest alignment scores. But a practitioner with a new video model might not have access to this specific text encoder, or might wonder whether a different text encoder would produce a different ranking of candidate video models. The paper shows that different text encoders produce different absolute alignment scores (Figure 11) but does not test whether the relative ranking of video models is preserved across text encoders beyond the retrieval task in Figure 2. If rankings depend on the text encoder, alignment cannot serve as a model-agnostic metric without specifying which text encoder to use.

What evidence exists in the paper. The correlation analysis in Figure 4 is the primary evidence for practical utility, but as discussed in Limitations 3 and 4, this evidence is narrow and descriptive rather than predictive. The paper provides no ablation on measurement reliability, no test of ranking stability, and no demonstration of model selection guided by alignment scores. The language throughout the paper ("can serve as," "could potentially be used") is appropriately tentative about practical deployment, but this tentativeness is not matched by concrete guidance on how to operationalize the metric.

Mitigation status. The paper does not address this limitation directly. It frames the work as opening a research direction rather than delivering a production-ready tool, which is a reasonable scope for a first investigation. However, the abstract and conclusion make strong claims about utility ("informative zero-shot way to probe the representation power of different encoders") without the caveats that a practitioner would need. A more accurate framing would be: "We provide initial evidence that video-text alignment correlates with downstream performance for a specific set of models and tasks, suggesting potential as a zero-shot evaluation signal, but substantial work remains to calibrate the metric, establish its reliability, and validate its predictive power across diverse model families and training paradigms." The paper's actual framing overstates the readiness of the metric for practical use relative to the evidence provided.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the study of cross-modal representational alignment from a static measurement paradigm into a dynamic function of test-time information. Prior work treated alignment as a fixed property you observe under a standardized protocol—single images, single captions, fixed layers. The paper demonstrates that alignment scores are not stable intrinsic quantities but rather smooth, predictable functions of how much data you feed the encoders at inference time. This is a conceptual reframing, not a paradigm shift: the core methodology (mutual k-NN) and the Platonic Representation Hypothesis framework are inherited, but the object of study transforms from "how aligned are these models?" to "how does alignment scale with test-time data richness, and what does that scaling reveal about each encoder?"

The magnitude of this reframing is substantial for the subfield of representation evaluation, but modest for the broader multimodal learning community. For researchers probing emergent alignment, the paper establishes that test-time data is a first-class experimental variable that must be controlled and reported—analogous to how training compute, model size, and dataset size became standard axes in scaling law studies after Hoffmann et al. (2022). A paper reporting a single alignment score without specifying frame count, caption count, and layer selection protocol will now look incomplete, much as a scaling law paper without a FLOPs budget would. This is a lasting methodological contribution even if the specific scaling law parameters turn out to be dataset- or model-specific.

The paper resolves a specific open question from Huh et al. (2024) that had been flagged as a key uncertainty: whether alignment scores of ~0.16–0.18 represented strong convergence near a ceiling or weak alignment with major unexplained variance. The answer is nuanced: the scores were genuinely low, but a substantial fraction of the "missing" alignment was recoverable through richer test-time inputs, not through better models. The fact that alignment scores can reach ~0.40 with 48 frames and 10 captions (Figure 3) means the previously observed ceiling was an artifact of impoverished evaluation, not of fundamental representational divergence. However, scores still fall well short of 1.0, leaving the deeper question—whether full convergence is achievable—unanswered.

For the self-supervised video learning community, the paper introduces a new type of evaluation signal. Currently, developing a video encoder requires expensive fine-tuning on multiple downstream benchmarks to assess whether pre-training improvements are meaningful. The paper's finding that text alignment correlates with downstream performance (R = 0.820–0.923 for 5 of 6 tested tasks, Figure 4) suggests that alignment could serve as a fast, zero-shot diagnostic to filter promising model variants before committing to full evaluation. This does not replace downstream benchmarking—the correlation isn't perfect, and point tracking (R = 0.396) shows clear boundary conditions—but it could dramatically accelerate the inner loop of model development by providing a cheap signal for early-stage comparisons. The practical impact depends on whether the correlation generalizes beyond the 9 self-supervised models tested (see Limitations 3 and 4), which remains unvalidated.

The paper also redirects attention within the alignment literature. Prior work focused on training-time resources—model size, dataset size, number of modalities—as drivers of convergence. This paper shows that inference-time resources (frames, captions) can produce comparable or larger improvements without any retraining. This opens a new axis of investigation orthogonal to the scaling of model capacity: the efficiency with which encoders extract information from test-time data. The finding that VideoMAEv2's frame coefficient C_f is triple DINOv2's (0.15 vs. 0.05), while its frame exponent α is less than half (0.75 vs. 1.76), establishes that different encoder architectures have fundamentally different test-time scaling profiles. This makes test-time scaling efficiency a new dimension for encoder comparison, complementary to parameter count, FLOPs, and downstream accuracy.

Finally, the temporal analysis (Section 8) provides a negative result that redirects research toward a harder problem. The finding that temporal reordering of captions in VideoComp reduces alignment by only ~1–3% (Figure 6) reveals that current video-text alignment is dominated by static semantic matching—models agree on what happens but not in what order. This means the alignment gains from multi-frame video (Figure 3) likely come from better coverage of static visual content (different viewpoints, different moments capturing different objects) rather than from genuine temporal reasoning. The implication is that improving temporal sensitivity in video representations is a distinct challenge from improving overall alignment, and one that current self-supervised video models have not solved. This reframes the goal: alignment is necessary but not sufficient for temporal understanding.

Follow-Up Research This Work Enables

Training-free difficulty estimation for adaptive test-time scaling of alignment measurement. The paper's most immediate practical bottleneck is that computing the full (n_f, n_c) alignment surface for a new dataset or encoder pair is prohibitively expensive—requiring multiple encoder passes per video, full layer extraction, and all-pairs distance computation across 1024 items for every grid point. A natural follow-up is to develop a lightweight predictor that estimates the optimal (n_f, n_c) configuration from a small number of initial measurements. The experiment would: (1) train a regression model (or even a simple heuristic) that predicts the alignment score at unseen (n_f, n_c) values given alignment scores at a sparse initial grid (e.g., n_f ∈ {1, 8}, n_c ∈ {1, 4}), (2) evaluate prediction error against the full surface on held-out videos, and (3) test whether the predicted surface correctly identifies the top-ranking models within a candidate set. If successful, this would transform the scaling law framework from a descriptive measurement tool into a practical model selection protocol, analogous to how early-stopping on validation loss approximates the full training curve.

Temporally-aware alignment metrics that isolate temporal from static contributions to the alignment signal. The VideoComp temporal reorder experiment (Figure 6) revealed that only ~1–3% of the alignment signal is sensitive to temporal ordering. This suggests that the standard mutual k-NN metric, operating on full embedding vectors, is dominated by static semantic features that swamp the temporal component. A follow-up study could develop targeted alignment probes that weight embedding dimensions by their temporal sensitivity—for example, by using canonical correlation analysis to find the subspace that maximally discriminates temporally-ordered from temporally-shuffled captions, and then measuring mutual k-NN alignment restricted to that subspace. A strong result would show that temporal-subspace alignment: (a) varies more across models than full-space alignment, (b) correlates with performance on temporally-demanding downstream tasks (action anticipation, temporal action localization), and (c) is not predicted by static alignment scores alone. This would address the paper's key unresolved tension: that video-text alignment improves with frames (suggesting temporal benefits) but is insensitive to temporal reordering (suggesting static matching). The experiment would clarify whether temporal information is present in the representations but swamped by the metric, or genuinely absent.

Cross-architectural validation of the alignment-to-downstream correlation with a large-scale model zoo. The paper's downstream correlation (Figure 4) uses only N = 9 models, all self-supervised, with 5 from a single family (4DS). This is insufficient to establish alignment as a reliable proxy metric. A high-impact follow-up would be a large-scale correlation study across 50–100 video encoders spanning diverse architectures (masked autoencoders, contrastive methods, teacher-student distillation, text-supervised models, multimodal models), training paradigms (self-supervised, weakly supervised, fully supervised), and scales (millions to billions of parameters). Each model would be evaluated on both alignment (against a fixed strong text encoder, with standardized n_f and n_c) and a battery of downstream tasks. The key outputs would be: (a) the overall correlation R and its confidence interval, (b) whether the correlation holds within architecture families (not just across them), (c) whether text-supervised models "break" the correlation (since they were explicitly trained to align with text, their alignment scores might overstate their general representation quality), and (d) which downstream tasks are consistently well-predicted by alignment and which are not. This study would either validate alignment as a general-purpose proxy metric or precisely characterize its domain of applicability, providing the calibration that the current paper lacks.

Training a video encoder to maximize alignment with a frozen text encoder as an auxiliary objective. The paper studies emergent alignment in models trained without text supervision. A natural interventionist follow-up asks: if alignment correlates with downstream performance, can we directly optimize for alignment during pre-training and observe improved downstream transfer? The experiment would add an auxiliary loss during self-supervised video pre-training that encourages the video encoder's intermediate representations to maximize mutual k-NN alignment with a frozen, powerful text encoder on a paired video-caption dataset. The key scientific question is whether alignment is causal—does forcing representations to be more text-aligned genuinely improve their quality for non-text tasks, or is alignment merely an epiphenomenon that correlates with quality driven by other factors (scale, data diversity)? If the auxiliary loss improves downstream performance, it provides causal evidence that the structure captured by text alignment is functionally useful. If it doesn't improve performance (or degrades it), it suggests alignment and downstream quality are correlated because of shared underlying causes (model scale, training data richness) rather than a direct causal link. This experiment would address the deepest unresolved question from the paper: whether alignment is a signal you can optimize for, or merely a signal you can measure.

Measuring test-time scaling laws for other modality pairs and alignment metrics. The paper establishes the saturation-based scaling law form for video-text alignment measured via mutual k-NN. Two natural extensions would test the generality of both the functional form and the additive error decomposition. First, apply the same framework to video-audio alignment: sample different numbers of video frames and different durations of audio spectrograms, measure mutual k-NN alignment between video and audio encoders, and fit the scaling law. The additive error assumption predicts that video and audio contributions will be separable. Second, replace the mutual k-NN metric with alternative alignment measures: centered kernel alignment (CKA), singular vector canonical correlation analysis (SVCCA), or linear probing accuracy. The scaling law form score(n_f, n_c) = S_∞ − (C_f n_f^{−α} + C_c n_c^{−β}) might or might not fit these alternative metrics, and the fitted parameters might reveal different aspects of the representations. For example, if CKA (which measures global similarity of representational kernels) produces a higher S_∞ than mutual k-NN (which measures local neighborhood structure), that would indicate that global structure converges faster than local structure—an insight into the geometry of alignment. A negative result (the scaling law form fails for some metric-modality combinations) would delineate the scope of the framework and prevent overgeneralization.

Synthetic video datasets to isolate the features driving alignment improvement from multi-frame scaling. The paper shows that more frames improve alignment (Figure 3, left) but cannot determine why—is it better coverage of static content (different objects, viewpoints), better capture of motion and dynamics, longer temporal context providing disambiguation, or simply more total visual information? A diagnostic experiment would construct controlled synthetic video datasets that independently vary: (a) the number of distinct static views of a scene versus (b) the amount of object motion and temporal dynamics, while holding total frame count constant. For example, one condition provides 32 frames from a single static camera angle with no motion (32 samples of the same visual content), while another provides 32 frames from 32 different camera angles (maximal static diversity, no motion), and another provides 32 frames tracking a single moving object (high motion, low static diversity). Measuring alignment against captions describing either static content or temporal events would isolate which component of multi-frame information each encoder type (image vs. video) actually exploits. The hypothesis from the paper's scaling law parameters is that VideoMAEv2 benefits from temporal dynamics (motion) while DINOv2 benefits primarily from static coverage diversity—but this has not been tested. A clean dissociation would strengthen the parameter interpretation and guide architecture design: if motion is key, invest in better temporal modeling; if static coverage suffices, simpler frame-averaging approaches may be adequate.

Practical Applications and Downstream Use Cases

Rapid screening of self-supervised video models during pre-training development. A team developing a new video encoder typically trains multiple variants (different hyperparameters, architectures, pre-training datasets) and must decide which are worth the expensive full evaluation of fine-tuning on 5–10 downstream tasks. Using the alignment measurement framework, the team could: (1) select a fixed, strong text encoder (e.g., Gemma 2-9b-it), (2) set a standardized evaluation protocol (e.g., n_f = 16 frames, n_c = 5 captions, layer optimization enabled, on a fixed 1024-video subset), and (3) compute alignment scores for each model checkpoint as a cheap screening metric. Checkpoints with alignment scores below a threshold (determined from prior correlation data on similar model families) can be deprioritized, saving the compute cost of full downstream evaluation. The efficiency gain depends on the correlation strength established for the specific model family, but even a weak-to-moderate correlation (R > 0.7) would filter out substantially worse models while preserving most of the good ones. The key practical requirement—not yet met by the paper—is a calibrated threshold mapping alignment scores to expected downstream performance ranges for the model family in use.

Data collection prioritization for video-caption datasets. When curating or expanding video-caption datasets, practitioners face a tradeoff: for a fixed annotation budget, is it better to collect more videos with single captions, or fewer videos with multiple diverse captions? The test-time scaling laws provide a quantitative basis for this decision. The scaling law parameters from Table 1 indicate that for VideoMAEv2 Huge on VATEX, C_c = 0.13 and β = 1.30. Using the fitted model, one can compute the expected marginal alignment gain from adding one more caption to existing videos versus adding a new video with one caption. Since β = 1.30 is relatively large (fast saturation), the marginal gain from additional captions drops quickly—moving from 1 to 2 captions provides a substantial boost, but moving from 8 to 10 captions provides little. This suggests that for this encoder pair, a strategy of collecting 2–3 diverse captions per video (rather than 10) for a larger number of videos would maximize alignment-measured representation quality per annotation dollar. The paper's finding that even 3 synthesized captions outperform a single detailed caption on PVD (Figure 8) provides convergent evidence that moderate caption diversity yields most of the benefit.

Zero-shot model selection for video retrieval deployment. In a production video retrieval system (e.g., search over a large video corpus using text queries), an engineer must choose which frozen encoder to use for embedding both the query text and the video corpus. The standard approach is to evaluate candidate encoders on a labeled retrieval benchmark with relevance judgments, which is expensive to construct and maintain. The alignment measurement framework offers a zero-shot alternative: for each candidate (video encoder, text encoder) pair, compute the mutual k-NN alignment score on an unlabeled set of video-caption pairs (which can be automatically harvested from the web or generated by an LLM from video metadata). The encoder pair with the highest alignment score is predicted to perform best on retrieval, based on the correlation observed in Figure 2 (r = 0.81–0.88 between alignment and retrieval accuracy on Kinetics-400 and SSv2). The paper's finding that this correlation holds across three different text encoders (Gemma 2-9b-it, Llama-7b, Gemma-7b) suggests the ranking is reasonably stable. The practical benefit is avoiding the cost of human relevance labeling for retrieval evaluation—a significant saving for large-scale or frequently-updated retrieval systems. However, the reliability of this approach depends on whether the alignment-to-retrieval correlation holds for the specific video domain and query distribution in deployment, which the paper's single-dataset (VATEX) validation cannot guarantee.