Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum

ArXiv: 2510.27571

🎯 Pitch

Popular video retrieval benchmarks like MSRVTT are nearly worthless for predicting real-world generalization, yet partially relevant retrieval—a dominant but ignored scenario—shows a 0.97 correlation with overall ability. This paper escapes the narrow benchmark trap by co-designing a 16-dataset diagnostic suite, a 1.55M-pair synthetic data pipeline, and a curriculum that trains a single embedder to beat specialized models across all tasks.

---

1. Executive Summary

This paper introduces a framework for Universal Video Retrieval (UVR) — retrieval spanning multi-task, cross-domain queries — built on the co-design of evaluation, data synthesis, and training. The authors construct the Universal Video Retrieval Benchmark (UVRB) — a suite of 16 datasets diagnosing nine distinct abilities across textual, composed, and visual query formats — and use its diagnostics to drive V-SynFlow, a multi-stage synthesis pipeline that generates 1.55 million high-quality training pairs (the Universal Video Retrieval Dataset (UVRD)) covering spatial, temporal, partially relevant, and long-context retrieval. They then train the General Video Embedder (GVE) using the Modality Pyramid curriculum — a training schedule that progresses from atomic tasks to composite ones by dynamically estimating task alignment and adjusting sampling probabilities (e.g., prioritizing well-aligned tasks early via an annealing temperature before incorporating harder ones). GVE-7B achieves state-of-the-art zero-shot generalization, outperforming the next-best model (Unite-7B) by +5.4% absolute on the dataset-averaged metric, while the 3B-parameter GVE-3B surpasses the 8.3B-parameter Unite-7B. Analysis reveals that popular benchmarks like MSRVTT are poor predictors of universal retrieval ability and that partially relevant video retrieval — despite low research attention — achieves the highest average correlation (ρ = 0.97) with overall performance, establishing that general retrieval capability is better assessed through fuzzy-query scenarios than through conventional coarse-grained tasks.

2. Context and Motivation

The Fundamental Problem: Video Retrieval Models Are Systematic Specialists, Not Generalists

The core problem this paper addresses is a structural misalignment in how video retrieval is evaluated, trained, and measured. The field has drifted into a cycle where narrow benchmarks incentivize narrow data curation, which in turn produces models that excel only within constrained settings — suppressing the development of genuinely universal retrieval capability. The authors frame this as a self-reinforcing trap (Section 1):

"narrow benchmarks incentivize correspondingly limited data and single-task training. Therefore, universal capability is suppressed due to the absence of a diagnostic evaluation that defines and demands multi-dimensional generalization."

To break this cycle requires simultaneous intervention at all three levels: evaluation must be redefined to measure the right things, data must be constructed to cover the semantic space those measurements reveal, and training must be designed to exploit the structure within that diverse data. The paper's tripartite framework — UVRB (evaluation), UVRD (data), and GVE with Modality Pyramid (modeling) — is a direct response to this systemic gap.

This matters because video retrieval underpins critical real-world systems: search engines, content recommendation, video archiving, surveillance analysis, and creative tools. A user searching for "the scene where the astronaut enters the wormhole" needs fine-grained spatio-temporal understanding; another searching with an image of an actor plus the text "find the sci-fi film they appeared in" needs compositional reasoning across modalities. Current models — even large ones — fail on these tasks because they were never systematically trained or evaluated for them. The gap between benchmark performance and real-world utility is not a minor inconvenience; it is a fundamental obstacle to deploying video retrieval systems in open-domain settings.

Three Interconnected Gaps in the Current Paradigm

The paper identifies three specific technical gaps that, together, prevent progress toward universal video retrieval. These are not independent problems — fixing any one in isolation is insufficient because evaluation shapes data, data shapes training, and training determines what models learn.

Gap 1: No Diagnostic Framework Exists for Measuring Multi-Dimensional Generalization

Existing video retrieval benchmarks are siloed within narrow task × domain cells. MSRVTT Xu et al. (2016) and DiDeMo Anne Hendricks et al. (2017) test coarse-grained text-to-video matching with short (∼10–50 second) clips and brief (∼9–30 word) captions. They reveal almost nothing about a model's ability to handle fine-grained spatial queries ("the worker in the gray sleeveless shirt"), temporal reasoning ("the camera pans left, then zooms in"), partially relevant matches (theme-to-clip retrieval where only abstract or local information aligns), or long-context retrieval in hour-plus videos. The paper's diagnostic analysis (Section 4.3, Finding 1) quantifies this: MSRVTT shows a Pearson correlation of only ρ = 0.58 with average universal retrieval performance — it is a demonstrably poor proxy for general ability.

More specialized benchmarks have emerged — CaReBench Xu et al. (2025) for fine-grained spatial and temporal captioning/retrieval, CameraBench Lin et al. (2025) for camera motion understanding, LoVR Cai et al. (2025) for long-video retrieval and thematic matching, PE-Video Bolya et al. (2025) for keyword-based retrieval, and MomentSeeker Yuan et al. (2025) and CoVR Ventura et al. (2024) for composed retrieval — but each exists in isolation. No prior benchmark combines them into a unified evaluation suite where models can be compared across all dimensions simultaneously. Without such a diagnostic, it is impossible to identify which specific capabilities each model lacks or to understand trade-offs between capabilities (e.g., does improving spatial understanding necessarily come at the cost of temporal reasoning?). The paper's UVRB fills this gap by assembling 16 datasets targeting nine distinct abilities (three query formats × six data domains) into a single evaluation framework.

Gap 2: Training Data Is Either Too Small and Biased or Too Noisy to Cover the Universal Retrieval Space

A universal video embedder needs training data that spans diverse query formats (text, image, video, and their compositions) and data domains (coarse-grained through long-context). Existing datasets fail on both axes:

• Web-crawled datasets like WebVid Bain et al. (2021) and InternVid-FLT Wang et al. (2023) are massive (millions of pairs) but consist of weakly aligned, noisy text-video pairs. The captions are often auto-generated, non-descriptive, or semantically misaligned with the video content. Training on such data teaches the model to match coarse, surface-level semantics but provides no signal for fine-grained spatial reasoning, temporal dynamics, or compositional understanding. The narrow semantic distribution — dominated by simple descriptive text — cannot produce embeddings that generalize to complex queries.

• Specialized datasets like CaReBench, VDC Chai et al. (2024), and DREAM-1K Wang et al. (2024b) provide high-quality, fine-grained annotations but are too small (typically 1,000–10,000 examples) to serve as the primary training resource for a large model. Their narrow scope also means they cover only a fraction of the semantic space needed for universality. Training on any single such dataset produces a specialist, not a generalist.

• Human annotation at scale is prohibitively expensive. Collecting a new, large-scale dataset with detailed spatial, temporal, and compositional annotations across diverse videos is economically infeasible for most research groups.

The paper positions its V-SynFlow pipeline as a scalable middle path: use MLLMs as conditional generative engines to enrich the semantic content of existing web-crawled video data, transforming noisy, simple captions into high-quality, multi-style, structurally diverse annotations without human annotation. This produces 1.55 million training pairs spanning all the capability dimensions identified by UVRB's diagnostics. Critically, because the synthesis is conditioned on the actual visual content of each video (via frame-level MLLM analysis), the resulting captions are grounded rather than hallucinated — a key quality control that distinguishes V-SynFlow from naive text-generation approaches.

Gap 3: Multi-Task Training Ignores the Hierarchical Structure of Retrieval Abilities

Even with diverse, high-quality training data, naive multi-task training fails. The paper's ablation (Table 3, GVE-s vs. GVE) shows that simply mixing all tasks into a uniform training schedule leaves performance on the table — the Modality Pyramid curriculum adds 1.8–3.1% absolute improvement.

The underlying reason is that retrieval abilities are not independent or equally difficult. Foundational abilities — coarse-grained text-video matching, spatial object recognition, basic visual similarity — serve as building blocks for higher-order skills: temporal reasoning (which requires recognizing that objects have changed state or position), composed retrieval (which requires binding a visual reference to a text query), and long-context understanding (which requires aggregating information across extended temporal windows). The paper explicitly frames this as a knowledge hierarchy (Section 3.3):

"Foundational abilities such as spatial perception (e.g., object recognition) serve as building blocks for higher-order temporal reasoning (e.g., action recognition). This principle is evidenced by the remarkable success of image-only trained models like GME Zhang et al. (2025) on video retrieval tasks (see Figure 2), highlighting latent cross-task adaptability."

The evidence for transfer is striking: GME-7B, trained exclusively on text-image and image-image pairs (no video data whatsoever), achieves strong performance on video retrieval tasks in UVRB — outperforming several video-trained models on long-context and fine-grained retrieval. This demonstrates that spatial understanding transfers to video, but also implies that training regimes should explicitly leverage this transfer structure rather than treating all tasks as independent.

Existing multi-task approaches — which sample tasks uniformly or with fixed weights — fail to exploit this hierarchy. Easy tasks dominate early optimization, consuming gradient budget, while harder, structurally dependent tasks receive insufficient signal early in training and converge poorly. The Modality Pyramid addresses this by scheduling tasks in a bottom-up order: well-aligned, foundational tasks are sampled with higher probability early in training (when the temperature σ(t) is low and sampling is sharply peaked), and harder, composite tasks are progressively introduced as the temperature increases. This is not merely a heuristic — it is a principled attempt to structure the learning process to respect the inherent dependencies between capabilities.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior work across three categories:

CLIP-based video models (CLIP4Clip, ViCLIP, VideoCLIP-XL, LanguageBind, InternVideo2). These models extend CLIP's image-text alignment to video by adding temporal modules (e.g., temporal attention, 3D convolutions) to process frame sequences. Their fundamental limitation is inherited from CLIP's weak language understanding: the text encoder is a shallow transformer trained on short, descriptive captions, which cannot parse complex, long, or compositional queries. The paper's correlation analysis (Section 4.3, Finding 3) reveals a stark asymmetry: CLIP-based models are spatially biased (ρ_S-CG = 0.99 — spatial understanding strongly predicts coarse-grained performance) but temporally weak (ρ_T-CG = 0.46 — temporal understanding is nearly uncorrelated with coarse-grained success). Spatial and temporal representations are decoupled (ρ_S-T = 0.12), meaning these models cannot jointly reason about where and when. Compositional retrieval (CMP) is inversely correlated with visual accuracy (ρ_CMP-VIS = −0.71), indicating that improving visual fidelity in CLIP-based models comes at the cost of compositional reasoning ability.

MLLM-based embedding models (GME, Unite, VLM2Vec-V2, BGE-VL, UniME, B3). These models replace CLIP's encoder with a multimodal large language model, gaining stronger language understanding and visual generalization. However, their training data remains narrow. Models like GME-2B and GME-7B are trained on text-image data only — while this transfers surprisingly well to video (supporting the paper's transfer hypothesis), it means they have no explicit training for temporal dynamics, long-context reasoning, or composed video queries. Unite-2B and Unite-7B are trained on text-video data as well, but the video data is the same noisy, coarse-grained web data (WebVid, InternVid) that limits CLIP-based models. The consequence is uneven capability: Unite-7B performs well on visual retrieval (VIS: 0.666) and partially relevant tasks (PR: 0.425) but substantially underperforms GVE-7B on compositional (CMP: 0.254 vs. 0.312) and temporal (T: 0.412 vs. 0.469) retrieval. The model has pockets of strength inherited from its training data distribution but lacks systematic coverage of the universal retrieval space.

Specialized composed retrieval models (EgoCVR, CoVR-2). These models handle the specific task of composed queries (text + image) but are not designed for — and do not generalize to — standard text-to-video, visual, or long-context retrieval. The paper argues that pursuing separate specialized models for each task format is costly and hinders progress toward a single, general-purpose system.

How This Paper Positions Itself

The paper's central thesis is that universality in video retrieval cannot be achieved by incrementally improving any single component — it requires a co-designed ecosystem where evaluation, data, and training are mutually reinforcing. This is a departure from prior work, which typically treats these as independent problems solved by separate research communities.

The benchmarking community focuses on constructing better test sets but does not close the loop to guide data generation or model design. The data synthesis community (MegaPairs Zhou et al. (2024), AuroraCap Chai et al. (2024)) generates large-scale training data but does not validate it against a comprehensive diagnostic of model capabilities. The model training community develops better architectures and contrastive losses but evaluates on standard, narrow benchmarks that cannot reveal capability gaps.

This paper's framework closes all three loops:

1. UVRB diagnoses capability gaps → informing what data dimensions need enrichment.
2. V-SynFlow generates targeted data → populating the semantic space UVRB identifies as deficient.
3. Modality Pyramid trains on this data → leveraging the structure UVRB's diagnostics reveal about task dependencies.
4. The resulting GVE model is evaluated on UVRB → confirming that the co-design worked and revealing new insights (e.g., the dominance of partially relevant retrieval as a universal metric).

This is not merely an engineering integration of existing ideas. The paper argues that each component — UVRB, V-SynFlow, Modality Pyramid — is novel in its own right and additionally that their combination produces insights (Section 4.3's four findings about capability correlations, architecture-dependent biases, and scaling effects) that would be invisible if any component were studied in isolation. The title's emphasis on "co-design" is not rhetorical — it reflects the paper's core methodological contribution: that evaluation, data, and training must be co-optimized to escape the narrow-specialist equilibrium that currently constrains the field.

3. Technical Approach

3.1 Reader Orientation

The system being built is a training pipeline and evaluation framework that produces a single, general-purpose video embedding model — called the General Video Embedder (GVE) — capable of measuring relevance between any query (text, image, video, or combinations thereof) and any video, across a wide range of data domains from coarse-grained matching to fine-grained spatial-temporal reasoning. The problem it solves is that video retrieval systems have historically been trained as narrow specialists (text-to-video only, on short clips with simple captions) and evaluated on benchmarks that cannot diagnose which specific capabilities are missing, creating a cycle where models plateau on real-world tasks without researchers understanding why. The "shape" of the solution is a co-designed triad: a diagnostic benchmark (UVRB) that measures nine distinct retrieval abilities and reveals their inter-correlations → a data synthesis pipeline (V-SynFlow) that uses MLLMs to transform noisy web videos into a 1.55M-pair dataset systematically covering all those abilities → a curriculum learning algorithm (Modality Pyramid) that trains GVE by progressing from foundational tasks to composite ones according to dynamically estimated alignment levels.

3.2 Big-Picture Architecture (Diagram in Words)

The framework has four major components, arranged in a design→execute→evaluate loop:

1. Universal Video Retrieval Benchmark (UVRB) — a suite of 16 test datasets covering three query formats (textual, composed with text+image or text+video, visual-only) and six data domains (coarse-grained, fine-grained-spatial, fine-grained-temporal, partially relevant, long-context, and composed). Its role is diagnostic: rather than reporting a single aggregate score, UVRB measures performance across nine distinct abilities and enables correlation analysis to reveal capability trade-offs. It serves as both the evaluation target and the design spec for what data and training must cover.

2. V-SynFlow Data Synthesis Pipeline — a three-stage process that ingests raw, weakly annotated video-text pairs from web datasets (PVD, InternVid-FLT, WebVid), filters them for quality, enriches their semantic content using MLLM-based captioning with controlled information profiles, and extends them into multi-task retrieval instances (text-to-video, composed, visual). The output is the Universal Video Retrieval Dataset (UVRD): 1.55 million training pairs with rich spatial-temporal details, diverse descriptive styles, and distinct task formats.

3. General Video Embedder (GVE) — an MLLM-based encoder derived from Qwen2.5-VL that fuses arbitrary combinations of text, image, and video inputs into a joint token sequence, processes them through a frozen vision encoder and LoRA-fine-tuned LLM backbone, and extracts the final-layer EOS token embedding as a fixed-length vector for cosine-similarity retrieval. This is the model that gets deployed and evaluated.

4. Modality Pyramid Curriculum — a training scheduler that, before each epoch, estimates how well-aligned each task's data is (using a prober model — either GME-7B at epoch 1 or the previous epoch's GVE checkpoint thereafter), then samples tasks with probability proportional to a softmax over those alignment scores at a temperature that anneals from low (focused on easy, well-aligned tasks) to high (progressively incorporating harder tasks). The loss function is a symmetric InfoNCE with in-batch negatives and externally mined hard negatives.

Information flows as follows: UVRB diagnostics identify capability gaps (Section 4.3 findings) → these gaps inform what types of data V-SynFlow should synthesize (spatial, temporal, partially relevant, composed) → UVRD provides the training mixture → Modality Pyramid schedules task exposure during GVE training → GVE is evaluated on UVRB, which validates the framework and surfaces further insights.

3.3 Roadmap for the Deep Dive

• First, the UVRB benchmark — what abilities it defines, how datasets are constructed to isolate those abilities, and what the diagnostic evaluation protocol looks like. This is the foundation because it defines what universality means and therefore what the rest of the system must achieve.
• Second, the V-SynFlow data synthesis pipeline — its three stages (quality control, information enrichment, task extension) and how each stage uses MLLMs as conditional generators while controlling for quality. Understanding this is essential because the synthesized data is what enables training for universality in a scalable way.
• Third, the GVE model architecture — how multimodal inputs are fused, how embeddings are extracted, and what design choices enable efficient fine-tuning of a large pretrained MLLM. This matters because the architectural decisions (last-token pooling, LoRA targeting, frozen vision backbone) determine which capabilities transfer from pretraining and which are learned during fine-tuning.
• Fourth, the Modality Pyramid curriculum — the alignment-aware dynamic scheduling mechanism, the prober model, the annealing temperature, and the unified contrastive loss with hard negatives. This is the paper's core training innovation and requires understanding both the probabilistic sampling formula and the loss function.
• Fifth, the training infrastructure — hyperparameters, LoRA configuration, data mixture ratios, optimizer settings, and distributed training strategy. These practical details matter for reproducibility and for understanding the scale of the training operation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-and-methods paper whose core idea is that universal video retrieval requires co-designed evaluation, data synthesis, and training — and that each component must be designed with explicit awareness of the structure and dependencies among retrieval abilities. The paper contributes a benchmark, a dataset, a curriculum, and a model; the technical depth lies in how each is constructed and how they interconnect.

---

Universal Video Retrieval Benchmark (UVRB): Defining and Diagnosing Universality

Motivation and design philosophy. UVRB is not merely a collection of existing datasets — it is a deliberately constructed diagnostic instrument. The authors define a video retrieval ability as proficiency in one query format or one data domain (e.g., "VIS" for visual queries, "T" for temporal reasoning), and a universal embedder as one that masters multiple abilities and their combinations. The benchmark's architecture reflects this: 16 test datasets are selected and adapted so that each primarily tests a specific cell in the ability matrix, making it possible to measure not just average performance but correlations, trade-offs, and gaps between abilities.

The ability taxonomy has two axes:

• Query format (three values): Textual (TXT) — natural language queries; Composed (CMP) — text + image or text + video queries; Visual (VIS) — image-only or video-only queries.
• Data domain (six values): Coarse-grained (CG) — high-level semantic matching; Fine-grained-Spatial (S) — object appearance, spatial relationships; Fine-grained-Temporal (T) — event dynamics, motion, camera movement; Partially Relevant (PR) — matching based on local, abstract, or thematic information rather than full semantic alignment; Long-context (LC) — retrieval within extended videos (minutes to hours) with long-form queries; plus composed tasks which combine query formats with fine-grained domains.

A model's capability profile is a vector of scores across these nine abilities (three query formats × six domains, with some combinations naturally co-occurring — e.g., LC is always textual, VIS is always coarse-to-fine-grained).

Dataset selection and adaptation. The 16 datasets are drawn from existing benchmarks but repurposed to isolate specific abilities. The construction strategy for each dataset is described in Appendix A.2. Key examples that illustrate the diagnostic logic:

• MSRVTT (Xu et al., 2016) — the canonical text-to-video retrieval benchmark. Uses 1,000 clip-text pairs from the JSFusion split (Yu et al., 2018). Queries are short (~9.4 words on average) descriptive captions; videos are ~15-second clips. This tests coarse-grained text-to-video matching — the simplest, most extensively studied ability.

• CRB-S (CaReBench-Spatial) and CRB-T (CaReBench-Temporal) (Xu et al., 2025) — taken from the same underlying video set but using different caption fields. CRB-S uses the spatial_caption field (~115 words, focused on object locations, appearances, and spatial relationships), while CRB-T uses the temporal_caption field (~103 words, focused on action sequences and temporal progression). By keeping videos identical and varying only the text domain, the benchmark isolates whether performance differences stem from the model's spatial vs. temporal understanding rather than video content.

• CMRB (CameraBench) (Lin et al., 2025) — queries are detailed camera motion descriptions (e.g., "The camera smoothly dollies forward, maintaining a steady and fluid motion..."), testing whether the model can recognize how a video was shot as a retrieval signal. This is a pure temporal-domain ability orthogonal to object recognition.

• DREAM-E (DREAM-1K-Event) (Wang et al., 2024b) — queries are event descriptions like "Wooden trap launches purple squirrel into the air" matched against 1,000 videos. The queries are short (~6.5 words) and describe specific actions, but the evaluation uses 6,251 queries against only 1,000 videos (many-to-few), making this a partially relevant retrieval task: multiple distinct events map to the same video.

• LoVR-TH (LoVR-Theme2Clip) (Cai et al., 2025) — queries are theme annotations of video clips (~48 words describing visual style, mood, or narrative tone — e.g., "The overall style of the animation is vibrant and whimsical..."), matched against 8,854 clips. The query-video relationship is thematic rather than literal, making this a partially relevant retrieval task where surface-level keyword matching fails.

• PEV-K (PE-Video-Keyword) (Bolya et al., 2025) — queries are keyword lists (~45 words, e.g., "colorful, paper, beautiful...") matched against 15,000 videos. This tests whether the model can retrieve videos matching abstract, multi-concept keyword combinations without full sentence structure — a partially relevant scenario distinct from theme-based matching.

• LoVR-V (LoVR-Text2Video) (Cai et al., 2025) — long-text to long-video retrieval. Videos average 1,560 seconds (26 minutes); queries average 17,365 words (essentially full-length video descriptions). The corpus is 467 videos with 100 queries — a small but extremely challenging test of long-context understanding.

• VDC-D (VDC-Detail) (Chai et al., 2024) — detailed captions (~508 words) matched against ~30-second videos. Unlike LoVR-V which stresses temporal extent, VDC-D stresses descriptive detail and fine-grained understanding at moderate video lengths.

• MS-TI and MS-TV (adapted from MomentSeeker; Yuan et al., 2025) — composed retrieval tasks. MS-TI uses text + image queries to retrieve a target video clip from 10 candidates; MS-TV uses text + video queries. The adaptation follows the method in Meng et al. (2025). Both use Precision@1 as the metric because multiple candidates can be correct.

• MSRVTT-I2V and LoVR-C2V — visual-only retrieval. MSRVTT-I2V uses a single randomly sampled frame from each MSRVTT test video as the query to retrieve the source video (1,000 pairs). LoVR-C2V uses one short clip from each LoVR long video as the query to retrieve the full video (467 pairs). These test whether the model can match based purely on visual content without any text signal.

Dataset statistics. Table 4 (Appendix A.2) provides the quantitative profile: query counts range from 100 (LoVR-V) to 14,427 (PEV-K); corpus sizes range from 10 (MS-TI, MS-TV — deliberately small to test fine-grained discrimination) to 15,000 (PEV-K); video durations range from 5.7 seconds (CMRB, camera motion clips) to 1,560 seconds (LoVR-V); query word counts range from 6.5 (DREAM-E) to 17,365 (LoVR-V). This systematic variation in scale, difficulty, and domain is what makes UVRB a diagnostic instrument rather than a monolithic benchmark.

Evaluation protocol. All models are evaluated under a controlled, standardized pipeline (Section 4.1, Appendix A.3) to ensure fairness. Key protocol decisions:

• Zero-shot evaluation: No model is fine-tuned on UVRB training data. Models that may have seen in-domain data during their original training (e.g., MSRVTT for Unite-7B) are evaluated as-is — this is noted as a potential unfair advantage for baselines, making GVE's superior performance more significant.

• Uniform video preprocessing: Every video is sampled into exactly 8 frames, uniformly spaced. Audio, speech, and metadata are excluded — only raw visual frames are used. This is a deliberate constraint that forces models to rely on visual understanding rather than multimodal shortcuts.

• Input token constraints: CLIP-based models operate at 224×224 resolution. MLLM-based models have their frame resolution adaptively adjusted so that each frame encodes to fewer than 200 visual tokens. Total input sequence length is capped at 8192 tokens, with truncation beyond this limit.

• Models without native video support (e.g., BGE-VL, which was designed for images) are adapted by treating each frame as an independent image input and inserting multiple special tokens — a multi-image embedding pipeline. This adaptation is noted but represents a fair attempt to evaluate the model's inherent visual understanding transferred to video.

• Embedding extraction: All output embeddings are L2-normalized. For models with learnable relevance estimation modules (e.g., the MLP head in InternVideo2), these are removed and the last hidden state before the head is used as the embedding. The relevance metric is always cosine similarity — no post-processing, re-ranking, or learned scoring functions are applied.

• Metrics: Recall@1 (R@1) is the primary metric. For datasets with inherently fuzzy queries where exact match is inappropriate, Recall@10 (R@10) is used (CMRB, LoVR-TH). For composed tasks with multiple positive candidates (MS-TI, MS-TV), Precision@1 (P@1) is used.

• Instruction prompts: Each dataset receives a task-specific instruction prompt injected into the query (Table 5, Appendix A.2). For example, MSRVTT queries are prefixed with "Find the clip that corresponds to the described scene in the given video." while CRB-S queries use "Find the video according to the spatial description." These prompts are standardized across all evaluated models and mirror real-world retrieval interfaces.

Why this design: The diagnostic architecture — nine separate ability scores rather than a single aggregate — enables the correlation analysis in Section 4.3 (Findings 1-4) that reveals which benchmarks are poor proxies for general ability, how CLIP-based and MLLM-based models develop capabilities differently, and which abilities are interdependent vs. independent. Without this multi-dimensional structure, the paper's key insights about partially relevant retrieval and spatial-temporal decoupling would be invisible. The standardized evaluation protocol ensures that differences between models reflect genuine capability differences rather than evaluation artifacts (different frame counts, different resolution, different scoring functions).

---

V-SynFlow: Three-Stage Data Synthesis Pipeline

V-SynFlow transforms raw, weakly annotated web videos into a structured, high-quality, multi-task training dataset (UVRD). The pipeline operates in three sequential stages, each building on the output of the previous one. The design is motivated by a pragmatic constraint: web-crawled video-text data is abundant (millions of pairs) but noisy and narrow in semantic coverage, while human annotation at the needed scale and quality is prohibitively expensive. The solution is to use MLLMs as conditional generative engines that enrich the data while being grounded in the actual visual content of each video — the MLLM sees the video frames, so its generated captions are descriptions of what is actually present rather than hallucinations.

Stage 1: Multi-Granular Quality Control

The input is a raw corpus D = {(v_i, t_i)} — video-text pairs from PVD (Bolya et al., 2025), InternVid-FLT (Wang et al., 2023), and WebVid (Bain et al., 2021). These datasets are large (WebVid alone provides over 10M pairs after filtering; InternVid-FLT contributes ~1.7M; PVD contributes ~104K) but the text annotations are often auto-generated, non-descriptive, or misaligned with the visual content.

The quality control stage applies three sequential filters to produce a clean asset pool A_tfc:

• Annotation Rectification: Remove captions that are non-descriptive — for example, captions that are URLs, file names, or entirely generic phrases that provide no semantic signal about the video content. The paper does not specify the exact filtering rules (they are presumably heuristic and language-dependent), but the goal is to eliminate text that would provide no useful training signal.

• Cross-Modal Consistency Filtering: For each remaining pair, compute the cosine similarity between the text embedding and the video embedding using a pretrained multimodal embedder Φ(·) (specifically GME-7B (Zhang et al., 2025), a strong off-the-shelf model trained on text-image pairs). Pairs where this similarity falls below a threshold are discarded. This is the key quality filter: it removes videos where the associated text does not semantically match the visual content — the most common failure mode in web-crawled data where captions may describe a different scene, be in the wrong language, or reference metadata rather than visual content. The paper does not specify the exact threshold value, but the principle is clear: only keep pairs where a strong pretrained model confirms semantic alignment.

• Temporal Dynamics Filtering: Remove videos with static content — those where consecutive frames are nearly identical, indicating a still image rather than a dynamic scene. Static videos are useless for training temporal reasoning (there are no object motions, camera movements, or event sequences to describe) and would dilute the training signal for fine-grained temporal tasks. The filtering criterion is not specified in detail but likely involves frame-to-frame similarity thresholds.

The output asset pool A_tfc contains: validated videos {v_j}, their original (or rectified) captions {t_j}, extracted key frames {f_jk} (individual representative frames from each video), and cropped clips {c_jl} (short temporal segments). These components — full videos, individual frames, and short clips — provide the raw material for the subsequent enrichment and task extension stages.

Why this stage is necessary: Without quality control, the subsequent enrichment stage would amplify noise — an MLLM asked to generate detailed captions for a video with a misaligned text annotation might either hallucinate to match the bad text or produce captions that conflict with the visual content. The cross-modal consistency filter ensures that the MLLM has a semantically coherent starting point. The temporal dynamics filter ensures that training data for temporal reasoning actually contains temporal content.

Stage 2: Multi-Dimensional Information Enrichment

Given the clean asset pool A_tfc, this stage enriches the semantic coverage of the text annotations. The core mechanism is controlled conditional captioning: an MLLM (specifically Keye-VL-8B (Team et al., 2025), a multimodal model with 32K context window) generates new captions for each video, but the generation is steered by randomly sampled information profiles that specify which semantic dimensions to emphasize.

For each video v_j, the MLLM generates multiple captions {t'_jk} (typically 5 distinct captions per video, as shown in the synthetic captioning prompt in Appendix A.6). Each caption is conditioned on a randomly generated profile that specifies the desired mix of:

• Spatial details (30% of profiles): The caption should emphasize object locations, spatial relationships, visual attributes, scene layout.
• Temporal details (60% of profiles): The caption should emphasize action sequences, motion patterns, event progression, camera movements.
• Theme/style/other (10% of profiles): The caption should emphasize emotional tone, narrative style, visual aesthetics, or partially relevant information such as theme or mood.

The 60/30/10 split is deliberately skewed toward temporal content because temporal understanding is the capability most lacking in existing models (as UVRB diagnostics would later confirm — see Section 4.3, Finding 2: CLIP-based models have near-zero temporal-spatial correlation, and GME models trained only on static images lack explicit temporal training). The information profile mechanism gives V-SynFlow fine-grained control over the semantic distribution of the training data in a way that would be impossible with human annotation at scale — the authors can adjust the profile distribution to target specific capability gaps identified by UVRB.

The synthetic captioning prompt (Appendix A.6) imposes additional quality and diversity constraints beyond the information profile:

• Style diversity: Each of the 5 captions per video must use a different sentence style — concise summary, spatial-temporal rich description, abstract understanding, keywords-only, and partially relevant information. This ensures the training data includes varied query formats, not just standard descriptive sentences.
• Length diversity: At least one caption must be under 20 words; at least one must exceed 100 words. This ensures the model encounters both brief keyword-style queries and long detailed descriptions.
• Factual grounding: The prompt explicitly instructs: "Do not invent fictional elements, dialogue, or backstory not visible in the video" and "Focus exclusively on visible content; do not mention absences or speculate about unseen elements." This is critical for preventing hallucination — the MLLM is being used as a describer, not a storyteller.
• Readability and education level: These are dynamically instantiated parameters that control linguistic complexity (e.g., varying between simple and academic language) — another axis of diversity.

The output of this stage is an enriched text-video dataset D+ where each video is associated with multiple high-quality, style-diverse captions. Additionally, the stage constructs visual pair collections: frame-to-video pairs P_{f↔v} = {(f_jk, v_j)} (each key frame mapped to its source video) and clip-to-video pairs P_{c↔v} = {(c_jl, v_j)} (each short clip mapped to its source video). These visual pairs are the raw material for the task extension stage.

Why controlled conditional captioning works: The MLLM sees the actual video frames, so its generated captions are grounded in visual reality rather than hallucinated. The information profiles prevent the MLLM from defaulting to its most common captioning style (which might be biased toward spatial descriptions or simple summaries). The explicit diversity constraints (style, length, readability) prevent the training data from being homogenized — a common failure mode in synthetic data generation where all examples end up with similar structure and vocabulary because they come from the same generator.

Stage 3: Multimodal Task Extension

The final stage assembles the unified training corpus D* (which becomes UVRD) by extending the enriched data into multiple retrieval tasks beyond simple text-to-video matching.

Composed retrieval data generation: For each visual pair from P_{f↔v} and P_{c↔v} — that is, for each (frame, video) and (clip, video) pair — the MLLM Keye-VL-8B generates a query text t_{fjk→v_j} that describes the temporal evolution from the static reference (frame or clip) to the full video. The structured prompts for text-image composed and text-video composed retrieval (Appendix A.6) enforce specific quality criteria:

• Combination necessity: "The query MUST become meaningless or significantly less specific if either the image or video is removed." This ensures the generated query genuinely requires both modalities — it is not a standalone text query that happens to mention an image. For example, "the person FROM REFERENCE IMAGE wearing red jacket now skiing" becomes "a person skiing" if the image is removed, losing the identity-binding constraint.

• Binding precision: The query must use explicit binding phrases like "FROM REFERENCE IMAGE" or "AS IN REFERENCE" to link the visual reference to the target. Generic connectors like "similar to" are prohibited.

• Specific visual attribute reference: At least one concrete visual feature from the reference (e.g., "red jacket," "ponytail," "blue hat") must be incorporated into the query. This ensures the model learns to extract and re-identify visual attributes across modalities.

• Logical coherence: The prompt explicitly checks for subject-verb-object consistency and prohibits semantic contradictions (e.g., "the jacket is running" where an inanimate object is the subject of an action verb).

• Relationship type classification: For text-video composed retrieval, the prompt distinguishes four relationship types between the reference clip and target video: temporal continuation (reference is earlier part of same sequence), perspective variation (same action from different angle), quality variation (same scene in different lighting/resolution), and minor action variation (same person doing a slightly different action). This categorization ensures diversity in the types of composed relationships the model encounters.

The resulting composed instances have the form ((t_{fjk→v_j}, f_jk), v_j) for text-image queries and ((t_{cjl→v_j}, c_jl), v_j) for text-video queries — a tuple of (query modalities, target video).

Basic alignment data generation: The unselected synthesized captions from Stage 2 (each video generated 5 captions, but only one is used for the primary text-video pair) are repurposed to create text-to-text pairs: any two captions describing the same video are paired, forming a text-to-text retrieval instance where the query and target are both textual descriptions of the same visual content. Text-image pairs are sampled from A_tfc using frames as targets. These basic alignment tasks — text-to-text, text-to-image — ensure the training mixture includes the foundational cross-modal alignments that underpin more complex tasks.

Dataset composition (Table 7, Appendix A.4). The final UVRD dataset comprises 1.55 million pairs across seven task formats, with sample sizes identical for both the 3B and 7B model variants:

The dominant task is text-to-video (879K pairs, ~57% of UVRD), reflecting the natural abundance of this format in the source data and its importance as the primary retrieval paradigm. Composed tasks (TI2V + TV2V) account for 124K pairs (8%) — smaller but critical for teaching the model to handle multimodal queries. Visual-only tasks (I2V + V2V) account for 236K pairs (15%), providing the signal for image-to-video and video-to-video matching without text intermediaries.

Hard negative configuration: Tasks that use hard negatives (T2V, TI2V, TV2V — 1 hard negative per positive) involve pre-mining negative examples that are superficially similar to the positive but actually irrelevant. The paper does not specify the mining procedure in detail (it is handled by the training infrastructure), but the principle is standard: for each query-positive pair, retrieve the top-K most similar videos from a large corpus using the prober model, and treat the highest-scoring non-positive as a hard negative. Hard negatives are stored with the dataset and loaded during training.

Why the task extension stage is necessary: Without it, the training data would be dominated by text-to-video pairs — training on UVRD-T2V alone would produce a model that excels at standard text-to-video retrieval but fails on composed queries, visual queries, and text-to-text alignment. The task extension stage deliberately constructs minority-task instances to ensure the model receives gradient signal for all the abilities UVRB evaluates. The composed retrieval data is particularly important because it cannot be obtained from any existing dataset at scale — the MLLM-based generation is the only practical way to create training data where queries genuinely require cross-modal binding.

---

General Video Embedder (GVE) Architecture

GVE is architecturally derived from Qwen2.5-VL (Bai et al., 2025), a pretrained multimodal large language model, but repurposed as a fixed-length embedding encoder rather than a text generator. The key architectural decisions are: (1) keep the vision encoder frozen to preserve pretrained visual features, (2) apply LoRA to specific LLM attention and feed-forward modules to enable efficient fine-tuning, and (3) extract embeddings from the final-layer EOS token to produce a single vector representing the entire multimodal input.

Input fusion. The model accepts three input modalities — text, image, and video — and fuses them into a single token sequence. The process works as follows:

1. Text tokenization: Raw text (which may include instruction prompts) is tokenized into a sequence of token IDs X_t ∈ R^{B × T_t}, where B is the batch size and T_t is the number of text tokens. The tokenizer is Qwen2.5-VL's native vocabulary.

2. Visual encoding: Images and video frames are processed by the frozen vision encoder. For a single image, the output is a sequence of visual tokens X_v ∈ R^{B × T_v × d}, where T_v = THW / p_s^2 — the number of patches at patch size p_s = 2 given the image height H and width W. For a video with K uniformly sampled frames, the visual token count is T_v = K · (THW / p_s^2), with absolute temporal position encodings added to each frame's tokens. Critically, the number of frames K must satisfy K mod p_t = 0 where p_t = 2 is the temporal patch size — this maintains alignment with the vision encoder's 3D spatiotemporal grid that processes video as a volume of patches across time and space.

3. Scatter-based fusion: The visual tokens are injected into the text sequence by replacing placeholder tokens (<image>, <video>). The fused input X_fused ∈ R^{B × T × d}, where T = T_t + Σ T_v — the total sequence length is the sum of text tokens and all visual tokens. This scatter-based insertion preserves positional coherence: visual tokens occupy contiguous spans within the sequence, separated by text tokens, enabling the model to attend across modalities at arbitrary positions.

4. Sequence length handling: The total input length is capped at 8192 tokens. For very long videos (e.g., LoVR-V with 26-minute videos at 8 frames) or very long text queries (LoVR-V queries average 17,365 words), truncation is applied. MLLM-based models have their frame resolution adaptively reduced to ensure each frame encodes to fewer than 200 visual tokens — this is a practical constraint to keep total sequence lengths manageable during batch inference.

Embedding extraction. After the fused sequence passes through the LLM backbone (described below), the final hidden states H ∈ R^{B × T × d} are produced. The embedding for each instance is extracted via last-token pooling:

$e^{(i)} = \frac{h_{p_i}^{(i)}}{||h_{p_i}^{(i)}||_2}, \quad \text{where} \quad p_i = \max\{j \mid M_j^{(i)} = 1\}$

where e^{(i)} is the L2-normalized embedding for the i-th instance, h_{p_i}^{(i)} is the hidden state at position p_i, M^{(i)} is the attention mask for instance i (with M_j^{(i)} = 1 indicating non-padded tokens), and p_i is the index of the last non-padded token — in practice, the EOS token position. The ||·||_2 indicates L2 normalization, projecting the embedding onto the unit hypersphere.

What it computes: For each instance in the batch, the model identifies the last non-masked token position (the EOS token in left-padded sequences or the final non-pad token in right-padded ones), extracts the hidden state vector at that position from the final transformer layer, and normalizes it to unit length. The result is a d-dimensional vector (where d is the LLM's hidden dimension — 2048 for the 3B model, 3584 for the 7B model) that represents the entire multimodal input as a single point on the unit hypersphere.

Why last-token pooling: In instruction-tuned MLLMs, the final token (typically EOS) is trained to encapsulate the model's "response intent" — after processing the entire input sequence through causal attention, the last token can attend to all previous tokens and thus aggregate information from the entire input. This is analogous to using the [CLS] token in BERT-style models or taking a weighted average of all token embeddings, but with the advantage that the LLM's pretraining has already optimized the last token to be a semantically meaningful summary. Alternatives like mean pooling over all tokens would dilute the embedding with padding tokens or visual patch tokens that contain low-level features rather than high-level semantics. First-token pooling would not have access to the full sequence context in a causal model.

LLM backbone and fine-tuning strategy. The LLM component of Qwen2.5-VL is fine-tuned using Low-Rank Adaptation (LoRA) (Hu et al., 2021), while the vision encoder and the base token embedding layer are kept frozen.

LoRA configuration (Appendix A.8):

• Rank r = 16
• Alpha α = 32 (the scaling factor applied to the LoRA output)
• Dropout rate 0.1 on the LoRA adapters
• Target modules: q_proj, v_proj, k_proj, up_proj, down_proj, gate_proj — this covers all attention query/key/value projections and all feed-forward network projections. The choice to target both attention and FFN modules means LoRA adapts the full transformer computation, not just the attention mechanism.
• Frozen components: The entire visual backbone (vision encoder) and the base token embedding layer. The vision encoder's pretrained features are preserved as-is; only the LLM's processing of those features is adapted.

Why LoRA over full fine-tuning: Full fine-tuning of an 8.3B-parameter MLLM would require storing optimizer states for all parameters — roughly 4× the model size in memory for AdamW, or ~33GB for the 7B model in bf16, plus activations. LoRA reduces trainable parameters to the low-rank adapter matrices (order of 10^7 parameters instead of 10^9), making training feasible on 32 A100 GPUs while also providing implicit regularization that helps prevent catastrophic forgetting of the pretrained knowledge. Why freeze the vision encoder: The vision encoder's features (patch-level visual tokens) are the foundation that the LLM reasons over; fine-tuning the vision encoder would risk degrading these features for the sake of the retrieval objective, potentially losing the rich visual representations inherited from Qwen2.5-VL's pretraining on diverse vision-language tasks. Why include FFN modules in LoRA: The feed-forward layers store factual and conceptual knowledge in transformer models; adapting them via LoRA allows the model to learn retrieval-specific transformations (e.g., projecting visual features into a space that aligns with text queries of different styles) while keeping the attention mechanism's ability to route information between modalities intact.

Model variants. Two sizes are trained:

• GVE-3B: Based on Qwen2.5-VL-3B-Instruct. Total parameters: 3.8B (the base model is ~3B visual + language; the additional ~0.8B comes from the full architecture including embeddings and heads). Hidden dimension: 2048. Trained on 12.55M instances across all data categories (Table 8).
• GVE-7B: Based on Qwen2.5-VL-7B-Instruct. Total parameters: 8.3B. Hidden dimension: 3584. Trained on 13.73M instances — the extra 1.18M instances come from upsampling text-only and image-centric datasets to leverage the larger model's capacity (see Table 6, Appendix A.4).

---

Modality Pyramid: Alignment-Aware Dynamic Curriculum

The Modality Pyramid is a training scheduler that addresses a specific failure mode of naive multi-task learning: when tasks of varying difficulty and data abundance are sampled uniformly, easy, data-rich tasks dominate the early gradient budget, while hard, data-scarce tasks receive insufficient optimization signal and converge poorly. The pyramid metaphor reflects the hierarchical structure of retrieval abilities: foundational tasks (coarse-grained text-video matching, image-text alignment, visual similarity) form the base, and composite tasks (composed retrieval, long-context understanding, temporal reasoning) build on top.

Core mechanism: alignment-aware dynamic scheduling. At the beginning of each training epoch t, the system estimates the current alignment level of every task k ∈ K in the training mixture. A task here refers to a specific input-output format (e.g., T→V, TI→V, I→V) — each task type has its own data distribution and difficulty profile. The alignment estimation uses a prober model Ψ_t:

• For t = 1 (first epoch): Ψ_1 is GME-7B (Zhang et al., 2025), a strong off-the-shelf multimodal embedder trained on text-image data. This provides a reference alignment measurement before any task-specific training has occurred.
• For t > 1 (subsequent epochs): Ψ_t is the GVE checkpoint from the end of epoch t - 1. This means the alignment estimation is on-policy with respect to the current training state — the model evaluates how well its own current parameters align the data for each task.

For each task k, the system samples N_p positive pairs (the paper does not specify N_p, but it is likely a small subset — perhaps 100-1000 pairs — to keep the estimation cost low) and computes the task alignment score:

$R_k^{(t)} = \frac{1}{N_p} \sum_{i=1}^{N_p} \cos\left(\Psi_t(x_i), \Psi_t(y_i)\right)$

where R_k^{(t)} is the average cosine similarity between query and target embeddings for task k at epoch t, (x_i, y_i) is the i-th query-target positive pair for task k, Ψ_t is the prober model (GME-7B at t=1, GVE checkpoint at t-1 for t>1), and cos(·,·) is cosine similarity.

What it computes: For each task, take a sample of positive pairs (queries matched to their correct targets), embed both the query and target using the prober model, compute their cosine similarity, and average across the sample. A higher R_k^{(t)} means the model already produces embeddings where queries and targets are close in the embedding space — the task is well-aligned. A lower R_k^{(t)} means the model separates queries from their targets — the task is poorly aligned, indicating it is either too difficult at the current training stage or receiving insufficient optimization.

Why cosine similarity as the alignment metric: Cosine similarity is the retrieval metric at test time — the model's entire purpose is to maximize it for relevant pairs and minimize it for irrelevant ones. Using the same metric for alignment estimation ensures the scheduling optimizes for the quantity that matters downstream. Alternatives like training loss or gradient norm could indicate optimization difficulty but wouldn't directly measure retrieval-quality alignment.

Task sampling distribution. During epoch t, each training batch's task is sampled with probability:

$P^{(t)}(k) \propto \exp\left(R_k^{(t)} / \sigma^{(t)}\right)$

where P^{(t)}(k) is the probability of sampling task k at epoch t, R_k^{(t)} is the alignment score, σ^{(t)} is a temperature parameter that anneals from σ_min = 0.1 to σ_max = 1.0 over the course of training, and ∝ denotes proportionality (the softmax over all tasks, unnormalized in this notation).

What it computes: For each task, exponentiate its alignment score divided by the current temperature, then normalize across all tasks to form a probability distribution. The softmax amplifies differences between well-aligned and poorly-aligned tasks when the temperature is low, and flattens the distribution as the temperature increases. At the start of training (σ = 0.1), even small differences in alignment produce large differences in sampling probability — well-aligned tasks are strongly favored. By the end of training (σ = 1.0), the distribution approaches uniform, and all tasks receive roughly equal gradient budget.

Why exponential sampling with annealing: This implements a progressive curriculum from easy to hard. Early in training, the model is incompetent at all tasks — but some tasks have higher alignment scores because the pretrained LLM happens to produce better embeddings for them (e.g., text-image alignment transfers from Qwen2.5-VL's pretraining; text-video matching with simple captions is easier than composed retrieval). Sampling these tasks more heavily early on ensures the model receives clear, consistent gradient signal that establishes a good embedding space before tackling harder tasks. As training progresses and the temperature increases, harder tasks are progressively introduced, by which point the model has developed foundational representations that make learning those tasks more tractable. This avoids the common multi-task failure where hard tasks produce noisy, high-variance gradients early in training that destabilize optimization. The specific temperature range (0.1 to 1.0) was chosen empirically — the paper does not provide an ablation over temperature schedules, but the principle is standard in curriculum learning.

Why the prober model is updated each epoch: Using GME-7B for all epochs would measure alignment with respect to an external model's representation, not GVE's own. The gap between GME's alignment and GVE's alignment could mean the curriculum would over-prioritize tasks that GME finds easy but GVE finds hard (or vice versa). Updating the prober to the previous epoch's GVE checkpoint makes the alignment estimation on-policy — it reflects what GVE itself has and hasn't learned at the current training stage, enabling the curriculum to adapt to GVE's actual learning trajectory rather than following a fixed schedule.

---

Training Infrastructure and Contrastive Optimization

Data mixture (Tables 6 and 8, Appendix A.4). Beyond UVRD's synthesized data (1.55M pairs), the training mixture includes large-scale publicly available datasets organized into three parts:

• Part 1: Text-only data (380K for 3B, 580K for 7B). Datasets: MSMARCO (T→T, 300K/500K), HotpotQA (T→T, 69K), WebQA (T→T, 11K). These provide pure language alignment signal — teaching the model that semantically equivalent texts should have similar embeddings regardless of phrasing.

• Part 2: Image-centric data (1.82M for 3B, 2.80M for 7B). A large collection including CC3M (T→I and I→T, 300K/500K and 100K/200K), Laion (T→I and I→T, 300K/500K and 200K), ImageNet (I→T, 100K/200K), VL3-Syn7M short and detailed variants (T→I and I→T), plus composed and knowledge-intensive tasks like CIRR (TI→T), OVEN (TI→TI and TI→T), VISTA (TI→I and T→TI), and MSCOCO (I→T). The image-centric data teaches cross-modal alignment between text and static visual content — spatial understanding, object recognition, attribute binding — which transfers to video as demonstrated by GME-7B's strong video performance despite never being trained on video.

• Part 3: Video-centric data (8.80M for both 3B and 7B). Datasets: VAST (T→V, 1.6M), InternVid-FLT (T→V, 1.7M), PE-Video (T→V, 104K), WebVid (T→V, 5.4M after filtering out queries that describe multiple videos to avoid overly generic text). These provide the primary video-text alignment signal at large scale, though the text quality is noisy (WebVid captions are auto-generated; InternVid-FLT captions are from a video captioning model).

The total training instances are 12.55M for GVE-3B and 13.73M for GVE-7B — the difference comes from upsampling certain high-value datasets (text-only and image-centric) to leverage the 7B model's greater capacity.

Unified contrastive loss. All tasks are trained with a symmetric InfoNCE loss (Oord et al., 2018) augmented with explicit hard negatives. For a query-target pair (q, v+):

$L_{q \to y}^{(i)} = -\log \frac{\exp(s_{q_i \to y_i^+} / \tau_l)}{\exp(s_{q_i \to y_i^+} / \tau_l) + \sum_{j \neq i} \exp(s_{q_i \to y_j^+} / \tau_l) + \sum_{y_k^- \in H_i} \exp(s_{q_i \to y_k^-} / \tau_l)}$

where L_{q→y}^{(i)} is the query-to-target directional loss for the i-th positive pair in the batch, s_{q_i→y_i^+} = cos(E_θ(q_i), E_θ(y_i^+)) is the cosine similarity between the query embedding and the positive target embedding, τ_l is a learnable or fixed temperature parameter (the paper uses τ_l = 0.03 — a low temperature that sharpens the softmax distribution and encourages the model to produce highly confident similarity scores), ∑_{j≠i} exp(s_{q_i→y_j^+} / τ_l) sums over all other positives in the batch treated as negatives (in-batch negatives), and ∑_{y_k^- ∈ H_i} exp(s_{q_i→y_k^-} / τ_l) sums over the set H_i of hard negatives mined for this query.

The total loss is the symmetric average:

$L_i = \frac{1}{2}\left(L_{q \to y}^{(i)} + L_{y \to q}^{(i)}\right)$

where L_{y→q}^{(i)} is the same loss computed in the reverse direction (target-to-query).

What it computes: For each positive pair in the batch, the model computes a contrastive loss that pushes the query embedding close to its positive target embedding while pushing it away from (a) all other positives in the batch (treating them as negatives for this query), and (b) a set of pre-mined hard negatives — targets that are superficially similar to the correct target but actually irrelevant. The softmax over similarities means the loss is dominated by the hardest negative (highest similarity among negatives) — the model focuses its learning on the most confusable cases. The symmetric formulation ensures the embedding space is consistent in both directions: q → y and y → q should produce the same rankings.

Why symmetric InfoNCE with hard negatives: Standard InfoNCE with in-batch negatives alone suffers from a well-known limitation: as training progresses, random in-batch negatives become too easy — the model easily separates them from positives, and the loss saturates. Hard negatives (mined from a large corpus using the prober model) remain challenging because they are deliberately selected to be confusable with the positive. The symmetric formulation is standard practice in representation learning — it ensures the embedding space is not biased toward one direction (e.g., text queries having tighter clusters than video targets). The very low temperature (τ_l = 0.03) is notable — lower than typical values of 0.07-0.1 in CLIP-style training. This produces a sharper softmax that strongly penalizes even moderately high similarity to negatives, pushing the model toward near-binary separation between positives and negatives. This aggressive contrastive objective is appropriate for retrieval where the model must discriminate among potentially millions of candidates.

Cross-device negative sharing. To increase the effective batch size of negatives without increasing per-GPU memory, embeddings from all GPUs are gathered (via all-gather communication) and shared. The effective negative pool for each GPU's loss computation includes positives from other GPUs' batches, multiplying the number of in-batch negatives by the number of GPUs (32 in the default configuration). The total effective batch size is at least 1024 (32 GPUs × batch sizes of 32-64 as specified in Tables 6-7). This is a standard technique in contrastive learning that provides richer negative signal without increasing memory usage — the communication overhead of gathering embeddings is small relative to the forward/backward pass compute.

Hard negative mining. For tasks with Neg. > 0 in the data configuration (T2V, TI2V, TV2V — 1 hard negative per positive), hard negatives are pre-mined using the same prober model Ψ_t used for alignment estimation. The mining procedure (inferred from standard practice, as the paper does not detail it) likely works as follows: for each query, embed it with the prober, retrieve the top-K most similar videos from a large corpus using approximate nearest neighbor search, filter out the correct positive, and keep the highest-scoring remaining video as the hard negative. Hard negatives are stored with the dataset and loaded alongside the positive pair during training. This pre-mining (rather than online mining) trades storage for training speed — the hard negatives are fixed for an epoch rather than being dynamically updated, which is simpler to implement but may provide slightly stale negatives by the end of an epoch.

Optimizer and training configuration (Appendix A.8):

• Optimizer: AdamW (Loshchilov and Hutter, 2017) with learning rate 3 × 10^{-5}, weight decay 0.1, and betas at PyTorch defaults (0.9, 0.999 — not explicitly stated but standard). AdamW is chosen over standard Adam because decoupled weight decay provides better generalization in fine-tuning settings by preventing weight norms from growing without suppressing the adaptive learning rate behavior.
• Learning rate schedule: Cosine decay over 3 training epochs, starting at 3 × 10^{-5} and decaying to near zero. Three epochs is relatively short — the model is being fine-tuned from a strong pretrained initialization (Qwen2.5-VL) rather than trained from scratch, so the risk of overfitting is higher and fewer epochs are needed.
• Precision: BFloat16 (bf16) mixed precision. BF16 has the same exponent range as FP32 (8 bits) but reduced mantissa (7 bits vs. 23), which means it can represent large and small values without overflow/underflow but with reduced precision. This is preferred over FP16 for training large transformers because the extended dynamic range eliminates the need for loss scaling.
• Batch sizes per task (Tables 6-7): Range from 32 to 64 per GPU, with an effective total batch size of at least 1024 (32 GPUs × 32 minimum per-GPU batch). The batch size varies by task — memory-intensive video tasks use smaller batches, while text-only tasks use larger batches.
• Video preprocessing: 8 frames uniformly sampled at 1.0 FPS (frames per second) from each video. This means a 15-second MSRVTT video contributes 8 frames (~0.53 FPS effective sampling rate given the uniform sampling over the clip duration); a 26-minute LoVR video also contributes 8 frames (~0.005 FPS). The fixed frame count rather than fixed FPS means short videos get denser temporal sampling than long videos — a practical compromise that keeps batch sizes manageable.
• Gradient checkpointing: Enabled to reduce memory by recomputing intermediate activations during the backward pass rather than storing them. This trades a ~20% training slowdown for significantly reduced memory usage.
• Distributed training: DeepSpeed framework with 32 NVIDIA A100 80GB GPUs. The paper does not specify the DeepSpeed stage (ZeRO-1, 2, or 3), but given the use of LoRA (which already dramatically reduces trainable parameters), ZeRO-1 or ZeRO-2 (partitioning optimizer states and gradients) is likely sufficient.

Why these hyperparameter choices: The learning rate 3 × 10^{-5} is relatively conservative for fine-tuning — lower than the 1 × 10^{-4} to 5 × 10^{-4} typical for pre-training contrastive models like CLIP — because the model starts from a strong pretrained initialization and the risk of catastrophic forgetting is real. The cosine decay ensures the model takes large steps early (when it needs to adapt to the retrieval objective) and fine-grained steps later (when it is refining the embedding space). The weight decay of 0.1 is higher than typical for fine-tuning (0.01 is more common) but is appropriate for contrastive learning where preventing embedding norm growth is important for stable cosine similarity computation.

---

Training Data Scaling Analysis

The paper includes a systematic data scaling study (Figure 6, Section 4.2; Appendix A.11, Figure 11) that measures how performance changes as the number of training pairs increases from approximately 10^5 to 10^7. The findings are fitted to a logarithmic scaling law:

$y = a \ln x + b$

where y is the performance metric (average across datasets or abilities), x is the number of training pairs, a is the scaling coefficient (performance gain per natural-log unit increase in data), and b is the intercept (performance at 1 training pair — not physically meaningful but sets the curve's vertical position).

Scaling per decade (10× data increase): The key quantifications are the absolute and relative performance gain per 10× increase in training data (one order of magnitude on the log scale, corresponding to a · ln(10) ≈ 2.303a):

• GVE-3B: +7.4% absolute gain per decade on dataset-averaged metric; +7.1% on ability-averaged metric.
• GVE-7B: +5.4% absolute gain per decade on both metrics.

The GVE-3B exhibits higher relative scaling efficiency (steeper slope) despite lower absolute performance, suggesting that the smaller model benefits more from additional data — a pattern consistent with neural scaling laws where smaller models are more data-efficient learners but have lower asymptotic performance ceilings. The GVE-7B's higher baseline but flatter slope implies that additional data beyond ~10M pairs yields diminishing returns for the larger model.

Task-specific scaling (Appendix A.11, Figure 11): The scaling behavior varies substantially by ability:

• Composed retrieval (CMP): GVE-3B gains +0.039 per decade (14.7% relative), nearly double GVE-7B's +0.025 per decade (8.7%). Composed retrieval benefits most from data scaling, likely because it requires learning to combine modalities — a skill that benefits from diverse examples.
• Coarse-grained (CG): GVE-3B gains +0.057 per decade (11.1%), far exceeding GVE-7B's +0.037 (6.6%). Coarse-grained matching saturates quickly for the larger model.
• Long-context (LC): The only ability where GVE-7B outperforms GVE-3B in both absolute (+0.042 vs. +0.029) and relative gain (5.4% vs. 3.8%). Long-context understanding appears to require the larger model's capacity — the 7B model is better able to aggregate information across extended temporal windows.

Why logarithmic scaling matters: The logarithmic fit implies that each additional order of magnitude of data provides a fixed absolute performance gain — data efficiency decreases as the dataset grows. This has practical implications: doubling the dataset from 1M to 2M pairs yields roughly the same gain as doubling from 100K to 200K, even though the absolute number of new examples is 10× larger in the first case. For practitioners deciding how much data to synthesize, the paper's scaling curves provide a concrete estimate of the return on investment for additional data generation.

4. Key Insights and Innovations

Innovation 1: Co-Design as a Methodological Principle, Not Just an Engineering Strategy

The paper's most fundamental intellectual contribution is not any single component — not UVRB, not V-SynFlow, not the Modality Pyramid — but rather the explicit argument that evaluation, data, and modeling must be co-designed to escape systemic specialism. This is a meta-contribution about how to do research in video retrieval, distinct from the what of any particular method.

Prior work operates in a fragmented ecosystem where benchmarking, data curation, and model training are treated as independent activities performed by separate communities with separate incentives. The benchmarking community builds test sets (CaReBench, CameraBench, LoVR) to measure progress but does not close the loop to inform what data should be generated or what training strategies would address revealed gaps. The data synthesis community (MegaPairs, AuroraCap) generates large-scale training data but validates it against narrow, existing benchmarks that cannot reveal whether the data actually covers the full semantic space needed for universality. The model training community (Unite, VLM2Vec, GME) optimizes architectures and losses but evaluates on MSRVTT and DiDeMo — benchmarks that UVRB's own diagnostic analysis proves are poor proxies for general ability (ρ = 0.58 correlation with average universal performance; Section 4.3, Finding 1).

The paper's contribution is to break this cycle by making the diagnostic output of evaluation directly causal to data generation and training design. UVRB does not just measure performance — it diagnoses which specific capabilities are missing (spatial vs. temporal, coarse vs. fine-grained, literal vs. partially relevant) and which capabilities interfere with each other (the spatial-temporal decoupling in CLIP models, the inverse correlation between compositional and visual accuracy). These diagnostics then become the design specification for V-SynFlow: the 60/30/10 temporal/spatial/theme split in the information profiles for caption generation is a direct response to the finding that temporal understanding is the most lacking capability. The Modality Pyramid's curriculum structure — progressing from foundational tasks to composite ones — is a direct response to the correlation analysis showing that image-trained models transfer to video, implying a knowledge hierarchy that training should exploit.

This is a fundamental shift in how video retrieval research is conducted, not an incremental improvement. The paper is arguing — implicitly but clearly — that the field's current equilibrium is self-reinforcing: narrow benchmarks incentivize narrow data, which trains narrow models, which perform well on narrow benchmarks, creating the illusion of progress while universal capability stagnates. Breaking this cycle requires simultaneous intervention at all three levels, because intervening at any one level in isolation would be absorbed back into the equilibrium (a better model on narrow benchmarks is still narrow; a richer dataset evaluated on narrow benchmarks looks unnecessary; a better benchmark without corresponding training data becomes a vanity metric). The tripartite framework — UVRB + UVRD + GVE — is the minimum viable intervention to escape this trap.

What distinguishes this from typical "we built a benchmark and a model" papers is the explicit diagnostic feedback loop. V-SynFlow's information profiles are not arbitrary — they are parameterized to target specific capability gaps identified by UVRB. The Modality Pyramid's curriculum is not a generic easy-to-hard schedule — it is structured to respect the specific dependency relationships that UVRB's correlation analysis reveals (e.g., spatial understanding transfers to temporal reasoning, so spatial tasks should be mastered first). This tight integration means the paper's contributions cannot be evaluated independently — UVRB without V-SynFlow would be a diagnostic without a treatment; V-SynFlow without UVRB would be data generation without quality assurance; GVE without the Modality Pyramid would be suboptimal (Table 3 proves this: the curriculum adds 1.8-3.1% absolute gain). The whole is genuinely greater than the sum of its parts.

The evidence is architectural rather than experimental — it resides in the design rationale connecting Sections 3.1 (UVRB), 3.2 (V-SynFlow), and 3.3 (Modality Pyramid) and in the ablation (Table 3) showing that each component independently matters. But the innovation is conceptual: it changes how to think about building universal retrieval systems from "let's build a bigger model on more data" to "let's diagnose what's missing, synthesize data to fill those gaps, and train with a curriculum that respects the structure of the capability space."

---

Innovation 2: Partially Relevant Retrieval as the Universal Diagnostic, Not a Niche Scenario

The paper's most surprising and potentially field-reorienting empirical finding is that partially relevant retrieval — matching videos to queries that capture only local, abstract, or thematic information — is not a marginal edge case but the single best proxy for universal retrieval ability. This is a fundamental reframing of what evaluation should prioritize.

The dominant assumption in the field — reflected in the canonical status of MSRVTT, DiDeMo, and LSMDC as the "standard" retrieval benchmarks — is that retrieval quality should be measured by how well a model matches complete, literal descriptions to full videos. These benchmarks test whether the model can find "the video of someone playing guitar on a stage" when queried with "a person playing guitar on stage." The queries and targets are designed for exact semantic correspondence.

UVRB's correlation analysis (Section 4.3, Finding 1; Figure 8) systematically undermines this assumption. Standard benchmarks like MSRVTT show a low correlation with average universal performance (ρ = 0.58), meaning a model that excels on MSRVTT tells you almost nothing about whether it excels on spatial reasoning, temporal dynamics, composed queries, or long-context retrieval. In contrast, the partially relevant retrieval tasks — LoVR-TH (theme-to-clip matching), PEV-K (keyword-based matching), and DREAM-E (event-to-video matching) — collectively achieve the highest average correlation with overall performance (ρ = 0.97). This means that how well a model handles fuzzy, imprecise, abstract queries is a better predictor of its general retrieval capability than how well it handles the literal queries that the field has focused on for a decade.

This finding makes conceptual sense once articulated but was non-obvious beforehand. Partially relevant retrieval is fundamentally harder than literal retrieval because it requires the model to abstract away from surface features and match at a semantic or thematic level. A model that can map "the overall style of the animation is vibrant and whimsical" to the correct short clip (LoVR-TH) must understand visual style — a higher-order concept that presupposes but goes beyond object recognition. A model that can map a list of keywords like "colorful, paper, beautiful" to the correct video (PEV-K) must aggregate across multiple abstract concepts and find a video that satisfies all of them without being distracted by irrelevant dominant content. These capabilities — abstraction, aggregation, semantic matching beyond literal description — are precisely what general retrieval demands across all tasks and domains, but they are invisible in literal benchmarks where surface-level keyword matching often suffices.

This reframes the research agenda: rather than continuing to optimize for MSRVTT and DiDeMo (where progress may largely reflect overfitting to narrow data distributions — a concern the paper explicitly raises), the field should prioritize partially relevant retrieval as the primary diagnostic for universality. Improved performance on PEV-K or LoVR-TH would be a stronger signal of genuine generalization than incremental gains on MSRVTT. This is a fundamental reframing of what "good" retrieval evaluation looks like — it is not an incremental methodological improvement but a shift in the target metric that, if adopted, would change which models are considered state-of-the-art and which research directions are pursued.

The evidence supporting this claim is statistical (Figure 8 shows the correlation structure; Figure 9 confirms that PR tasks have the highest ρ with the average of all abilities for both CLIP-based and MLLM-based models) but the insight is conceptual: it reveals that the field's evaluation paradigm has been measuring the wrong thing, and it provides a concrete alternative with supporting evidence.

---

Innovation 3: Architecture-Dependent Capability Development — CLIP vs. MLLM Divergence as a Diagnostic Lens

A third conceptual contribution is the paper's systematic demonstration that video retrieval capabilities develop in qualitatively different patterns depending on model architecture, and that these patterns reveal fundamental limitations that cannot be fixed by scaling alone.

Prior work compares CLIP-based and MLLM-based models on aggregate benchmarks, noting that MLLMs generally outperform CLIP. The standard interpretation is "MLLMs are better" — a performance fact, not an insight. UVRB's correlation analysis (Section 4.3, Findings 2-4; Figure 7) decomposes this aggregate superiority into specific, interpretable patterns of capability divergence that reveal why MLLMs are better and, more importantly, what specific limitations each architecture class has that architecture-agnostic scaling cannot address.

Finding 2: Spatial-temporal decoupling. CLIP-based models exhibit near-zero correlation between spatial and temporal capabilities (ρ_S-T = 0.12), meaning a CLIP model that excels at recognizing objects and spatial relationships tells you nothing about whether it can track action sequences or camera movements. For MLLM-based models, this correlation rises to ρ = 0.64 — substantial but far from 1.0, indicating that even MLLMs have not fully integrated spatial and temporal reasoning. The implication is that joint spatiotemporal modeling is not an emergent property of scale — it requires explicit architectural or training design. The paper's analysis reveals that both architecture classes have this limitation, just to different degrees.

Finding 3: Divergent trade-offs. CLIP-based models show a strong inverse correlation between compositional retrieval and visual accuracy (ρ_CMP-VIS = −0.71), meaning the better a CLIP model is at pure visual matching, the worse it is at understanding composed queries involving text+image binding. MLLM-based models show no such trade-off — compositional and visual capabilities are positively correlated (ρ ≈ 0.41). This suggests that CLIP's contrastive vision-language training creates a tension between visual fidelity and linguistic reasoning that MLLM pretraining resolves. It also explains why CLIP-based models plateau on complex retrieval tasks — improving one capability actively degrades another, creating a zero-sum optimization landscape.

Finding 4: Scaling has minimal impact on visual perception. Parameter scaling from 87M (CLIP4Clip) to 8.3B (Unite-7B) yields negligible gains in low-level visual retrieval (VIS: 0.714 for CLIP4Clip vs. 0.702 for Unite-7B). But high-level semantic coherence — measured by the correlation between partially relevant and coarse-grained performance — improves dramatically with MLLM architectures (ρ_PR-CG = 0.98 for MLLMs vs. 0.70 for CLIP). This means MLLMs are better at abstract semantic understanding but not at raw visual matching — a capability profile that is invisible in aggregate scores but clear in the multi-dimensional diagnostic.

These findings collectively argue that architecture choice is not merely a performance dial but fundamentally shapes which capabilities a model can develop and which trade-offs it faces. The implication — which the paper does not fully explore but which follows naturally — is that future universal retrievers may need hybrid architectures that combine CLIP's visual precision with MLLMs' semantic abstraction, rather than continuing to scale either paradigm in isolation. This is a fundamental insight, not an incremental finding, because it challenges the assumption that scaling any architecture will eventually converge to universal capability.

---

Innovation 4: Verifier-Free Difficulty Estimation via Task Alignment for Multi-Task Curriculum Learning

The Modality Pyramid's alignment-aware scheduling mechanism introduces a conceptual innovation that extends beyond the specific video retrieval context: using a model's own current representations to dynamically estimate task difficulty and structure curriculum without external verifiers or predefined difficulty labels.

Standard curriculum learning approaches (for multi-task or multi-domain training) typically rely on one of three strategies: (1) Fixed schedules where tasks are ordered by human-annotated difficulty (e.g., train on short sequences before long sequences) — these are brittle and may not match the actual difficulty ordering for a specific model. (2) Loss-based scheduling where tasks with higher training loss are sampled more frequently — this confounds difficulty with noise (a task can have high loss because it's inherently stochastic, not because it's hard). (3) Verifier-based scheduling where an external model estimates task performance and adjusts sampling — this introduces a dependency on a second model that may have its own biases.

The Modality Pyramid's mechanism is distinct: it uses the model's own embeddings (via the prober, which after epoch 1 is just the previous epoch's GVE checkpoint) to measure alignment — the average cosine similarity between query and target embeddings for each task. High alignment means the model has already learned to place queries and targets close together in the embedding space; low alignment means it hasn't. The temperature-annealed softmax over alignment scores produces a curriculum that is simultaneously model-specific (because alignment depends on GVE's own representations), dynamic (because alignment is re-estimated each epoch as the model learns), and optimization-target-aligned (because the alignment metric — cosine similarity — is exactly the retrieval metric at test time).

This is a conceptual refinement of curriculum learning rather than a fundamental breakthrough, but it addresses a genuine limitation in prior approaches: the disconnect between how difficulty is estimated and what the model is actually optimizing. Using the model's own retrieval metric as the difficulty signal ensures that the curriculum is always aligned with the downstream task. The annealing from low temperature (focused on well-aligned tasks early) to high temperature (progressively incorporating harder tasks) is a principled way to implement "easy-to-hard" without needing to define "easy" ahead of time — the model discovers what is easy for itself by measuring its own alignment.

The evidence for this innovation's effectiveness is in the ablation (Table 3): GVE-s (UVRD data with uniform task sampling) vs. GVE (UVRD data with Modality Pyramid) shows a consistent 1.8-3.1% improvement from adding the curriculum, with gains concentrated in the harder tasks (CMP, T, FG) where curriculum scheduling matters most. The innovation is not the raw performance gain but the mechanism design: it provides a general recipe for multi-task contrastive training that could transfer to other domains (e.g., multi-lingual retrieval, multi-domain image retrieval) because the alignment estimation is domain-agnostic — it only requires the ability to compute cosine similarity between query and target embeddings for each task.

---

Innovation 5: Synthesis Quality Control via Cross-Modal Consistency as a Principled Filter

While data synthesis pipelines (using LLMs or MLLMs to generate training data) are increasingly common in the field, V-SynFlow's multi-granular quality control stage introduces a conceptual contribution that distinguishes it from prior synthesis work: using a pre-existing strong embedder as a semantic gatekeeper to ensure synthetic data is grounded in visual reality rather than model hallucination.

The standard failure mode of synthetic data generation for multimodal tasks is that the generator model (an MLLM captioner) hallucinates — it produces captions that are linguistically fluent and plausible but not actually describing the visual content of the paired video. This happens because MLLMs are trained on vast text corpora and can generate reasonable-sounding descriptions of generic scenes without attending carefully to the specific visual input. When such hallucinated captions are used as training data for a retrieval model, they teach the model to match text to video based on spurious correlations rather than genuine visual understanding — the model learns that "a person walking in a park" is a valid label for many videos because the synthetic data contains many such hallucinated matches.

V-SynFlow's cross-modal consistency filter addresses this by introducing an independent verifier (GME-7B) that measures whether the original, weakly-annotated caption actually matches the video content. Pairs where the caption-video similarity falls below a threshold are discarded before any enrichment occurs. This ensures that the MLLM captioner in Stage 2 is describing videos that have been validated as having at least minimal semantic alignment with their original text — the captioner is enriching an already-grounded pair rather than hallucinating from scratch. The filter acts as a semantic floor: it removes catastrophically misaligned pairs (wrong language, wrong scene, metadata-only text) while keeping pairs with noisy but semantically connected annotations that can be improved through enrichment.

This is a conceptual refinement of data synthesis pipelines rather than a fundamental breakthrough, but it addresses a genuine and underexplored problem: how to ensure synthetic multimodal training data is faithful to the visual content. Prior synthesis work often relies solely on the generator model's ability to attend to visual input (trusting that the MLLM "sees" the video correctly) or on post-hoc filtering of synthetic outputs (which catches only obviously bad generations). V-SynFlow's approach is more principled: filter the inputs to the synthesis pipeline for quality, ensuring the generator has a semantically coherent starting point. The multi-granular aspect — combining annotation rectification (text quality), cross-modal consistency (text-video alignment), and temporal dynamics filtering (content quality) — provides layered assurance that surviving data is worth enriching.

The evidence is indirect (the quality of UVRD is validated by the downstream performance of GVE rather than by human evaluation of the captions), but the conceptual contribution — using an off-the-shelf embedder as a principled quality gate for synthetic data generation — is transferable to other domains and data synthesis pipelines.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the Universal Video Retrieval Benchmark (UVRB), the 16-dataset suite constructed by the authors (Section 3.1, Appendix A.2). The test sets are drawn from existing benchmarks — MSRVTT (1,000 pairs, JSFusion split), DiDeMo (1,004 pairs), CaReBench subsets (CRB-G, CRB-S, CRB-T; 1,000 pairs each), VDC-O and VDC-D (1,027 and 1,000 pairs), CameraBench (CMRB; 728 queries, 1,071 corpus), DREAM-1K-Event (6,251 queries, 1,000 corpus), LoVR-Theme2Clip (8,854 queries, 8,854 corpus), PE-Video-Keyword (14,427 queries, 15,000 corpus), LoVR-Text2Video (100 queries, 467 corpus), MomentSeeker adapted as MS-TI and MS-TV (400 queries each, 10 corpus), MSRVTT-I2V (1,000 pairs), and LoVR-Clip-to-Video (467 pairs). Videos range from 5.7 seconds (CMRB) to 1,560 seconds (LoVR-V); query word counts range from 6.5 (DREAM-E) to 17,365 (LoVR-V). The benchmark evaluates nine distinct abilities — three query formats (textual, composed, visual) across six data domains (coarse-grained, fine-grained-spatial, fine-grained-temporal, partially relevant, long-context, and composed) — but results are reported per-dataset and aggregated across abilities.

• Base model(s). The primary models are GVE-3B and GVE-7B, both derived from Qwen2.5-VL-Instruct (Bai et al., 2025) and fine-tuned using LoRA (Section 3.3, Appendix A.7-A.8). The 3B variant contains 3.8B total parameters; the 7B variant contains 8.3B. These sizes were chosen to span a representative range — the 3B model tests whether a compact architecture can achieve universal retrieval through better data and curriculum, while the 7B model establishes the upper bound. The vision encoder is kept frozen; only the LLM backbone is adapted via LoRA (r=16, α=32, targeting q_proj, v_proj, k_proj, up_proj, down_proj, gate_proj). Training uses 12.55M instances for GVE-3B and 13.73M for GVE-7B across 3 epochs with AdamW (lr=3×10⁻⁵, weight decay 0.1, cosine schedule) on 32 A100 80GB GPUs.

• Metrics. The primary metric is Recall@1 (R@1), measuring whether the highest-ranked retrieved item is correct. For datasets where queries are inherently fuzzy and exact-match ranking is too strict — CMRB (camera motion descriptions) and LoVR-TH (theme-to-clip matching) — Recall@10 (R@10) is used instead, measuring whether the correct item appears in the top-10 results. For composed tasks with multiple valid positives (MS-TI and MS-TV, where multiple video clips can satisfy a text+image or text+video query among only 10 candidates), Precision@1 (P@1) is used, measuring whether the single top-ranked item is one of the valid positives. The dataset-averaged metric (AVG of D) and ability-averaged metric (AVG of A) are reported as aggregate scores — AVG of D averages across the 16 individual dataset scores (combining R@1, R@10, and P@1 directly, which is mathematically questionable but standard practice in the field for comparability), while AVG of A averages across the nine ability categories derived from grouping datasets.

• Baselines. Fourteen models spanning CLIP-based and MLLM-based architectures are evaluated (Section 4.1, Appendix A.9, Table 10). CLIP-based models: CLIP4Clip (Luo et al., 2022; 87M parameters, text-video training only), ViCLIP (Wang et al., 2023; 0.4B), VideoCLIP-XL (Wang et al., 2024a; 0.4B), LanguageBind-Video-Huge-V1.5 (Zhu et al., 2024; 1.2B), InternVideo2-Stage2-1B (Wang et al., 2024c; 1.4B), InternVideo2-Stage2-6B (6.4B). MLLM-based models: GME-2B (Zhang et al., 2025; 2.2B, text-text and text-image training, no video data), Unite-Base-2B (Kong et al., 2025; 2.2B, text-text, text-image, and text-video training), VLM2Vec-V2.0 (Meng et al., 2025; 2.2B, all three modalities), BGE-VL-v1.5-mmeb (Zhou et al., 2024; 7.6B, text-text and text-image only), UniME-7B (Gu et al., 2025; 8.0B, text-text and text-image only), B3-7B (Thirukovalluru et al., 2025; 8.3B, text-text and text-image only), GME-7B (Zhang et al., 2025; 8.3B, text-text and text-image only), and Unite-Base-7B (Kong et al., 2025; 8.3B, all three modalities). The training data composition of baselines varies significantly — critically, several baselines may have been trained on in-domain data for certain UVRB test sets (e.g., MSRVTT is a common training dataset), giving them an unfair advantage over GVE which is evaluated strictly zero-shot. Baseline models with learnable relevance estimation modules (e.g., InternVideo2's MLP head) have those modules removed, with the last hidden state before the head used as the embedding.

• Generation budget / compute accounting. For this paper, there is no generation budget in the traditional sense — models produce fixed-length embeddings rather than generating tokens. The relevant compute budgets are training-side: number of training pairs (data scaling experiments in Section 4.2, Figure 6, Appendix A.11), training epochs (3 for all GVE variants), and GPU-hours (32 A100s for training). At inference time, the relevant budget is test-time compute scaling (Appendix A.12): the number of sampled frames (8, 16, or 48) and the maximum visual tokens per frame (200, 400, or 800). These are varied independently to study how input density affects retrieval quality without retraining. For the data scaling experiments, the x-axis is the number of training pairs on a log scale from ~10⁵ to ~10⁷; different points correspond to training runs with subsets of the full data mixture.

• Cross-validation / statistical protocol. No formal cross-validation is used — GVE is evaluated once on the fixed UVRB test sets after training on the full data mixture. The data scaling curves (Figures 6, 11) include shaded bands showing ±1 standard deviation, but the source of variance (multiple training runs? variance across data subsets?) is not specified. Baseline models are evaluated with their publicly released weights without fine-tuning. For the test-time scaling experiments (Appendix A.12), the paper performs controlled sweeps over frame count and token budget, evaluating the same model checkpoint at each configuration rather than retraining. All evaluation uses cosine similarity as the relevance metric with no post-processing or re-ranking. Embeddings are L2-normalized. Videos are uniformly sampled into exactly 8 frames (except in the test-time scaling experiments where frame count varies). Frame resolution is adaptively adjusted per model to enforce the constraint that each frame encodes ≤200 visual tokens. Input sequences are capped at 8,192 tokens with truncation applied above this limit.

Main Quantitative Results

Overall Performance on UVRB (Tables 1 and 2)

GVE-7B achieves state-of-the-art zero-shot generalization across UVRB. The headline numbers: GVE-7B obtains a dataset-averaged score (AVG of D) of 0.573 and an ability-averaged score (AVG of A) of 0.600, outperforming the next-best model — Unite-7B — by +6.5% relative on AVG of D (0.573 vs. 0.538) and +7.3% relative on AVG of A (0.600 vs. 0.559). GVE-3B scores 0.544 (AVG of D) and 0.571 (AVG of A), surpassing the 8.3B-parameter Unite-7B despite having fewer than half as many parameters (3.8B vs. 8.3B). This provides one of the paper's central claims: the gains come from better data synthesis and curriculum design, not from model scale or data leakage.

Per-dataset breakdown (Table 1). GVE-7B leads on 11 of 16 datasets outright and ties or is within margin on the remaining 5. Specific highlights:

• Coarse-grained text-to-video: On MSRVTT, GVE-7B achieves R@1 = 0.464 vs. Unite-7B at 0.439 and InternVideo2-6B at 0.485 (the only dataset where GVE-7B is not first — InternVideo2-6B's advantage may reflect in-domain training on MSRVTT data). On DiDeMo, GVE-7B achieves 0.433 vs. Unite-7B at 0.386 and LanguageBind at 0.421. On CRB-G (CaReBench general), GVE-7B scores 0.865 vs. VLM2Vec-V2 at 0.828.

• Fine-grained spatial: On CRB-S, GVE-7B (0.847) leads VLM2Vec-V2 (0.843) and GVE-3B (0.846). On VDC-O (object-focused retrieval), GVE-7B achieves 0.794 vs. VLM2Vec-V2 at 0.775. The fine-grained spatial tasks show relatively compressed performance — many models score above 0.800 — suggesting spatial understanding is a strength of MLLM-based architectures generally.

• Fine-grained temporal: This is where GVE-7B opens larger gaps. On CRB-T, GVE-7B scores 0.539 vs. Unite-7B at 0.472 (a 14.2% relative improvement). On CMRB (camera motion), GVE-7B achieves R@10 = 0.398 vs. Unite-7B at 0.351. Temporal reasoning is the most challenging fine-grained domain — the best scores are substantially lower than spatial scores — and GVE-7B's advantage here is the clearest single-task evidence that the Modality Pyramid and temporally-biased data synthesis (60% temporal information profiles) are working as intended.

• Partially relevant: On DREAM-E, GVE-7B scores 0.302 vs. UniME-7B at 0.293 and Unite-7B at 0.279 — a narrow lead. On LoVR-TH (theme-to-clip), GVE-7B achieves R@10 = 0.542 vs. Unite-7B at 0.555 — one of the few datasets where Unite-7B outperforms GVE-7B. On PEV-K (keyword-based), GVE-7B scores 0.413 vs. Unite-7B at 0.440 — another Unite-7B advantage. These results suggest that partially relevant retrieval is a relative strength of Unite-7B (which was trained on text-video data including noisy web captions that may inherently be partially relevant matches) — GVE-7B is competitive but not dominant in this domain.

• Long-context: GVE-7B dominates. On LoVR-V (long-text to long-video), GVE-7B scores 0.680 vs. GME-7B at 0.710 (GME-7B leads here — suggesting image-trained models transfer surprisingly well to long-context video retrieval without explicit video training). On VDC-D (detailed captions to 30-second videos), GVE-7B achieves 0.948 vs. VLM2Vec-V2 at 0.913 and GVE-3B at 0.918. The 0.948 score is the highest single-dataset performance in the entire benchmark, indicating that long-context retrieval at moderate video lengths is nearly solved by GVE-7B.

• Composed: On MS-TI (text+image to video), GVE-7B scores P@1 = 0.343 vs. GME-2B at 0.350 (GME-2B leads). On MS-TV (text+video to video), GVE-7B scores 0.280 vs. UniME-7B at 0.305. Composed retrieval is the weakest domain for GVE — it leads on neither dataset — suggesting that the 124K composed training pairs (89K TI2V + 35K TV2V) may be insufficient relative to the difficulty of the task. Alternatively, the baseline models may have architectural advantages for compositional reasoning from their pretraining.

• Visual: On MSRVTT-I2V (image-to-video), GVE-7B scores 0.899 vs. B3-7B at 0.884 and Unite-7B at 0.883. On LoVR-C2V (clip-to-video), GVE-7B scores 0.415 vs. UniME-7B at 0.537 — a large gap where UniME-7B significantly outperforms.

Per-ability breakdown (Table 2). This table aggregates per-dataset scores into the nine ability categories, providing a higher-level view of model strengths. GVE-7B leads in 7 of 9 abilities:

• Tasks: TXT (textual) 0.657 vs. Unite-7B 0.609; CMP (composed) 0.312 vs. UniME-7B 0.308; VIS (visual) 0.657 vs. UniME-7B 0.702. Visual retrieval is the only task category where GVE-7B does not lead — CLIP4Clip, with only 87M parameters, scores 0.714 on VIS, outperforming all MLLM-based models. This is the paper's Finding 4 (Section 4.3): visual perception does not improve with scale or architecture change, and it is largely uncorrelated with overall retrieval success (ρ_AVG-VIS = 0.26).

• Domains: CG (coarse-grained) 0.587 vs. Unite-7B 0.541; FG (fine-grained) 0.570 vs. Unite-7B 0.539; LC (long-context) 0.814 vs. GME-7B 0.788. The long-context score of 0.814 is the highest ability-level score achieved by any model on any ability, confirming that GVE's combination of detailed synthetic captions and the Modality Pyramid's progressive curriculum is particularly effective for extended temporal reasoning.

• Sub-domains: S (spatial) 0.821 vs. VLM2Vec-V2 0.809; T (temporal) 0.469 vs. Unite-7B 0.412; PR (partially relevant) 0.419 vs. Unite-7B 0.425. The partially relevant sub-domain is the only one where GVE-7B does not lead — Unite-7B's advantage in this category (0.425 vs. 0.419) is consistent with its per-dataset leads on LoVR-TH and PEV-K.

GVE-3B vs. GVE-7B comparison. The 3B model achieves 0.544 AVG of D and 0.571 AVG of A — competitive with or surpassing models 2-4× its size. Specifically, GVE-3B outperforms Unite-7B (8.3B) on AVG of A (0.571 vs. 0.559) and on AVG of D (0.544 vs. 0.538). The 3B model's advantages concentrate in tasks where data quality and curriculum matter most: TXT 0.619 (vs. Unite-7B 0.609), CG 0.552 (vs. 0.541), FG 0.541 (vs. 0.539), S 0.816 (vs. 0.779), T 0.430 (vs. 0.412). The 7B model's advantages concentrate in long-context and composed tasks: LC 0.814 vs. 0.764; CMP 0.312 vs. 0.304. This pattern — the 3B model competitive on spatial, temporal, and coarse-grained tasks but the 7B model pulling ahead on long-context and composed — is consistent with the data scaling results (Section 4.2, Appendix A.11) showing that LC is the only ability where GVE-7B has higher scaling efficiency than GVE-3B, and CMP requires more model capacity.

The unfair baseline advantage. The paper notes that "competing models may have an unfair advantage for using training data corresponding to several test sets" (Section 4.2). InternVideo2-6B's strong MSRVTT performance (0.485 vs. GVE-7B's 0.464), Unite-7B's strong DiDeMo performance (0.386, though still below GVE-7B's 0.433), and various baselines' performance on in-domain tasks likely reflect training data overlap. GVE is evaluated strictly zero-shot — none of the UVRB test sets appear in its training data. This makes GVE's superior average performance more significant: it achieves better generalization without the benefit of training on the test distribution.

---

Ablation Study: Contributions of UVRD and Modality Pyramid (Table 3)

Table 3 isolates the distinct contributions of the synthesized dataset (UVRD) and the Modality Pyramid curriculum through a three-way comparison: GVE-i (baseline: trained only on the main data mixture from Table 6 — text-only, image-centric, and video-centric datasets — without UVRD), GVE-s (adds UVRD to the training mixture but uses uniform task sampling rather than the Modality Pyramid), and GVE (the full system with both UVRD and the Modality Pyramid).

Headline findings (7B model): The progression from GVE-i-7B → GVE-s-7B → GVE-7B shows:

• AVG of D: 0.563 → 0.568 (+0.005) → 0.573 (+0.005). Total gain: +0.010 (1.8% relative).
• AVG of A: 0.587 → 0.594 (+0.007) → 0.600 (+0.006). Total gain: +0.013 (2.2% relative).

UVRD's contribution (GVE-i → GVE-s). Adding UVRD provides the most dramatic gains on composed retrieval (CMP). For the 3B model: CMP jumps from 0.237 (GVE-i-3B) to 0.301 (GVE-s-3B) — a 27.0% relative improvement. For the 7B model: CMP jumps from 0.274 to 0.313 — a 14.2% relative improvement. This is the single largest per-ability gain from any component, confirming that composed retrieval is the task most dependent on synthesized training data — existing public datasets provide almost no signal for text+image-to-video or text+video-to-video matching, so UVRD's 124K composed pairs fill a critical gap.

UVRD also provides substantial gains in visual retrieval (VIS). For 3B: 0.632 → 0.617 (slight decrease — unexpected, possibly within noise). For 7B: 0.678 → 0.662 (also a decrease). This is counterintuitive — adding I2V and V2V pairs should improve visual retrieval, yet scores slightly drop. One possible explanation: the visual pairs in UVRD are synthesized (frame-to-video, clip-to-video) and may have different statistical properties than the image-to-video and video-to-video pairs in the test sets; adding this data may shift the model's visual embedding space in a way that doesn't transfer perfectly. Alternatively, the decreases may be within the noise of a single training run. The paper does not discuss this anomaly.

Long-context (LC) shows mixed effects: for 3B, adding UVRD increases LC from 0.808 to 0.775 (a decrease), while for 7B, LC increases from 0.795 to 0.804. The 3B decrease suggests that UVRD's long-context data may require the 7B model's greater capacity to integrate productively — the 3B model may be distracted by the additional long-context training instances rather than benefiting from them.

Modality Pyramid's contribution (GVE-s → GVE). The curriculum provides the most consistent gains on fine-grained and temporal tasks. For the 7B model: T (temporal) increases from 0.458 to 0.469; FG (fine-grained) increases from 0.563 to 0.570; TXT increases from 0.648 to 0.657. For the 3B model: T increases from 0.421 to 0.430; S (spatial) increases from 0.811 to 0.816. These are modest but consistent — typically +0.005 to +0.009 absolute — and concentrated in the domains where the curriculum's progressive structure (foundational spatial → advanced temporal) is designed to help.

The Modality Pyramid's effect on composed retrieval is mixed: for 3B, CMP increases slightly (0.301 → 0.304); for 7B, CMP slightly decreases (0.313 → 0.312). The curriculum's alignment-aware scheduling may not benefit composed tasks because they are genuinely harder regardless of training stage — their alignment scores remain low throughout training, so sampling probability stays low even at high temperatures. The paper does not analyze this directly but it's a plausible explanation for the flat results.

Cross-model patterns. The 3B model benefits more from UVRD (absolute gain from GVE-i to GVE: 0.528 → 0.544 = +0.016 on AVG of D) than the 7B model (0.563 → 0.573 = +0.010). This aligns with the data scaling analysis (Section 4.2): GVE-3B has higher scaling efficiency per decade of data, so adding 1.55M high-quality pairs provides a larger relative boost. The 7B model's higher baseline means the same absolute data addition yields smaller percentage gains.

---

Data Scaling Analysis (Figure 6, Appendix A.11 Figure 11)

The paper quantifies how retrieval performance scales with training data quantity by training GVE-3B and GVE-7B on subsets of the full 12.55M/13.73M data mixtures at four scales (~10⁵, 3×10⁵, 10⁶, 3×10⁶, 10⁷ pairs) and fitting a logarithmic scaling law y = a·ln(x) + b.

Headline numbers (Figure 6): On the dataset-averaged metric, GVE-3B improves from approximately 0.450 at 10⁵ pairs to 0.544 at 10⁷ pairs — a gain of ~0.094, corresponding to +7.4% relative per 10× data increase. GVE-7B improves from approximately 0.475 at 10⁵ pairs to 0.573 at 10⁷ pairs — a gain of ~0.098, corresponding to +5.4% relative per 10× data increase. The ability-averaged metric shows the same pattern: GVE-3B +7.1% per decade, GVE-7B +5.4% per decade.

The curves in Figure 6 show clear logarithmic behavior with diminishing returns — the improvement from 10⁶ to 10⁷ is roughly the same as from 10⁵ to 10⁶ in absolute terms, confirming the logarithmic fit. Neither model shows signs of saturation at 10⁷ pairs, suggesting that additional data beyond the current scale would continue to yield improvements, though with progressively lower efficiency.

Per-ability scaling (Appendix A.11, Figure 11). The scaling behavior varies substantially by ability:

• Textual (TXT): GVE-3B improves from ~0.53 to ~0.62 (absolute gain ~0.09, +7.1% per decade). GVE-7B improves from ~0.57 to ~0.66 (gain ~0.09, +5.8% per decade). Textual retrieval scales steadily for both models.

• Composed (CMP): GVE-3B shows the steepest scaling: from ~0.19 to ~0.30 (gain ~0.11, +14.7% per decade) — nearly double the scaling rate of any other ability. GVE-7B: from ~0.23 to ~0.31 (gain ~0.08, +8.7% per decade). Composed retrieval is the most data-hungry ability — the 3B model is still on the steep part of its scaling curve at 10⁷ pairs, suggesting that substantially larger composed datasets could yield further gains.

• Visual (VIS): Scaling is weak for both models. GVE-3B: ~0.61 to ~0.65 (gain ~0.04, +3.8% per decade). GVE-7B: ~0.62 to ~0.66 (gain ~0.04, +2.1% per decade). Visual retrieval benefits minimally from more data, consistent with Finding 4 that visual perception is architecture-bound rather than data-bound.

• Coarse-grained (CG): GVE-3B: ~0.44 to ~0.55 (gain ~0.11, +11.1% per decade). GVE-7B: ~0.50 to ~0.59 (gain ~0.09, +6.6% per decade). The 3B model scales better for coarse-grained tasks, suggesting data is more important than capacity for this ability.

• Fine-grained (FG): GVE-3B: ~0.44 to ~0.54 (gain ~0.10, +7.9% per decade). GVE-7B: ~0.49 to ~0.57 (gain ~0.08, +5.6% per decade). Similar pattern to coarse-grained — the 3B model benefits more from data scaling.

• Long-context (LC): This is the only ability where GVE-7B has higher absolute gain than GVE-3B. GVE-3B: ~0.69 to ~0.76 (gain ~0.07, +3.8% per decade). GVE-7B: ~0.72 to ~0.81 (gain ~0.09, +5.4% per decade). The larger model leverages additional data more effectively for long-context reasoning, confirming that capacity matters for aggregating information across extended temporal windows.

The key implication: scaling training data helps most for abilities that are underspecified in existing datasets (composed retrieval, where UVRD provides novel training signal) and helps least for abilities that are architecture-bound (visual perception) or already well-covered by existing data (long-context for the 7B model is approaching saturation). For practitioners, this suggests that data synthesis efforts should prioritize composed retrieval examples — each additional composed pair provides more marginal value than an additional text-video pair.

---

Test-Time Scaling: Frame Count and Token Budget (Appendix A.12, Figures 12-13)

The paper investigates how inference-time parameters — the number of sampled frames and the maximum visual tokens per frame — affect retrieval performance without retraining. This is distinct from the training-time scaling in Section 4.2: it measures whether GVE can benefit from denser visual input at test time.

Temporal scaling (Figure 12): Increasing frames from 8 to 16 to 48 (with max tokens fixed at 200 per frame) shows:

• Long-context (LC) benefits most: GVE-3B improves from ~0.76 (8 frames) to ~0.91 (48 frames) — a +19.6% relative gain. GVE-7B improves from ~0.78 to ~0.88 (+12.8%). Denser temporal sampling significantly aids long-range reasoning, as expected — more frames capture more of the video's temporal structure.

• Most abilities show monotonic improvement but with diminishing returns: Moving from 8 to 16 frames provides larger gains than from 16 to 48. On the dataset average, GVE-7B improves from ~0.57 (8 frames) to ~0.59 (16) to ~0.60 (48). The gains are modest beyond 16 frames for most tasks.

• Composed retrieval (CMP) degrades with more frames: GVE-7B drops from ~0.31 (8 frames) to ~0.30 (16) to ~0.29 (48). GVE-3B shows a similar pattern. Composed queries require binding a visual reference to a text query — additional frames from the target video may introduce distractors that interfere with matching the reference, or the query-side representation (which likely uses fewer frames) may not benefit from the target-side increase.

• The 7B model is more frame-efficient: The performance gap between GVE-7B and GVE-3B narrows as frames increase (from 0.029 at 8 frames to 0.024 on AVG-D at 48 frames). The 7B model extracts more information per frame, so additional frames provide smaller marginal benefit.

Spatial scaling (Figure 13): Increasing the maximum tokens per frame from 200 to 400 to 800 (with frames fixed at 8) shows non-monotonic behavior:

• Performance peaks ~400 tokens for most abilities: On the dataset average, GVE-7B scores ~0.568 (200 tokens), ~0.575 (400), ~0.565 (800). The 200→400 increase helps modestly; the 400→800 increase either plateaus or degrades.

• Long-context drops at 800 tokens: GVE-7B: ~0.78 (200) → ~0.80 (400) → ~0.77 (800). Excessive per-frame detail dilutes attention from salient features — the model has a fixed total token budget of 8,192, and spending it on high-resolution frames leaves less room for text tokens on long queries, or causes the vision tokens to dominate the attention computation.

• Composed retrieval again degrades with more information: As with temporal scaling, more visual detail hurts composed matching. GVE-7B CMP: ~0.30 (200) → ~0.29 (400) → ~0.28 (800). This reinforces the pattern that composed retrieval is sensitive to input redundancy — the model struggles to bind the textual and visual query components when the target representation becomes too rich.

• The 7B model is more robust to over-resolution: GVE-3B shows steeper degradation at 800 tokens than GVE-7B, suggesting the larger model has greater capacity to process fine-grained visual information without being overwhelmed.

Key practical insight: Naively increasing test-time compute by adding more frames or higher resolution is not uniformly beneficial. Temporal scaling is a robust strategy (more frames help, especially for long-context tasks). Spatial scaling has a clear optimal point (~400 tokens per frame) beyond which performance degrades. The paper frames this as motivating "adaptive input mechanisms, such as dynamic token or frame selection, that can intelligently manage information density" (Appendix A.12). For deployment, the recommendation is: use 16-48 frames at moderate resolution (~400 tokens) rather than 8 frames at very high resolution or very many frames at low resolution.

---

Video Classification Transfer (Appendix A.13, Table 11)

As a supplementary experiment, the paper evaluates GVE's embeddings on video classification benchmarks — a different task (classify a video into one of N action categories) using the same fixed embeddings produced by the model. This tests whether the retrieval-optimized representations transfer to recognition tasks.

Results (Table 11): On the five-dataset suite (Kinetics-700, UCF101, HMDB51, SomethingSomething-V2, Breakfast), LanguageBind achieves the highest mean accuracy (0.553), followed by InternVideo2-6B (0.526) and GVE-7B (0.526). GVE-3B scores 0.476.

GVE-7B leads on UCF101 (0.757 vs. LanguageBind 0.479 — though note: this is not an apples-to-apples comparison because baseline models may have been trained on UCF101 while GVE is zero-shot). On SomethingSomething-V2 (a fine-grained action recognition benchmark requiring temporal reasoning), InternVideo2-6B leads at 0.569 vs. GVE-7B at 0.521. On Breakfast (procedural activity recognition), InternVideo2-6B scores 0.453 vs. GVE-7B at 0.289 — a very large gap.

The classification results are not a primary contribution — they are included to show that GVE's embeddings generalize beyond retrieval — but they are notably weaker than GVE's retrieval results (where GVE-7B dominates). This suggests that GVE's contrastive training optimizes for pairwise similarity (which is what retrieval requires) but may not produce linearly separable class clusters (which is what classification with a simple nearest-centroid or linear probe requires). InternVideo2's stronger classification performance may reflect its training on classification-oriented datasets (Kinetics, SomethingSomething-V2).

---

Ablation Studies and Robustness Checks

The paper's ablation strategy is structured as a single, systematic comparison (Table 3) rather than a series of independent ablations. The following components are tested:

Synthesized dataset (UVRD) contribution: Table 3 compares GVE-i (no UVRD) vs. GVE-s (with UVRD, uniform sampling). The largest effect is on composed retrieval — the 3B model's CMP score increases by 27.0% relative (0.237 → 0.301) and the 7B model's by 14.2% (0.274 → 0.313). This confirms that UVRD's composed retrieval data (124K pairs) fills a genuine gap in existing training resources. Visual retrieval shows a small decrease for both models (3B: 0.632 → 0.617; 7B: 0.678 → 0.662), which the paper does not explain — possible explanations include distribution mismatch between synthetic visual pairs and test visual pairs, or noise from a single training run.

Modality Pyramid curriculum contribution: Table 3 compares GVE-s (UVRD with uniform sampling) vs. GVE (UVRD with Modality Pyramid). The curriculum provides consistent gains on text-to-video tasks (TXT: +0.009 for 7B, +0.002 for 3B), temporal reasoning (T: +0.011 for 7B, +0.009 for 3B), and fine-grained retrieval (FG: +0.007 for 7B, +0.005 for 3B). Composed retrieval is essentially flat (3B: +0.003; 7B: −0.001) — the curriculum does not help with the hardest task, suggesting that alignment-aware scheduling cannot compensate for fundamental difficulty. Long-context shows an unexpected decrease for the 3B model (0.775 → 0.764) but a small increase for the 7B (0.804 → 0.814) — the 3B model may struggle to integrate UVRD's long-context data effectively regardless of scheduling.

Data scaling behavior: Figure 6 and Appendix A.11 Figure 11 test four data scales (~10⁵ to ~10⁷ pairs). The key robustness finding is that scaling is approximately logarithmic for both model sizes and all abilities, with no evidence of saturation at the largest scale. Composed retrieval (CMP) scales steepest, visual retrieval (VIS) scales shallowest, and long-context (LC) is the only ability where GVE-7B scales better than GVE-3B. The shaded ±1σ bands in the figures are narrow for most abilities (suggesting consistent scaling behavior across data subsets) but wider for CMP and VIS — these are the most variable abilities.

Test-time frame and token scaling: Appendix A.12 (Figures 12-13) tests the robustness of the fixed 8-frame, 200-token evaluation protocol. The non-monotonic spatial scaling (peak at ~400 tokens, degradation at 800) is a negative result that the paper appropriately highlights: "naive spatial scaling is not" a robust strategy. This finding is important because it cautions against the natural assumption that "more resolution = better performance." The degradation of composed retrieval under both more frames and more tokens suggests that composed tasks are fundamentally sensitive to input density — a finding that was not obvious a priori and may have architectural implications (e.g., composed queries may need separate attention pathways for text and visual reference rather than fused attention).

Architecture robustness (CLIP vs. MLLM): The correlation analysis in Section 4.3 (Figures 7-8) is essentially a robustness check across model architectures. The finding that CLIP-based models show spatial-temporal decoupling (ρ = 0.12) while MLLM-based models show stronger coupling (ρ = 0.64) is replicated across 6 CLIP models and 8 MLLM models — this is not a single-model artifact. Similarly, the inverse correlation between compositional and visual accuracy in CLIP models (ρ_CMP-VIS = −0.71) is consistent across the CLIP models tested.

Training dynamics monitoring (Appendix A.10, Figures 9-10): The paper reports training loss, mean relevance score, max negative gap, and mean positive score across training steps for both GVE-3B and GVE-7B. These are monitoring metrics, not controlled ablations, but they provide evidence of training stability: loss decreases smoothly, positive scores increase monotonically, and the negative gap (margin between positive and hard negative scores) widens throughout training. For GVE-7B, all four metrics show clean convergence without divergence or collapse, suggesting the training hyperparameters (lr=3×10⁻⁵, τ=0.03, LoRA r=16) are appropriate for stable contrastive learning at this scale.

Missing ablations. Several experiments that would strengthen the paper's claims are absent:

• No ablation over the Modality Pyramid temperature schedule. The annealing from σ_min=0.1 to σ_max=1.0 is stated as default but never compared to alternatives (fixed temperature at 1.0 for uniform sampling, annealing at different rates, different min/max values). Without this, it's unclear whether the annealing specifically matters or if any increased task diversity late in training would produce the same gains.

• No ablation over the information profile distribution (60% temporal, 30% spatial, 10% theme). V-SynFlow's controlled captioning uses this split based on the diagnostic that temporal understanding is the most lacking ability, but the paper never tests whether a different split (e.g., 40/40/20 or 70/20/10) would work better or worse. The claim that UVRD is "guided by UVRB's diagnostics" would be stronger if the paper showed that deviating from the diagnostics-guided split degraded performance.

• No ablation over hard negative count. Table 7 shows that some tasks use 1 hard negative per positive while others use 0. The effect of hard negatives on retrieval quality is not isolated — would adding hard negatives to T2T or I2V improve those tasks? Would increasing hard negatives beyond 1 for T2V help?

• No ablation over LoRA rank. The paper uses r=16 across all target modules. Testing r=8 or r=32 would reveal whether the adaptation capacity is appropriately sized for the retrieval task.

• No comparison to training GVE from scratch vs. fine-tuning from Qwen2.5-VL. The paper assumes that fine-tuning a pretrained MLLM is superior, but never compares to a randomly initialized or CLIP-initialized baseline. This is defensible (modern practice strongly favors fine-tuning), but the contribution of Qwen2.5-VL's specific pretrained knowledge vs. the architecture itself is not isolated.

---

Critical Assessment

Claim 1: GVE achieves state-of-the-art zero-shot generalization on UVRB, outperforming all baselines by significant margins. The experimental evidence (Tables 1-2) supports this claim with qualifications. GVE-7B leads on 11 of 16 datasets and 7 of 9 ability categories. The margins are meaningful: +6.5% on AVG of D and +7.3% on AVG of A over the next-best model (Unite-7B). The 3B model surpassing the 8.3B Unite-7B on AVG of A (0.571 vs. 0.559) is a genuinely strong result that supports the claim that data and curriculum quality can compensate for model scale.

However, three qualifications are important. First, GVE does not dominate every task — Unite-7B leads on partially relevant retrieval (0.425 vs. 0.419), UniME-7B leads on visual retrieval (0.702 vs. 0.657), and InternVideo2-6B leads on MSRVTT (0.485 vs. 0.464). These exceptions may reflect training data overlap (baselines trained on MSRVTT) or architectural specialization, but they mean the claim of "state-of-the-art" is task-dependent rather than universal. Second, the baseline models are evaluated at their default configurations — no attempt is made to optimize baseline pre-processing (frame count, resolution, prompt format) for UVRB specifically, while GVE's evaluation protocol was presumably designed for GVE's architecture. This is standard practice but introduces a potential fairness concern. Third, the "zero-shot" claim is asymmetric — GVE is zero-shot on UVRB test sets, but several baselines likely have in-domain training data. This makes GVE's performance more impressive, but it also means the comparison is not a clean measurement of generalization ability because the baselines may be overfit to their training distribution rather than genuinely less capable.

Claim 2: UVRD provides the diverse knowledge required for complex tasks, with composed retrieval showing a 27% relative improvement. The ablation evidence (Table 3) supports this strongly for composed retrieval: the GVE-i-3B → GVE-s-3B jump of 0.237 → 0.301 (27.0% relative) is the largest per-ability effect in the entire paper. The claim is weaker for other abilities — UVRD's contribution to text-to-video, visual, and long-context retrieval is modest or even slightly negative (visual retrieval decreases for both model sizes). This suggests UVRD's primary value is in supplying data for tasks that existing datasets entirely lack (composed retrieval) rather than in improving data quality for already well-represented tasks (text-to-video). The paper's framing of UVRD as filling "identified semantic and structural gaps" is accurate — the largest gap was composed retrieval, and UVRD fills it — but the claim that UVRD is necessary for "diverse knowledge" across all tasks is overstated.

Claim 3: The Modality Pyramid curriculum enables progressive knowledge acquisition, producing consistent gains over uniform sampling. The evidence (Table 3) shows consistent but small gains: approximately +0.005 to +0.009 absolute per ability category. These are modest — the curriculum adds 1.8-3.1% relative improvement over UVRD with uniform sampling. The claim that the curriculum "enables progressive knowledge acquisition" is supported by the pattern of gains (concentrated in harder, later-stage tasks like temporal reasoning) but the paper provides no direct evidence of progressive learning — no analysis showing that task alignment scores actually evolve as predicted (spatial tasks aligning earlier than temporal tasks), no demonstration that early-epoch embeddings differ qualitatively from late-epoch embeddings in curriculum-vs-uniform training. The mechanism is plausible but not directly validated. The ablation confirms that the curriculum helps, but not why — it could be that any non-uniform sampling schedule that increases task diversity over time would produce similar gains, or that the annealing temperature specifically matters (the missing temperature ablation prevents distinguishing these possibilities).

Claim 4: Partially relevant retrieval is a better proxy for universal ability than standard benchmarks. The correlation evidence (Section 4.3, Figure 8) supports this: partially relevant retrieval achieves ρ_avg = 0.97 with overall performance, substantially higher than MSRVTT (ρ = 0.58) or other standard benchmarks. The finding is robust across model architectures — the pattern holds for both CLIP-based and MLLM-based models. However, the causal interpretation ("partially relevant retrieval better reflects universality") is based on correlation, not causation. It's possible that a third factor — overall model quality — drives both PR performance and general performance, and PR tasks happen to be the most discriminating (i.e., they have higher variance across models). The paper does not discuss this alternative interpretation. Additionally, the correlation analysis uses 14 data points (one per baseline model) for each correlation coefficient — with n=14, the confidence intervals on ρ values are wide, and the claim that PR has "the highest average correlation" may not be statistically distinguishable from the second-highest (FG, which also shows strong correlations). The paper reports no confidence intervals or significance tests for the correlation coefficients.

Genuine weaknesses in the experimental design:

• Test set sizes vary dramatically. LoVR-V has only 100 queries over 467 videos — statistical noise in this evaluation is high, yet it is weighted equally with MSRVTT (1,000 pairs) in the AVG of D metric. A model could appear 2-3 percentage points better or worse on AVG of D purely due to variance on small test sets. The paper does not address this or use weighted averaging.

• Single training run per configuration. All GVE variants (GVE-i, GVE-s, GVE) are trained once — there are no error bars on Table 3's numbers. The data scaling curves (Figure 6) include ±1σ bands but the source of variance is not specified. Given that contrastive training loss can be sensitive to random seed and batch ordering, the lack of multiple runs means small differences (e.g., the GVE-s → GVE improvement of 0.005 on AVG of D) could be within noise.

• No human evaluation of synthetic captions. UVRD's quality is validated only implicitly through downstream task performance. The paper provides no human-annotated quality scores, no comparison of synthetic captions to human-written captions on the same videos, and no measurement of hallucination rate in the MLLM-generated captions. The cross-modal consistency filter removes catastrophically misaligned original captions but does not verify the generated captions — a caption that passes the filter could still be hallucinated by the MLLM captioner.

• The data scaling analysis conflates data quantity with data diversity. As the training set grows, the model sees not just more pairs but more diverse pairs (UVRD is only in the full-scale data mixture, not in the smaller subsets). The scaling curves therefore mix the effects of "more examples" and "more types of examples." An experiment that controls for diversity (e.g., scaling UVRD alone while keeping the base mixture fixed) would better isolate the pure data quantity effect.

• No comparison to training exclusively on UVRD. The ablation in Table 3 compares GVE-i (no UVRD) to GVE-s (UVRD added to the base mixture) — this shows UVRD's marginal contribution but not whether UVRD alone would be sufficient. If UVRD alone produces similar performance, the base data mixture (8.8M web-video pairs) is unnecessary. If UVRD alone is insufficient, the base mixture provides essential scale. This distinction matters for practitioners deciding whether to invest in data synthesis or in curating larger web datasets.

• Missing baseline: a model trained on UVRD with a simple uniform sampling baseline but no Modality Pyramid. The GVE-s variant already fills this role (Table 3), but the paper does not explore whether a simpler alternative to the Modality Pyramid — e.g., fixed task weights tuned on a validation set, or a simple round-robin schedule — would match the curriculum's performance. The curriculum's conceptual novelty is high, but its practical benefit over simpler baselines is not established.

Experiments that would have strengthened the paper:

• Out-of-domain generalization: All UVRB datasets involve general-domain videos (movies, animations, everyday activities). Testing GVE on a held-out domain — e.g., medical videos, industrial inspection, sports analytics with specialized terminology — would test whether universality extends beyond the training distribution or merely interpolates within it.

• Scaling UVRD size independently of the base mixture: The paper shows data scaling for the combined mixture (Figure 6). An experiment that fixes the base mixture at full scale and varies UVRD size (e.g., 100K, 500K, 1.55M pairs) would isolate UVRD's scaling properties and reveal whether the current 1.55M is near-optimal or far below saturation.

• Ablation over the information profile distribution: The 60/30/10 temporal/spatial/theme split is claimed to be guided by UVRB diagnostics, but never ablated. Comparing this split to 40/40/20, 80/10/10, or a uniform 33/33/33 would validate (or refute) the claim that UVRB diagnostics meaningfully guide data synthesis decisions.

• Confidence intervals and significance tests for the correlation analysis (Section 4.3): With n=14 models, the correlation coefficients have substantial uncertainty. Reporting 95% confidence intervals (via bootstrap) and testing whether the PR-vs-MS correlation difference is significant would strengthen Finding 1 considerably. As reported, it's unclear whether ρ_PR = 0.97 is statistically distinguishable from ρ_FG = 0.92 (which also has high average correlation).

• Comparison to a retrieval-specific architecture: All baselines are either CLIP-based or MLLM-based. Comparing to a model specifically designed for universal retrieval — if one existed — would test whether GVE's gains come from the co-design framework or simply from using a stronger base architecture (Qwen2.5-VL) than competitors (Qwen2-VL for GME and Unite). The paper partially addresses this by including VLM2Vec-V2 and B3 which use different base models, but a head-to-head with the same base architecture and different training data would be the cleanest comparison.

Overall assessment of experimental support: The paper's central claims are supported by the reported experiments, but with important boundary conditions that the paper acknowledges to varying degrees. The claim of state-of-the-art zero-shot generalization (Claim 1) is well-supported with the caveat that GVE does not dominate every sub-task and the baseline comparison is asymmetric (zero-shot vs. potentially in-domain baselines). The claim that UVRD and the Modality Pyramid are independently beneficial (Claims 2-3) is supported by a clean ablation (Table 3) that shows both components matter, though the curriculum's contribution is modest (+1.8-3.1%) and its mechanism is not directly validated. The claim that partially relevant retrieval is a better universal proxy (Claim 4) is supported by correlation evidence but lacks statistical rigor (no confidence intervals, no significance tests, small n=14). The data scaling analysis is solid but conflates quantity and diversity. The test-time scaling analysis provides genuinely useful negative results (non-monotonic spatial scaling, composed degradation with more input density) that are not central claims but add practical guidance. The most significant experimental gap is the lack of statistical rigor throughout — single training runs, no confidence intervals on correlations, small test sets with unweighted averaging — which means some of the finer distinctions (particularly between GVE-7B and Unite-7B on specific abilities, and between different configurations in the ablation) may not be robust.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation (Information Profile Distribution) Is Heuristic and Not Directly Validated

The assumption or constraint. V-SynFlow's controlled conditional captioning uses a fixed information profile distribution — 60% temporal, 30% spatial, 10% theme/other (Section 3.2, Stage 2) — based on the diagnostic that temporal understanding is the most lacking capability in existing models. The paper states that UVRB diagnostics "guide our design of data synthesis," but the mapping from diagnostic findings (e.g., spatial-temporal decoupling in CLIP models, ρ_S-T = 0.12) to the specific 60/30/10 split is heuristic: the authors observe that temporal reasoning is weak, so they generate more temporal captions. There is no systematic exploration of what the optimal profile distribution would be, and no validation that the chosen split actually targets the right capability gaps with the right intensity.

The consequence. The entire V-SynFlow pipeline is driven by these information profiles — they determine what kind of captions the MLLM generates for each video. If the profiles are misweighted (e.g., too much temporal content crowds out spatial detail that is actually more transferable, or too little theme content underserves partially relevant retrieval), the resulting UVRD dataset may be suboptimal for universality. Worse, because the profile distribution is fixed and never ablated, there is no evidence that UVRB diagnostics actually improved the data synthesis over a simpler strategy (e.g., uniform 33/33/33, or matching the natural distribution of caption styles in human-written data). The claim that UVRB "guides" data synthesis is a design rationale, not a demonstrated causal relationship. A practitioner replicating V-SynFlow for a different domain would not know whether to use the paper's exact profile split or to derive their own from diagnostic analysis — the paper provides no methodology for translating diagnostic findings into profile weights.

What evidence exists in the paper. No ablation over the information profile distribution exists anywhere in the paper or appendix. The ablation in Table 3 tests UVRD as a whole against not using UVRD, but never compares different profile splits. The paper's diagnostic analysis (Section 4.3) establishes that temporal understanding is the weakest capability in existing models (Finding 2), and GVE-7B's strong temporal performance (T: 0.469 vs. Unite-7B 0.412, Table 2) is consistent with the temporal-heavy synthesis strategy, but this is correlational — it could be that any additional temporal training data would produce similar gains, or that the 60% weight is near-optimal, or that a 40/40/20 split would work better. The paper provides no way to distinguish these possibilities.

Mitigation status. Not addressed. The authors present the 60/30/10 split as "guided by UVRB's diagnostics" but never test alternatives or provide a principled method for setting profile weights. A future study could systematically vary the profile distribution and measure per-ability performance on UVRB, creating a response surface that maps profile weights to capability outcomes — this would transform the heuristic into a validated design tool. Without such evidence, the profile distribution is an educated guess embedded in the pipeline's architecture.

---

6.2 The Cost of Difficulty Estimation (Prober-Based Alignment Scoring) Is Not Accounted for in Training Efficiency Claims

The assumption or constraint. The Modality Pyramid curriculum requires estimating task alignment before each training epoch by running a prober model Ψ_t over N_p positive pairs per task (Section 3.3). For epoch 1, the prober is GME-7B — an 8.3B-parameter model whose forward passes add computational overhead. For epochs 2 and 3, the prober is the previous epoch's GVE checkpoint — smaller models but still requiring separate inference passes. This alignment estimation is not free, yet the paper's training cost accounting (Section 4.1: "32 NVIDIA A100 GPUs, each with 80GB of memory... overall batch size is at least 1024") does not include the prober's computation, nor does any ablation compare to a fixed schedule that would eliminate this overhead entirely.

The consequence. The Modality Pyramid's reported gains — approximately +1.8% to +3.1% relative improvement over uniform sampling (Table 3) — come with an unquantified computational cost. A practitioner deciding whether to implement the curriculum needs to know whether the performance improvement justifies the additional infrastructure complexity and GPU-hours of running a separate prober model before each epoch. If the alignment estimation takes, say, 10% of the total training time, the effective gain per GPU-hour may be near zero or negative — the same total compute could have been spent on more training steps with uniform sampling. Furthermore, the requirement to maintain a prober model (GME-7B for epoch 1) adds a dependency on an external model that may not be available or suitable for all base architectures or domains. The paper acknowledges the broader exploration-exploitation tradeoff of test-time difficulty estimation (Section 3.2) but does not extend this reasoning to the training-time cost of the prober.

What evidence exists in the paper. None. The paper does not report the computational cost of alignment estimation — neither the number of pairs N_p, nor the GPU-hours required, nor the wall-clock time added per epoch. The training dynamics figures (Appendix A.10, Figures 9-10) show loss and metric curves over steps, not wall-clock time, so the prober's overhead is invisible. The ablation in Table 3 compares GVE-s (uniform sampling) to GVE (Modality Pyramid) — this shows that the curriculum helps, but does not control for total FLOPs: GVE may have used more effective compute than GVE-s due to the prober overhead. A FLOPs-matched comparison (where GVE-s is trained for more steps to match GVE's total compute) would reveal whether the curriculum's gains are from better scheduling or simply from more computation.

Mitigation status. Not addressed. The authors present the Modality Pyramid as a training algorithm improvement without discussing its computational cost. A natural mitigation — which the paper does not explore — would be to use a lightweight prober (e.g., a distilled model, or a small subset of GVE's own parameters) or to estimate alignment less frequently (e.g., every N steps rather than every epoch). Alternatively, the prober's cost could be amortized by using the alignment estimation passes as additional training signal (e.g., using the prober's embeddings as distillation targets). The paper flags cost-related future work for test-time difficulty estimation (Section 3.2) but not for the training-time prober.

---

6.3 Generalization Is Assessed Only Within a Single Model Family (Qwen2.5-VL) and a Single Visual Domain (General Web Video)

The assumption or constraint. GVE is derived from Qwen2.5-VL (Bai et al., 2025), and all training and evaluation use general-domain web videos — movies, animations, everyday activities, vlogs (Section 3.2, Appendix A.4: PVD, InternVid-FLT, WebVid). The paper acknowledges in its limitations (Appendix A.14) that UVRB "does not include specialized domains such as medical, industrial, or surveillance videos, where visual semantics and query intent differ significantly from general-domain content." A second, unstated constraint: all results are specific to the Qwen2.5-VL initialization. The paper does not test whether the co-design framework — UVRB diagnostics → V-SynFlow synthesis → Modality Pyramid training — produces similar gains when applied to a different base architecture (e.g., LLaVA-based, InternVL-based, or a CLIP-based model adapted for MLLM-style training).

The consequence. A practitioner in a specialized domain (medical video retrieval, industrial inspection, sports analytics) cannot assume that the framework transfers. The synthetic captions generated by Keye-VL-8B (Stage 2 of V-SynFlow) may lack the domain-specific terminology and visual concepts needed for specialized retrieval — the captioner was not fine-tuned on domain-specific data. The Modality Pyramid's alignment estimation (which relies on cosine similarity of embeddings) may produce different task orderings for a base model with different pretrained representations, potentially making the curriculum less effective or even counterproductive. The paper's claim of "universality" — while explicitly scoped to the nine abilities in UVRB — may be misunderstood by readers as general-domain universality rather than truly cross-domain universality. Appendix A.14 is transparent about this gap, but it appears only in the appendix, not in the main paper's claims.

What evidence exists in the paper. The paper tests GVE exclusively on UVRB's 16 datasets, all drawn from general-domain benchmarks. The video classification experiment (Appendix A.13, Table 11) tests on Kinetics, UCF101, HMDB51, SSV2, and Breakfast — these are action recognition benchmarks that are closer to specialized domains but still within the general activity recognition space; GVE-7B's performance (0.526 mean) is strong but not dominant. The paper provides no out-of-domain evaluation (e.g., medical videos from surgical datasets, industrial inspection videos, egocentric video from EPIC-Kitchens). Regarding model family, all GVE variants use Qwen2.5-VL — there is no evidence that V-SynFlow or the Modality Pyramid would benefit a different base architecture, nor any discussion of what properties of Qwen2.5-VL (its vision encoder, its pretraining data mixture, its tokenizer) are load-bearing for the framework's success.

Mitigation status. Partially addressed via explicit acknowledgment in Appendix A.14, but no experiments mitigate the concern. The paper states that "efficient variants and training strategies are left for future work" (regarding computational requirements) and implicitly leaves domain generalization to future work by listing it as a limitation. The framework is designed for generality — UVRB is domain-agnostic in principle, V-SynFlow could ingest domain-specific videos, and the Modality Pyramid is task-format-agnostic — but none of these extensions are tested. A practitioner would need to replicate the entire pipeline on their domain to know whether it works.

---

6.4 Visual Perception Does Not Improve with Scale or Architecture, Creating a Hard Ceiling on Universal Capability

The assumption or constraint. The paper's Finding 4 (Section 4.3) is that "scaling has limited impact on visual perception" — low-level visual retrieval ability (matching images to videos or videos to videos based purely on visual content) does not improve with larger models, more data, or MLLM architectures. Specifically, CLIP4Clip (87M parameters, CLIP-based) achieves VIS = 0.714, outperforming Unite-7B (8.3B, MLLM-based) at VIS = 0.666 and GVE-7B (8.3B) at VIS = 0.657 (Table 2). The correlation between visual retrieval and overall universal performance is ρ_AVG-VIS = 0.26 — visual ability is essentially independent of general retrieval quality. GVE's training framework — more diverse data, better curriculum — does not move the VIS score beyond what a 87M CLIP model achieves.

The consequence. Visual retrieval acts as a capability ceiling that current training methods cannot breach. For applications where pure visual matching is critical — content-based video copy detection, near-duplicate identification, visual search without text queries — GVE is not an improvement over much smaller, simpler models. More importantly, because visual understanding underpins higher-order capabilities (spatial reasoning requires recognizing objects; temporal reasoning requires tracking objects across frames), the inability to improve visual perception through data diversity or curriculum design suggests a fundamental bottleneck: MLLM-based architectures may trade raw visual fidelity for semantic abstraction, a trade-off that scaling alone cannot resolve. The paper's correlation analysis (Finding 2, Figure 7) shows that spatial (S) and temporal (T) capabilities are still only moderately coupled even in MLLMs (ρ = 0.64), and temporal reasoning remains the weakest fine-grained ability (T: 0.469 for GVE-7B). If visual perception cannot be improved, temporal reasoning — which depends on perceiving visual changes — may also have a hard ceiling.

What evidence exists in the paper. The evidence is strong and consistent across all experiments. Table 2 shows VIS scores: CLIP4Clip 0.714, InternVideo2-6B 0.660, GME-7B 0.615, Unite-7B 0.702, GVE-7B 0.657 — no trend with parameter count or architecture class. Figure 8 shows ρ_AVG-VIS = 0.26 — near-zero correlation with overall performance. The data scaling curves (Appendix A.11, Figure 11, VIS panel) show the shallowest scaling of any ability for both GVE-3B (+3.8% per decade) and GVE-7B (+2.1% per decade). The test-time scaling experiments (Appendix A.12) show that increasing spatial resolution (tokens per frame) beyond ~400 actually degrades performance, suggesting the bottleneck is not input fidelity but representational quality. The paper does not explore why visual perception fails to scale — possible explanations (vision encoder frozen during fine-tuning, contrastive loss optimizing for semantic rather than perceptual similarity, MLLM pretraining trading visual detail for language alignment) are not investigated.

Mitigation status. Not addressed as a solvable problem. The paper reports Finding 4 as an empirical observation and notes that "future progress requires targeted improvements in visual grounding" (Section 4.3) but does not propose or test any mitigation — e.g., fine-tuning the vision encoder (currently kept frozen), adding perceptual losses beyond contrastive InfoNCE, using higher-resolution vision encoders, or ensembling with a dedicated visual matching model. This limitation is fundamental to the current approach: GVE is optimized for semantic cross-modal alignment, not visual fidelity, and the two objectives may be in tension. The paper's framework provides no mechanism to improve visual perception alongside semantic understanding, implying that a truly universal retriever may need to combine GVE-style semantic embeddings with a separate visual matching module — a hybrid architecture the paper does not explore.

---

6.5 Ablations Are Sparse and Do Not Establish Causality for the Core Design Decisions

The assumption or constraint. The paper makes several design claims that are supported by design rationale rather than controlled experiments: (1) UVRB diagnostics caused better data synthesis decisions (Section 3.2), (2) the specific 60/30/10 information profile split is optimal or near-optimal (Section 3.2, Stage 2), (3) the Modality Pyramid's annealing temperature schedule (σ_min=0.1, σ_max=1.0) is better than alternatives (Section 3.3), (4) the combination of all three components (UVRB + UVRD + Modality Pyramid) is greater than the sum of its parts in a way that pairwise combinations would not achieve. The ablation in Table 3 tests only two degrees of freedom: presence/absence of UVRD, and presence/absence of the Modality Pyramid. It does not test the quality of UVRB's diagnostic guidance, the parameters of the synthesis or curriculum, or the necessity of the full tripartite framework.

The consequence. A practitioner attempting to adapt this framework to a new domain faces substantial uncertainty. Without ablation over the information profile distribution, they don't know whether to invest effort in diagnostic benchmark construction (to derive domain-specific profile weights) or whether a fixed, domain-agnostic split works equally well. Without ablation over the Modality Pyramid's temperature schedule, they don't know whether the annealing specifically matters or whether any schedule that increased task diversity over time would match the performance. Without comparison to simpler baselines (e.g., fixed task weights, round-robin scheduling, loss-based weighting), they don't know whether the conceptual complexity of alignment-aware dynamic scheduling is necessary. The paper's central claim — that co-design of evaluation, data, and modeling is necessary for universality — is a strong causal statement that would require showing that deviating from the co-design (e.g., using a non-diagnostic benchmark to guide synthesis, or using a simpler training schedule) produces worse results. The current ablation only shows that adding UVRD and the Modality Pyramid helps over a baseline without them — it does not test whether UVRB's diagnostic structure specifically matters for the data synthesis decisions.

What evidence exists in the paper. Table 3 provides the only controlled ablation. It compares GVE-i (no UVRD, no curriculum) → GVE-s (with UVRD, no curriculum) → GVE (with UVRD, with curriculum). This establishes that both UVRD and the curriculum independently contribute, but does not establish that UVRB's diagnostics (as opposed to a simpler benchmark) were necessary for UVRD's quality, or that the curriculum's specific mechanism (as opposed to any non-uniform schedule) is responsible for the gains. The paper reports no ablation over: information profile distribution, temperature schedule (σ_min, σ_max, annealing rate), prober model choice (GME-7B vs. alternatives), hard negative count, LoRA rank, or number of synthetic captions per video. Each of these is a design degree of freedom where alternative choices could produce different — possibly better — results. The data scaling curves (Figure 6, Appendix A.11) show how performance varies with total data quantity but do not isolate the effect of data diversity (UVRD vs. base mixture composition) independently of data quantity.

Mitigation status. Not addressed. The paper does not acknowledge the sparsity of its ablation space as a limitation, and the strong causal language throughout ("guided by UVRB's diagnostics," "the pyramid provides a principled curriculum") implies a degree of validation that the experiments do not fully support. A more thorough ablation study — even on a subset of the design parameters — would substantially strengthen the paper's causal claims. At minimum, testing one alternative profile distribution (e.g., 40/40/20) and one alternative curriculum schedule (e.g., fixed temperature at 1.0, equivalent to uniform sampling with alignment estimation disabled) would establish whether the specific design choices matter or whether any reasonable instantiation of the framework works.

---

6.6 Composed Retrieval Remains Weak Across All Configurations and Does Not Benefit from the Curriculum

The assumption or constraint. Composed retrieval — queries combining text with an image or video reference — is one of the three core task formats in the UVR taxonomy (Section 3.1) and receives explicit training data from V-SynFlow (124K pairs: 89K TI2V + 35K TV2V, Table 7). The Modality Pyramid is designed to progressively introduce harder tasks (Section 3.3), and composed retrieval is presumably one of the "composite" tasks that should benefit from the pyramid structure. The paper positions composed retrieval as a key capability for universal video retrieval (Figure 1 shows a composed query example as a primary use case).

The consequence. Despite dedicated training data and the curriculum, composed retrieval performance remains low and appears resistant to improvement. GVE-7B scores CMP = 0.312 (Table 2), only marginally better than GVE-s-7B (0.313 without curriculum, Table 3) and GVE-i-7B (0.274 without UVRD). The 3B model's CMP score actually decreases slightly when the curriculum is added (0.301 → 0.304, within noise). On individual composed datasets (Table 1), GVE-7B does not lead: MS-TI P@1 = 0.343 (vs. GME-2B 0.350), MS-TV P@1 = 0.280 (vs. UniME-7B 0.305), LoVR-C2V R@1 = 0.415 (vs. UniME-7B 0.537). The test-time scaling experiments (Appendix A.12, Figures 12-13) show that composed retrieval is the only ability that degrades with both more frames and more tokens — adding visual information hurts rather than helps. This suggests a fundamental difficulty: the model cannot effectively bind the visual reference to the text query when the target video representation is rich, perhaps because the attention mechanism conflates the reference and target visual features.

What evidence exists in the paper. The evidence spans multiple experiments. Table 2 shows CMP as the lowest-scoring task format for all models (GVE-7B: TXT 0.657, VIS 0.657, CMP 0.312 — a gap of ~0.345). Table 3 shows the curriculum provides zero benefit for CMP (GVE-s-7B 0.313 → GVE-7B 0.312). Figure 11 (Appendix A.11) shows that CMP has the steepest data scaling curve for GVE-3B (+14.7% per decade) but relatively shallow for GVE-7B (+8.7%) — the 7B model appears to saturate on composed data more quickly, suggesting a capacity bottleneck or a representation limitation rather than a data limitation. The test-time scaling degradation (Figures 12-13) is consistent across both models and both types of scaling (frames and tokens). The paper does not analyze why composed retrieval is resistant to improvement — possible explanations include: the binding problem (how to associate specific visual features in the reference with the query text without interference from the target video), the scarcity of composed data (124K pairs is small relative to 879K text-video pairs), or a fundamental limitation of the single-vector embedding approach (composed queries may require structured representations that a single fixed-length vector cannot capture).

Mitigation status. Not addressed. The paper reports the CMP scores without discussing their weakness relative to other abilities. There is no analysis of composed retrieval failure modes, no qualitative examples of composed queries that GVE succeeds or fails on, and no proposed architectural modification to address the binding problem. The Modality Pyramid's inability to improve composed retrieval is a significant negative result — it suggests that the curriculum's progressive structure does not help for tasks that are fundamentally hard regardless of training stage — but the paper does not interpret it as such. A practitioner building a composed retrieval system would need to look beyond GVE's architecture; the paper provides no guidance on what alternative approaches (e.g., late fusion, cross-attention between query modalities, multi-vector representations) might address this limitation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the video retrieval field from a fragmented specialist paradigm — where models, benchmarks, and training data are developed independently within narrow task × domain cells — toward a unified, diagnostic-driven ecosystem where evaluation actively shapes data generation and training design. This is not merely a new benchmark or a new model; it is a methodological reframing of how to build general-purpose retrieval systems.

The magnitude of the shift is best understood by what the paper makes visible that was previously invisible. Before UVRB, a model's performance on MSRVTT and DiDeMo was treated as a proxy for "video retrieval ability." The paper's correlation analysis (Section 4.3, Finding 1) systematically demolishes this assumption: MSRVTT shows a Pearson correlation of only ρ = 0.58 with average universal performance. A researcher optimizing for MSRVTT — which describes essentially the entire field prior to this work — was optimizing for a metric that captured barely a third of the variance in genuine retrieval generality. This is not an incremental measurement improvement; it is evidence that the field's core evaluation signal has been misleading. The paper provides both the diagnosis and the alternative: partially relevant retrieval (PR) achieves ρ = 0.97 with universal performance, establishing a new, empirically validated target for what evaluation should measure.

Equally important is the paper's reconciliation of a latent tension in the literature: the puzzling transfer of image-trained models to video tasks. GME-7B, trained exclusively on text-image pairs (no video data whatsoever), achieves strong performance on UVRB's video retrieval tasks, outperforming several video-trained models on long-context and fine-grained retrieval (Table 2). This was an empirical curiosity before this paper — a surprising but unexplained observation. UVRB's ability-level correlation analysis (Section 4.3, Finding 2-3) provides the explanation: spatial understanding transfers to video because it serves as a building block for temporal reasoning (ρ_S-T = 0.64 in MLLMs, Figure 7), and MLLM architectures naturally couple these capabilities through their unified attention over multimodal token sequences. CLIP-based models, by contrast, show near-zero spatial-temporal correlation (ρ = 0.12) because their separate vision and text encoders never learn joint spatiotemporal representations. The paper thus explains why some models transfer and others don't — converting an anecdotal observation into a diagnostic principle that can guide architecture design.

The paper also redirects research attention toward data synthesis quality control as a first-class problem. V-SynFlow's multi-granular filtering stage — particularly the cross-modal consistency filter using GME-7B as an independent semantic gatekeeper — establishes a principled approach to preventing hallucinated training data that distinguishes this work from prior synthesis efforts. The finding that adding UVRD's synthesized data provides a 27% relative improvement in composed retrieval (Table 3, GVE-i-3B → GVE-s-3B) while adding virtually nothing to visual retrieval (VIS scores slightly drop) demonstrates that synthesis value is highly task-dependent — it helps most where existing data is absent, not where existing data is merely noisy. This reframes data synthesis from "generate more training data" to "diagnose which capabilities lack training signal, then synthesize targeted data for those capabilities." The diagnostic-driven synthesis loop — UVRB identifies gaps → V-SynFlow fills them → UVRB validates the improvement — is the operational instantiation of the co-design principle.

Perhaps the most consequential reframing is the paper's implicit argument that evaluation benchmarks should be diagnostic instruments, not leaderboard metrics. UVRB does not report a single number; it reports nine ability scores whose correlations reveal architecture-dependent trade-offs. This makes it impossible to "game" UVRB by overfitting to a single task — a model that excels on coarse-grained retrieval but fails on temporal reasoning will have its weakness exposed by the correlation structure. If the field adopts this diagnostic philosophy, the nature of benchmark construction changes: the goal is no longer to build a "harder" test set for the current task, but to build a structured test suite that reveals capability relationships. This is a higher bar — it requires benchmark designers to have hypotheses about which capabilities should and should not correlate, and to construct test sets that can discriminate those hypotheses — but it produces evaluation that is genuinely informative rather than merely competitive.

The work also elevates curriculum learning from a training heuristic to a diagnostically-informed optimization strategy. The Modality Pyramid is not a generic easy-to-hard schedule; it is a curriculum whose structure is derived from the capability dependencies revealed by UVRB's correlation analysis. The finding that foundational image-text alignment transfers to video (GME-7B's performance) becomes the design rationale for scheduling image-centric and coarse-grained video tasks before fine-grained temporal and composed tasks. The alignment-aware dynamic scheduling mechanism — using the model's own embeddings to estimate task difficulty via cosine similarity — provides a general recipe for multi-task contrastive training that is more principled than fixed task weights or loss-based scheduling because the difficulty metric is exactly the downstream evaluation metric.

What becomes more attractive as a research direction:

• Diagnostic benchmark construction for other retrieval domains (image retrieval, cross-modal retrieval, document retrieval) that measure capability correlations, not just aggregate scores.
• Targeted data synthesis driven by capability gap analysis — identifying which semantic dimensions are underspecified in training data and using conditional generation to fill those gaps, rather than generating data uniformly.
• Architecture-aware training that explicitly models capability dependencies — e.g., designing loss functions or modular architectures that encourage spatial and temporal representations to be jointly learned rather than independently optimized.
• Partially relevant retrieval as a primary evaluation paradigm — moving beyond literal matching benchmarks to fuzzy-query scenarios that better reflect real-world retrieval ambiguity.

What becomes less attractive:

• Incremental improvements on MSRVTT and DiDeMo as standalone contributions — the paper demonstrates these benchmarks are poor proxies for general ability, so marginal gains on them are likely overfitting rather than genuine progress.
• Uniform data scaling without diversity considerations — the paper's finding that visual retrieval barely scales with data (+2-4% per decade, Appendix A.11) while composed retrieval scales steeply (+9-15% per decade) implies that "more data" is not a uniform solution; what the data contains matters more than how much there is.
• Single-task specialized models for each retrieval format — the paper shows that a single GVE model with proper data and curriculum can match or exceed task-specific models (Unite-7B, UniME-7B) across most abilities, making the case for unified architectures over task-specific ones.

---

Follow-Up Research This Work Enables

Diagnostic benchmarks for composed retrieval that decompose the binding problem. GVE's composed retrieval scores (CMP: 0.312 for GVE-7B) are dramatically lower than its textual (0.657) and visual (0.657) scores, and composed retrieval is the only ability that degrades with more test-time input (Appendix A.12, Figures 12-13). This suggests a fundamental binding failure: the model cannot effectively associate specific visual features in a reference image with the text query when the target video representation is rich. A follow-up benchmark would systematically vary the type of binding required: identity binding ("the person FROM REFERENCE IMAGE now doing action X"), attribute binding ("the red jacket FROM PHOTO now in scene Y"), spatial binding ("the object FROM IMAGE now at location Z"), and temporal binding ("what happens next after REFERENCE CLIP"). By measuring performance on each binding type separately, such a benchmark would reveal which specific binding operations cause the failure — enabling targeted architectural interventions (e.g., cross-attention between query modalities, structured representations that separate identity from context, or modular encoders that process reference and query text in separate streams before fusing).

Scaling UVRD's composed data independently of other tasks to find the saturation point. The data scaling curves (Appendix A.11, Figure 11) show that composed retrieval (CMP) has the steepest scaling for GVE-3B (+14.7% per decade) but much shallower scaling for GVE-7B (+8.7% per decade), and the gap between GVE-s and GVE (Table 3) shows the curriculum provides essentially zero benefit for CMP (0.313 → 0.312). This suggests a capacity or representation bottleneck, not a data bottleneck — the 7B model may be saturating on the current 124K composed pairs. A direct experiment: fix the base training mixture (text-only, image-centric, video-centric data) at full scale, then independently scale UVRD's composed data from 12.4K to 124K to 1.24M pairs while holding UVRD's other tasks constant. This would reveal whether composed retrieval performance is limited by data quantity (in which case the 1.24M configuration should substantially outperform 124K), model capacity (in which case GVE-7B saturates while a hypothetical GVE-13B would continue improving), or architectural constraints (in which case no amount of data helps without changing the single-vector embedding approach). If composed retrieval saturates at ~124K pairs across model sizes, the implication is that the single-vector embedding paradigm is insufficient for compositional binding — a negative result that would motivate fundamentally different architectures for composed queries.

Ablation over the information profile distribution in V-SynFlow to validate the diagnostic-driven synthesis claim. The paper's central methodological argument is that UVRB diagnostics cause better data synthesis decisions, but the 60/30/10 temporal/spatial/theme split in V-SynFlow's controlled captioning is never ablated. A targeted experiment would train three GVE-3B variants on UVRD generated with different profile splits: the paper's 60/30/10, a uniform 33/33/33, and a counter-diagnostic 10/60/30 (minimizing temporal, maximizing spatial). Each variant would be evaluated on UVRB's per-ability scores. If the diagnostic-driven split produces the best temporal performance (T) without degrading spatial (S) — and the counter-diagnostic split degrades T while perhaps helping S — this validates the claim that UVRB diagnostics causally improve data synthesis. If the uniform split performs equivalently, the diagnostic guidance is decorative rather than functional. If the counter-diagnostic split unexpectedly helps (e.g., more spatial data improves temporal reasoning through better object tracking), this reveals a capability dependency that UVRB's correlation analysis missed. This experiment directly tests the co-design hypothesis that gives the paper its conceptual novelty.

A FLOPs-matched comparison of the Modality Pyramid against simpler scheduling baselines. The Modality Pyramid is conceptually elegant — alignment estimation via cosine similarity, temperature-annealed softmax sampling — but its practical benefit over simpler alternatives is not established. The prober model (GME-7B at epoch 1, GVE checkpoint thereafter) adds unquantified computational overhead. A controlled experiment would compare, at fixed total training FLOPs (including any prober overhead): (a) the Modality Pyramid as described, (b) fixed task weights tuned once on a small validation set and held constant, (c) a simple round-robin schedule that cycles through tasks in a fixed order, and (d) loss-based weighting where tasks with higher InfoNCE loss are sampled more frequently. All four variants would use identical data (UVRD + base mixture) and identical architecture (GVE-3B or GVE-7B). The key measurement: does the Modality Pyramid's dynamic, on-policy alignment estimation provide gains beyond what fixed or loss-based scheduling achieves at equivalent compute? If the simpler baselines match the curriculum's performance (currently +1.8-3.1% relative, Table 3), the curriculum's conceptual complexity is not justified by empirical benefit. If the curriculum substantially outperforms, the experiment reveals how much the alignment estimation matters and whether the prober's cost is worthwhile.

Testing GVE's framework on a domain-shifted video corpus to measure out-of-distribution generalization. UVRB's 16 datasets all involve general-domain web videos — movies, animations, vlogs, everyday activities. The paper acknowledges this as a limitation (Appendix A.14) but does not test domain shift. A concrete follow-up: apply the entire GVE framework (UVRB diagnostics, V-SynFlow synthesis with domain-specific source videos, Modality Pyramid training) to a specialized domain such as medical procedure videos (e.g., surgical footage from Cholec80 or JIGSAWS) or industrial inspection (e.g., manufacturing defect videos). The experiment would measure: (1) how well GVE's general-domain embeddings transfer zero-shot to the specialized domain (expected to be poor if domain-specific visual concepts are absent from training), (2) whether V-SynFlow can generate useful domain-specific training data when the MLLM captioner (Keye-VL-8B) has not been fine-tuned on domain terminology, and (3) whether the Modality Pyramid's task ordering generalizes to a domain where the capability hierarchy may differ (e.g., in medical video, spatial understanding of anatomical structures may be harder than temporal understanding of procedure steps). If the framework transfers successfully, it establishes genuine domain universality. If it fails, the failure mode (poor caption quality? wrong curriculum ordering? insufficient visual representation?) reveals which component is domain-dependent and needs adaptation — providing a research roadmap for making the framework truly domain-agnostic.

Investigating why visual retrieval does not scale and whether frozen vision encoders are the bottleneck. Finding 4 (Section 4.3) — that visual retrieval performance is essentially flat across parameter scales from 87M to 8.3B and across CLIP-based to MLLM-based architectures — is a significant negative result that the paper reports but does not explain or attempt to fix. A diagnostic experiment would systematically vary GVE's vision encoder configuration: (a) keep the vision encoder frozen (current approach), (b) fine-tune the vision encoder jointly with the LLM (via LoRA on vision transformer blocks), (c) replace Qwen2.5-VL's vision encoder with a higher-resolution or video-specialized encoder (e.g., ViT-G from DINOv2, or a VideoMAE-pretrained encoder), and (d) add an explicit perceptual loss (e.g., LPIPS or a self-supervised distillation loss from a frozen teacher) alongside the contrastive InfoNCE. Each variant would be evaluated on the VIS ability score (image-to-video and video-to-video retrieval) and on downstream tasks that depend on visual fidelity (spatial S, temporal T). If fine-tuning the vision encoder substantially improves VIS without degrading semantic abilities (TXT, CMP), the frozen encoder is the bottleneck and should be adapted. If improving VIS degrades semantic abilities, the paper has discovered a genuine trade-off between visual fidelity and semantic abstraction that constrains universal embedding — a finding that would motivate hybrid architectures with separate visual and semantic embedding streams.

---

Practical Applications and Downstream Use Cases

Content-based video search with heterogeneous query formats. A video platform (e.g., a stock footage library, an educational video archive, or a media asset management system) deploys GVE as its unified embedding model. Users can search using natural language ("sunset over mountain lake"), compose a reference image with text ("find videos matching this color palette but with ocean instead of lake"), or submit a short clip to find similar or temporally extended videos. Currently, supporting all three query formats requires maintaining separate specialized models — a text-to-video retriever, a composed retriever, and a visual similarity engine — each with its own embedding space and infrastructure. GVE's single embedding space handles all three formats (Table 2: TXT 0.657, CMP 0.312, VIS 0.657 for GVE-7B), reducing infrastructure complexity and enabling cross-format queries that would be impossible with separate models (e.g., "find videos similar to this clip but described by this text"). The 3B model (GVE-3B) enables on-device or edge deployment for privacy-sensitive applications (e.g., local video search on a user's device) while maintaining competitive performance — GVE-3B matches or exceeds the 8.3B Unite-7B on most abilities (Table 2: AVG of A 0.571 vs. 0.559).

Training data filtering and enrichment for video generation models. Text-to-video generation models (e.g., Sora, VideoPoet) require massive, high-quality text-video pairs for training, but web-crawled data is noisy and captions are often misaligned with visual content. V-SynFlow's multi-granular quality control stage — particularly the cross-modal consistency filter using GME-7B — provides a scalable, automated pipeline for filtering millions of web video-text pairs to retain only those with semantic alignment above a threshold. Stage 2's controlled captioning can then enrich the surviving pairs with detailed spatial, temporal, and thematic captions, replacing generic auto-generated text with MLLM-produced descriptions that are grounded in visual content (the MLLM sees the video frames). The resulting enriched dataset — filtered for quality and semantically diversified — would provide higher-quality training signal for video generation than raw web data. The paper's finding that UVRD's enriched data provides a 27% relative improvement in composed retrieval (Table 3) suggests that enriched captions contain visual detail that raw captions lack; this detail is equally valuable for generation, where precise text-video alignment determines output quality.

Multi-modal academic search over lecture videos. A university deploys GVE to index its lecture video archive. Students can search using: natural language queries about specific topics ("explain backpropagation through time with an example"), composed queries pairing a slide image with text ("find the lecture where THIS SLIDE was presented and the professor discussed convergence proofs"), or visual queries using a short clip to find related lectures. The partially relevant retrieval capability — identified by the paper as the best proxy for universal ability (ρ = 0.97, Section 4.3, Finding 1) — is critical here: a student's query about "convergence proofs" may match a lecture segment that discusses convergence only briefly within a broader optimization lecture, and a literal-matching retriever would miss it. GVE-7B's strength on long-context retrieval (LC: 0.814, Table 2) and partially relevant matching (PR: 0.419) makes it suitable for hour-plus lecture videos where relevant content may span only a few minutes. The framework's ability to handle text queries ranging from 6.5 words (DREAM-E) to 17,365 words (LoVR-V, Table 4) means it supports both brief keyword searches and long, detailed topic descriptions equally — matching the diverse ways students formulate academic queries.

Diagnostic auditing of deployed retrieval systems for capability gaps. An organization deploying a video retrieval system in production uses UVRB not as a training benchmark but as an auditing tool. By evaluating their deployed model on UVRB's 16 datasets and computing the ability-level scores and correlations, they can identify which specific capabilities their system lacks before users encounter failures. For example, if the correlation analysis reveals spatial-temporal decoupling (ρ_S-T near 0.12, the CLIP pattern from Section 4.3, Finding 2), they know their model cannot jointly reason about object locations and action sequences — temporal queries involving specific objects will fail. If composed retrieval shows inverse correlation with visual accuracy (ρ_CMP-VIS ≈ −0.71, the CLIP trade-off from Finding 3), they know improving visual fidelity will degrade composed query handling. These diagnostics directly inform remediation: the organization can use V-SynFlow to generate training data targeted at the identified gaps (e.g., if temporal reasoning is weak, bias the information profiles toward 80% temporal content) and retrain with the Modality Pyramid to integrate the new capabilities. The paper's demonstration that predicted difficulty bins (using the model's own scores) match oracle difficulty bins (Figures 4, 8 in the original revision experiments) suggests that this diagnostic process can be automated using the deployed model's own embedding quality estimates rather than requiring ground-truth labels for the target domain.

---

When to Prefer This Method

The paper does not explicitly frame GVE against a named alternative with a clear decision boundary — it positions the entire co-design framework (UVRB + V-SynFlow + Modality Pyramid) as a path to universality that prior specialist approaches fail to achieve. The implicit trade-off is between:

• Specialized models (e.g., separate CLIP-based retrievers for text-to-video, dedicated composed retrieval models like EgoCVR, task-specific fine-tuned variants) — which may achieve higher performance on their narrow target task but require maintaining separate infrastructure and provide no cross-task transfer.
• GVE's unified embedding — which achieves competitive or superior performance across 7 of 9 abilities (Table 2) with a single model but may underperform specialists on specific sub-tasks (e.g., UniME-7B leads on visual retrieval and LoVR-C2V; Unite-7B leads on partially relevant retrieval).

The paper's data and results suggest the following practical decision rule, though it is not stated as such:

• Prefer the GVE unified framework when the retrieval system must handle heterogeneous query formats (text, composed, visual) from a single API endpoint; when infrastructure simplicity outweighs per-task maximum performance; and when the problem distribution is unknown or shifting (because GVE's breadth provides robustness to query format variation that specialists lack). The 3B variant (GVE-3B) is specifically attractive when compute budgets are constrained — it outperforms the 8.3B Unite-7B on ability-averaged metrics (0.571 vs. 0.559, Table 2) at less than half the parameters.
• Prefer a specialized model when the deployment context requires only a single query format (e.g., pure text-to-video search on short clips where InternVideo2-6B's MSRVTT performance of 0.485 exceeds GVE-7B's 0.464, Table 1) and the target domain closely matches the specialist's training distribution. The paper's Finding 1 — that MSRVTT performance correlates poorly with generality (ρ = 0.58) — implies that specialist advantages on narrow benchmarks may not transfer to real-world query variation, so this preference holds only when the deployment query distribution is certifiably narrow.
• Prefer additional pretraining (scaling model size or training data for a specialist) when visual perception is the primary requirement — the paper's Finding 4 shows that visual retrieval ability (VIS) does not improve with GVE's approach (Table 2: CLIP4Clip 0.714 > GVE-7B 0.657) and is essentially uncorrelated with other capabilities (ρ_AVG-VIS = 0.26). For pure image-to-video or video-to-video similarity search, simpler, visually-optimized architectures remain preferable.