Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking

ArXiv: 2601.04720

🎯 Pitch

Distilling a cross-encoder reranker into an embedding model boosts retrieval but catastrophically breaks classification—until model merging reveals that you can have both. Qwen3-VL-Embedding-8B claims the top MMEB-V2 spot by resolving this trade-off, while the companion 8B reranker adds another 4.1 points over its 2B counterpart.

1. Executive Summary

This report introduces the Qwen3-VL-Embedding and Qwen3-VL-Reranker model series, a unified framework that maps diverse modalities—text, images, visual documents, and video—into a shared representation space for multimodal retrieval. The Qwen3-VL-Embedding model employs a multi-stage training paradigm (progressing from large-scale contrastive pre-training through reranker-model distillation, then merging checkpoints to balance retrieval gains against classification/QA performance) and the Qwen3-VL-Reranker adopts a cross-encoder architecture for fine-grained relevance estimation (computing a relevance score via the logit difference between "yes" and "no" tokens). Qwen3-VL-Embedding-8B achieves a score of 77.8 on the MMEB-V2 benchmark, ranking first among all models (6.7% above the prior best open-source model), while the reranker improves results by 4.1 points over its 2B counterpart. The multi-stage design establishes that distilling a reranker into the embedding model substantially boosts retrieval-oriented tasks—at the cost of degrading classification and QA—but that this trade-off can be resolved through model merging, yielding balanced state-of-the-art performance across all evaluation domains.

2. Context and Motivation

The Core Problem: Moving Beyond Single-Modality Search in a Multimodal World

The fundamental challenge this paper addresses is deceptively simple to state but enormously complex in practice: how do you build a retrieval system that can find relevant information regardless of what format the query or the stored documents happen to be in? A user might ask a question in text but need an answer contained in a video segment. A developer might search with an image to find similar products described only in text. A researcher might query a scientific paper's chart with natural language. The traditional approach—building separate retrieval pipelines for each modality combination (text-to-text, text-to-image, image-to-text, text-to-video, etc.)—creates a fragmented, expensive, and ultimately brittle system that cannot handle the fluid, multi-format nature of real-world information.

This problem has become critically important because the internet has undergone a fundamental shift in its content composition. The paper frames this in Section 1 by noting that "modern digital ecosystems are increasingly populated with diverse data modalities, including natural images, text documents, infographics, screenshots, and videos." This isn't a niche concern—it affects e-commerce product discovery (where products are described in text but searched via images), scientific literature exploration (where papers contain interleaved text, tables, figures, and equations), and social media navigation (where content mixes images, video clips, and text captions in arbitrary combinations). The paper explicitly cites these as representative application domains (Section 1, paragraph 2).

The technical difficulty stems from what the paper calls the need for a unified multimodal representation space (Figure 1): a mathematical space where semantically similar content is positioned close together regardless of its modality—where the text "urban architecture" and a photograph of a city skyline produce nearly identical vector representations. Building such a space requires solving several sub-problems simultaneously: cross-modal alignment (making text and images comparable), fine-grained visual understanding (distinguishing between subtly different visual concepts), multi-page document comprehension (understanding how text, charts, and layout interact in visual documents), and temporal reasoning (tracking events across video frames).

What Prior Approaches Failed to Deliver

The paper surveys three broad families of prior work, each with significant limitations that motivate the Qwen3-VL-Embedding approach.

CLIP-style dual-encoder models (Radford et al., 2021) demonstrated that large-scale contrastive learning on image-text pairs can produce powerful aligned representations. CLIP and its descendants learn by pushing matched image-text pairs together in embedding space while pushing mismatched pairs apart. These models are elegant, fast at inference (they can pre-compute corpus embeddings), and well-understood. However, their architecture has fundamental constraints. CLIP-style models use separate encoders for each modality (typically a vision transformer for images and a text transformer for text), which means any interaction between modalities happens only through the final similarity computation (cosine similarity between independently produced vectors). There is no mechanism for the model to perform cross-attention between image regions and text tokens—the kind of fine-grained alignment that allows a system to understand that "the red car in the upper left corner of the image" corresponds to a specific visual element. This makes CLIP-style models particularly weak on tasks requiring detailed visual reasoning, multi-page document understanding, or comprehension of charts and infographics where text and visual layout are deeply intertwined.

ColPali-style visual document models (Faysse et al., 2025; Team, 2025; Xu et al., 2025) represent a more recent approach specifically designed for document retrieval. These models treat document pages as images and use vision-language models to produce patch-level embeddings that capture the spatial layout of text and visual elements. The paper benchmarks against several such models in Table 3 (llama-nemoretriever-colembed, colnomic-embed-multimodal, colqwen2.5, tomoro-colqwen3). While these models achieve strong performance on visual document retrieval tasks, they have a critical limitation: they are specialized for document understanding and do not generalize well to general image retrieval, video retrieval, or text-only tasks. A production system using ColPali-style models for documents would still need separate models for text search, image search, and video search—reproducing the fragmented architecture problem.

Prior VLM-based unified embedding attempts (E5-V by Jiang et al., 2024; GME by Zhang et al., 2025b; BGE-VL by Zhou et al., 2025; VLM2Vec by Meng et al., 2025 and Jiang et al., 2025) took a step toward unification by building embedding models on top of pretrained vision-language models. VLMs have inherent advantages for this task: they are pre-trained on large-scale image-text datasets, giving them cross-modal alignment from the start; they use sophisticated attention mechanisms that can capture fine-grained interactions between visual and textual elements; they can naturally handle complex multimodal documents where visual and textual information are intertwined; and they inherit the extensive multilingual and multi-domain knowledge encoded in their foundation model training. However, as the paper's benchmark results in Table 2 make clear, these prior VLM-based models show clear performance gaps. The best prior open-source model on MMEB-V2 at the time of the paper's evaluation was RzenEmbed-7B at 72.9 (Table 2, "All" column). Qwen3-VL-Embedding-8B reaches 77.8—a 6.7% improvement. The gap is particularly stark in video tasks, where the best prior closed-source model (Seed-1.6-embedding-1215) achieves 67.7 overall on video while Qwen3-VL-Embedding-8B achieves 67.1—essentially matching it—but the best prior open-source model (RzenEmbed-7B) reaches only 55.7, a gap of over 11 points. This suggests that prior VLM-based approaches struggled specifically with the temporal reasoning demands of video understanding.

Text-only embedding models like Qwen3-Embedding (Zhang et al., 2025c), BGE-M3 (Chen et al., 2024), and GritLM (Muennighoff et al., 2024) represent another strand of prior work. These models achieve strong results on text retrieval benchmarks (Table 4 shows Qwen3-Embedding-8B at 70.6 on MMTEB versus Qwen3-VL-Embedding-8B at 67.9). The slight degradation of the multimodal variant on pure text tasks (a drop of 2.7 points on the mean task score) is acknowledged by the paper as a limitation, but the authors frame it as an acceptable trade-off for gaining multimodal capability. The critical point is that text-only models simply cannot handle non-text queries or documents—they are fundamentally single-modality systems.

The Gap: No Unified, High-Performance, Deployment-Ready System

The paper identifies a specific gap that none of the prior approaches fill. It's not enough to have a model that works on text, another that works on images, and a third that works on documents—that reproduces the fragmented architecture the paper aims to replace. It's also not enough to have a unified model that achieves mediocre performance across modalities. What the paper claims to deliver—and what the benchmark results in Table 2 are designed to demonstrate—is a single model series that simultaneously achieves:

1. State-of-the-art performance across all four modalities (text, image, visual document, video) on the comprehensive MMEB-V2 benchmark, ranking first overall among both open-source and closed-source models.
2. Competitive text-only performance that, while slightly below specialized text embedding models, remains strong enough to serve as a single deployment solution (67.9 on MMTEB versus 70.6 for the specialized Qwen3-Embedding-8B).
3. Practical deployment characteristics through Matryoshka Representation Learning (allowing users to trade embedding dimension for storage and speed without retraining) and quantization-aware training (enabling int8 or binary precision embeddings).
4. An integrated reranking capability through a companion cross-encoder model that can refine retrieval results with fine-grained relevance scoring—the paper argues that reranking is necessary because even the best bi-encoder embeddings lose some precision compared to cross-attention between query and document.

How This Paper Positions Itself: A Unified Training Pipeline, Not Just a Model

The paper's central positioning move is to argue that the key to building a unified multimodal retrieval system is not just architectural—it's about training methodology. The introduction (Section 1) explicitly states that VLMs provide a strong foundation because they "possess inherent cross-modal alignment" and "sophisticated attention mechanisms," but these capabilities alone are insufficient. The contribution is the multi-stage training pipeline (Figure 5) that progressively refines the base VLM's representations:

• Stage 1 (Contrastive Pre-training) uses 300 million synthesized examples (Section 4.1) to teach the model the basic task of multimodal relevance judgment at scale. The synthetic data comes from a seed pool of carefully curated and labeled images and videos (Figure 4 shows the distribution across natural images, artificial/man-made imagery, artistic content, and various video categories) that are annotated using Qwen3-VL-32B itself. This stage produces a model with broad but somewhat noisy retrieval capability.

• Stage 2 (Multi-Task Contrastive Learning + SFT) switches to higher-quality curated data (40 million examples, combining public datasets and proprietary in-house data) and uses the Stage 1 model to perform data mining—identifying hard negatives and refining positive pairs via the recall-and-relevance-filtering pipeline described in Section 3.3. This stage also trains the separate Qwen3-VL-Reranker on retrieval-specific data.

• Stage 3 (Distillation + Model Merging) is where the paper's most distinctive technical idea appears. The authors distill the reranker's fine-grained relevance discrimination into the embedding model (using the cross-entropy distribution-matching objective in Equation 3). This substantially boosts retrieval performance—Table 6 shows retrieval-oriented tasks (VDR, VR) jump from 77.1 at stage s1 to 80.9 at stage s2 for the 2B model. However, classification and QA performance degrade (the "All" image score drops from 74.8 to 71.3). The solution is model merging: combining the s2 checkpoint (strong at retrieval) with the s1 checkpoint (strong at classification/QA) using the methodology from Li et al. (2024). The merged model s3 achieves balanced performance: 75.0 on images, 61.9 on video, 79.2 on VisDoc, and 73.2 overall—better than either s1 or s2 alone.

This merging-as-resolution-of-task-conflict framing is what distinguishes the paper from prior work that also used multi-stage training. The paper doesn't just observe that distillation helps retrieval but hurts other tasks—it provides a mechanism to recover the lost performance, treating the embedding model's capabilities as something that can be composed from checkpoints optimized for different objectives.

Why This Matters Beyond Benchmark Numbers

The paper's significance extends beyond topping the MMEB-V2 leaderboard. The analysis of embedding efficiency (Section 7.1, Figure 6) demonstrates that the Matryoshka and quantization techniques make the model practically deployable at scale. The paper reports concrete numbers: in text retrieval on MS MARCO, reducing the embedding dimension from 1024 to 512 drops MRR@10 by only 1.4% while achieving 50% storage reduction and 2× retrieval speed (inference time drops from 43ms to 12ms, memory from 32,539MB to 8,135MB). For int8 quantization, performance is essentially preserved. For cross-modal retrieval on VL3-Syn (2 million images), the same pattern holds: 1024→512 dimension reduction costs only 2% in MRR@10 (0.497→0.487) while reducing memory from 7,812MB to 1,953MB and latency from 2.87ms to 0.94ms. These numbers make the case that the model isn't just accurate—it's deployable in production retrieval systems with millions or billions of items.

The multilingual support (30+ languages, inherited from Qwen3-VL) and the release of both 2B and 8B parameter sizes further position the series for diverse deployment scenarios, from on-device applications to large-scale cloud retrieval services.

3. Technical Approach

3.1 Reader Orientation

This paper presents a unified embedding-and-reranking pipeline built on top of the Qwen3-VL vision-language foundation model, where an embedding model (bi-encoder) produces dense vector representations for any combination of text, images, visual documents, and video, and a separate reranking model (cross-encoder) performs fine-grained relevance scoring on query-document pairs. The system solves the problem of multimodal search at scale: given a query in any modality (text, image, video, or a mix) and a corpus of documents in any modality, it retrieves the most relevant documents through fast approximate nearest-neighbor search in a shared embedding space, then optionally refines the top candidates with the slower but more precise cross-encoder reranker. The "shape" of the solution is a multi-stage training strategy that progressively refines the base VLM's representations—from coarse contrastive learning on massive synthetic data, through high-quality multi-task training with mined hard negatives, to a final distillation stage where the reranker's fine-grained relevance judgment is transferred into the embedding model, with model merging used to reconcile the trade-off between retrieval quality and performance on classification/question-answering tasks.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, arranged as both a training pipeline (Figure 5) and an inference architecture (Figure 2):

1. Qwen3-VL Foundation Model (backbone) — A pretrained vision-language model that processes interleaved text, images, and video through a unified causal-attention transformer. It serves as the shared initialization for both the embedding model and the reranker. Key specifications (Table 1): the 2B variant has 28 layers, the 8B variant has 36 layers; both support 32K token context windows.

2. Qwen3-VL-Embedding Model (bi-encoder) — Takes a multimodal input (text tokens, vision tokens from images or video frames, or both), appends a special <|endoftext|> token, and extracts the last hidden state at that token's position as a single dense vector. This vector represents the input in a unified embedding space where cosine similarity measures semantic relevance. Supports Matryoshka Representation Learning (variable output dimensions) and quantization-aware training. Produces embeddings of dimension 2048 (2B) or 4096 (8B).

3. Qwen3-VL-Reranker Model (cross-encoder) — Takes a query-document pair as a single concatenated input and processes them through full cross-attention in the transformer layers. Computes a relevance score by extracting the logits for the special "yes" and "no" tokens and applying sigmoid(logit(yes) - logit(no)). This is slower than embedding-based retrieval (it cannot pre-compute document representations) but provides more accurate relevance estimates because the model can attend to fine-grained interactions between query and document tokens.

4. Data Synthesis Engine — A pipeline (Section 3.2) that uses Qwen3-VL-32B as an annotator to generate 300 million training examples from a curated seed pool of images and videos. Produces classification, question-answering, and retrieval tasks across all modalities. The seed pool is constructed through quality filtering, scene-cut detection for videos, fine-grained category labeling, and cross-modal alignment filtering using the GME embedding model.

5. Hard Negative Mining Pipeline — An automated two-stage process (Section 3.3) that uses an embedding model to retrieve top-K candidates for each query, then refines positive pairs (discarding queries where no positive exceeds a score threshold) and selects hard negatives (candidates whose similarity scores are close to but below the average positive score). This pipeline is applied iteratively: the Stage 1 model improves data quality for Stage 2, and the Stage 2 model could theoretically improve data for further iterations.

Information flows through the training pipeline as follows: raw seed images/videos → Qwen3-VL-32B annotation (producing 300M synthetic examples) → Stage 1 contrastive pre-training (producing model s0) → s0 used for hard negative mining on curated data → Stage 2 multi-task contrastive learning (producing model s1) + reranker training → reranker used to score a distillation subset → Stage 3 distillation training (producing model s2) → model merging of s1 and s2 (producing final model s3). At inference, a query enters the embedding model (with optional task-specific instruction), produces a dense vector, and the system retrieves the top-K corpus documents by cosine similarity; optionally, the reranker then re-scores these candidates with cross-attention.

3.3 Roadmap for the Deep Dive

• First, the embedding method and reranking method as inference-time procedures — what happens when you actually use these models, including the exact input templates, token processing, and how vectors/scores are extracted. This grounds everything that follows in concrete operational terms.
• Second, the data synthesis engine — the seed pool construction, the annotation prompts, and the task taxonomies for images and video. Understanding the data is essential because the multi-stage pipeline's effectiveness depends entirely on what the model is trained on.
• Third, the hard negative mining pipeline — the recall-and-relevance-filtering mechanism that transforms noisy data into high-quality training examples, since this is what enables the progressive refinement across stages.
• Fourth, the three-stage training strategy in detail, with explicit loss functions for each data type (retrieval, classification, STS, distillation) and the reranker's binary classification loss. This is the core technical contribution.
• Fifth, the efficiency techniques (Matryoshka Representation Learning and Quantization-Aware Training) as auxiliary objectives integrated into the training process.
• Sixth, the model merging procedure that resolves the retrieval-vs-classification trade-off introduced by distillation, since this is the paper's distinctive solution to a known problem.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that a unified multimodal embedding model can be built by (a) synthesizing massive diverse training data from a VLM annotator, (b) progressively refining that data through embedding-model-based hard negative mining, and (c) transferring a reranker's fine-grained relevance judgment into the embedding space through distillation, then (d) merging checkpoints to recover performance on non-retrieval tasks lost during distillation.

Embedding Method: How Vectors Are Extracted at Inference

The embedding model operates as a bi-encoder: it processes a single input (query or document) independently and produces one dense vector per input. This independence is what makes it fast for retrieval—all documents in a corpus can be pre-encoded offline, and search reduces to nearest-neighbor lookup in embedding space.

Input format. The model uses the Qwen3-VL chat template structure with three messages. The paper provides the exact template:

<|im_start|>system
{Instruction}
<|im_end|>
<|im_start|>user
{Instance}
<|im_end|>
<|im_start|>assistant
<|endoftext|>

The {Instruction} is passed as a system message and defines the task context. The default instruction when none is specified is "Represent the user's input." This instruction-aware design (Table 1 confirms all models support it) means the same query can produce different embeddings depending on the task—for example, a query about "jaguar" could be embedded differently if the instruction says "Classify the animal" versus "Retrieve information about the car brand."

The {Instance} is passed as a user message and can be text, images, video, or any combination of these modalities. For images, the model preserves the original aspect ratio but caps the maximum token consumption at 1,280 visual tokens (approximately 1.3 million pixels, as stated in Section 4.2). For video, the model samples at 1 frame per second with a maximum of 64 frames, maintaining aspect ratio per frame, with a total token budget across all frames constrained to 4,500 visual tokens (approximately 9.2 million pixels). These caps are training-time values; at evaluation, the paper uses different limits: 1,800 tokens for image tasks, 15,000 total tokens and 64 frames for video tasks, and 16,384 total context tokens across all modalities (Section 6.1).

Vector extraction. After the multimodal tokens are processed through the full transformer stack, the model appends a <|endoftext|> token (the standard end-of-sequence token, also referred to as "PAD" in the paper's notation). The last hidden state corresponding to this <|endoftext|> token is taken as the dense vector representation of the entire input instance. This is a standard "pooling" technique used in decoder-only LLM-based embedding models (also used in Qwen3-Embedding): rather than averaging all token representations (mean pooling) or taking the first token (CLS pooling), the model is trained to compress all relevant information into the representation at the final token position.

Why this pooling choice. The paper inherits this design from Qwen3-Embedding (Zhang et al., 2025c) and from the broader convention in causal-decoder embedding models. Mean pooling across all tokens would give equal weight to padding tokens and visual patch tokens that may carry little semantic content. CLS pooling would require a special token added at the input start, which doesn't naturally exist in a causal decoder architecture designed for autoregressive generation. The <|endoftext|> token pooling leverages the causal attention mechanism: since this token attends to all previous tokens in the sequence (it's the last position), its hidden state can theoretically capture a summary of the entire input through the self-attention operations in the final layers.

Similarity computation. Relevance between two embedded instances is measured by cosine similarity: cos(e_q, e_d) = e_q · e_d / (||e_q|| · ||e_d||). This is the standard metric in embedding-based retrieval because it normalizes for vector magnitude, making the similarity score independent of embedding norm (which can vary with input length or content).

Reranking Method: How Cross-Attention Scores Are Computed

The reranker operates as a cross-encoder: it takes both the query and the candidate document together as a single input, processes them jointly through transformer layers with full cross-attention between query tokens and document tokens, and produces a single relevance score. This is fundamentally different from the bi-encoder—it cannot pre-compute document representations because the document encoding depends on the specific query it's being compared against. The trade-off is accuracy for speed: cross-attention captures fine-grained interactions (e.g., whether a specific phrase in the query matches a specific region in an image) that are lost when query and document are embedded independently.

Input format. The input follows the Qwen3-VL chat structure, but unlike the embedding model, both the instruction and the query-document pair are passed as user messages:

<|im_start|>system
Judge whether the Document meets the requirements based on the Query and the Instruct
provided. Note that the answer can only be "yes" or "no".
<|im_end|>
<|im_start|>user
<Instruct>: {Instruction}
<Query>: {Query}
<Document>: {Document}
<|im_end|>
<|im_start|>assistant

The system message embeds a fixed prompt that constrains the model to binary relevance judgment. The user message includes three tagged fields: the {Instruction} (the same task description used for the embedding model), the {Query} (the multimodal query), and the {Document} (the candidate multimodal document). This format tells the model exactly what to compare and by what criteria.

Scoring mechanism. The paper frames reranking as binary classification. The model is trained to predict either the token "yes" (the pair is relevant) or "no" (the pair is not relevant) as the next output token. At inference, the relevance score is computed by extracting the raw logits (pre-softmax values) for both the "yes" and "no" tokens from the model's output head at the final position, then applying:

$s = \text{sigmoid}(\text{logit}(\text{yes}) - \text{logit}(\text{no}))$

where logit(yes) is the unnormalized score the model assigns to the "yes" token, logit(no) is the unnormalized score for the "no" token, and sigmoid(x) = 1/(1 + e^{-x}) squashes the difference into the range (0, 1).

What it computes: the model first computes two raw logits—one for each possible answer token—based on the full cross-attention processing of the concatenated query and document. The difference logit(yes) - logit(no) measures how much more strongly the model "believes" in relevance versus irrelevance. Passing this difference through a sigmoid converts it into a probability-like score where 0.5 represents indifference (equal logits), values near 1.0 represent strong relevance, and values near 0.0 represent strong irrelevance.

Why this form: using logit differences rather than softmax probabilities avoids the calibration issues that arise when the model's probability distribution over the two tokens is sharp or flat for reasons unrelated to relevance. The sigmoid of the logit difference is equivalent to the softmax probability of "yes" when there are exactly two classes, but it's numerically more stable and directly interpretable as a relevance score. The alternative—using the raw softmax probability of "yes"—would produce scores that are always near 0.5 when the model is uncertain, while the logit-difference formulation can produce scores at the extremes even when the absolute logits are small, as long as the relative difference is large. This matters for ranking because relative ordering (is document A more relevant than document B?) is more important than absolute probability calibration.

Data Synthesis Engine: Building 300M Training Examples from a Seed Pool

The paper's data strategy rests on a crucial observation made explicit in Section 3: "both publicly available and proprietary in-house data exhibit significant imbalances and, in specific scenarios, notable scarcity across these dimensions." In other words, existing datasets are skewed toward certain modalities (text-to-text retrieval has abundant data, text-to-video retrieval has much less), certain tasks (retrieval is common, classification is less common in multimodal paired data), and certain domains (natural images dominate, visual documents and screenshots are underrepresented). Rather than attempting to collect balanced data—which would be prohibitively expensive—the paper synthesizes balanced data using a strong VLM as an annotator.

Seed pool construction (five-stage filtering). The synthesis process begins with raw image and video datasets. The paper describes a multi-step curation pipeline (Section 3.2, "Seed Pool Construction"):

1. Coarse-grained quality filtering: assets with low resolution or irregular aspect ratios are pruned. This eliminates inputs that would be too small for meaningful visual understanding or too distorted for the VLM annotator to process reliably.

2. Structural refinement for video: scene cut detection is applied to identify natural boundaries in video content, and static or corrupted segments are removed. This ensures that the temporal dynamics used in subsequent annotation are genuine content transitions rather than encoding artifacts or stationary camera footage.

3. Fine-grained category labeling: Qwen3-VL-32B (the largest available VLM in the Qwen family at the time) is used to generate detailed categorical labels for all remaining assets. The resulting category distribution is shown in Figure 4. For images, categories include natural images (portraits, landscapes, animals, plants, food, architecture, indoor scenes, objects), artificial/man-made content (documents, illustrations, UI/screenshots), and artistic/synthetic content (3D renders, art). For video, categories span human-centric content (interviews, performances, sports, daily activity), media/entertainment (movie clips, gaming, animation, news), and nature/scenery (wildlife, time-lapse, natural phenomena).

4. Cross-modal alignment filtering: this is a critical quality control step. The paper uses the GME embedding model (Zhang et al., 2025b) to compute similarity scores between visual content and its associated text descriptions. Samples with low-confidence annotations or "poor visual-text correspondence"—meaning the text description doesn't match what's actually in the image or video—are excluded. This prevents the synthesized training data from teaching the model incorrect cross-modal associations.

5. Category-wise rebalancing: the refined dataset is resampled to ensure balanced representation across categories. Without this step, the natural tendency for some categories (e.g., portraits, landscapes) to be overrepresented in raw data collections would bias the model toward those domains.

Annotation protocol (two-step approach). Before generating task-specific annotations, the paper requires Qwen3-VL-32B to first produce a descriptive caption for each image or video (Section 3.2, final paragraph before the task descriptions). This two-step approach—caption first, then annotate—is described as ensuring "higher quality and consistency in the subsequent annotation generation." The rationale, though not explicitly spelled out in the paper, is that producing a detailed caption forces the model to fully process the visual content before generating task-specific labels; without this intermediate step, the model might attend only to superficial features relevant to the specific task instruction and miss details that would improve annotation quality.

Image task synthesis (three paradigms). The paper describes three task types for image data (Section 3.2, "Image Tasks Annotation"):

1. Image Classification: The query q consists of an image paired with a classification instruction (e.g., "What object is in this image?"), and the document d is a text category label. For each sample, the annotator model selects a specific task type (object recognition, scene parsing, landmark identification, or action recognition), annotates the image with its ground-truth category, and generates a "semantically confusing negative label"—a plausible but incorrect category designed to serve as a hard negative during contrastive training. For example, for an image of a wolf, the positive label might be "wolf" and the hard negative might be "husky" or "coyote" rather than an obviously unrelated label like "car."

2. Image Question Answering: The query q consists of an image paired with a "grounded question"—a question whose answer is directly derivable from the visual content. The document d is the text answer. The annotator is instructed to generate QA pairs across four subtypes: factoid identification (e.g., "What color is the car?"), visual reasoning (e.g., "How many people are sitting at the table?"), OCR-based data extraction (e.g., "What is the title of the book in the image?"), and domain-specific knowledge inquiry (e.g., "What architectural style is this building?"). For each pair, the model provides a correct answer and a "plausible but deceptive distractor"—a wrong answer that someone might reasonably give if they misunderstood the image.

3. Image Retrieval: The query q is a text search description, and the document d is the candidate image. The annotator generates retrieval queries across a "hierarchy of semantic depths": direct visual descriptions (e.g., "a red bicycle leaning against a brick wall"), abstract narrative scenarios (e.g., "a commuter's morning routine"), compositional logical constraints (e.g., "a kitchen with both a gas stove and a window over the sink"), and knowledge-centric textual localization (e.g., queries about text content visible in the image). The model assigns a specific retrieval intent and generates a search query that captures "either the salient visual features or the embedded textual logic within the image."

Video task synthesis (four paradigms). The video annotation adds temporal reasoning dimensions (Section 3.2, "Video Tasks Annotation"):

1. Video Classification: Analogous to image classification but for video content. Task subtypes include activity recognition, scene parsing, event categorization, and sentiment/intent analysis. The annotator provides a correct label and a semantically related negative label.

2. Video Question Answering: Query is a video plus a question; document is a text answer. Subtypes include factual identification, temporal grounding (questions about when something happened), thematic reasoning (questions about the video's overall message or theme), and cinematic analysis (questions about editing, camera work, or narrative structure). Again, a correct answer and a deceptive distractor are generated.

3. Video Retrieval: Query is a text description; document is the video. Queries span a "spectrum of semantic granularities" from entity/action-centric searches (e.g., "person playing guitar") to temporal-event descriptions (e.g., "a goal being scored during a soccer match"), thematic/emotional discovery (e.g., "a video that conveys a sense of peaceful solitude"), and instructional tutorial localization (e.g., "how to fold a fitted sheet").

4. Moment Retrieval: This is a fine-grained temporal grounding task not present in the image annotation. The query q is a text description optionally including a keyframe from the video, and the document d is a specific temporal segment (not the whole video). The annotator identifies a specific target—such as an action, object, or character—and localizes a relevant temporal segment (the positive). Simultaneously, it identifies an irrelevant segment with a "clear temporal gap" from the positive to serve as a hard negative. This task teaches the model to distinguish between temporally proximate events, which is crucial for video search where the correct moment might be a 10-second clip within a 10-minute video.

Prompt design. The paper provides two example prompts in Appendix B—one for Image Question Answering and one for Video Classification. These reveal the annotation structure: the VLM is instructed to follow a multi-step procedure (description first, then task selection, then populating a structured JSON output), with hard constraints that the output be only a JSON object with specific keys. The prompts specify that the task_type must be exactly one of the enumerated options, that the question must be "directly answerable from the visual or embedded textual content," and that all text fields should be in a specified language (the {language} placeholder suggests the prompts support multilingual annotation, consistent with the model's 30+ language capability).

Scale. The paper states (Figure 5) that Stage 1 uses 300 million synthesized examples. Stage 2 uses approximately 40 million examples drawn from curated public datasets, proprietary in-house data, and sampled synthetic data to address task imbalance. Stage 3 uses only 4 million examples for distillation, selected to ensure balanced distribution across retrieval categories.

Hard Negative Mining Pipeline: Recall and Relevance Filtering

The quality of contrastive learning depends critically on the quality of negative examples. Random negatives are easy to distinguish from positives, so they provide little learning signal. Hard negatives—documents that are similar to positives but not actually relevant—force the model to learn fine-grained distinctions. The paper implements an automated two-stage pipeline (Section 3.3) that can be applied to any sub-dataset D_i in the training collection.

Stage 1: Recall. For each sub-dataset D_i = (I_i, Q_i, C_i, R_i), an embedding model (specifically, the GME model in Stage 1, and Qwen3-VL-Embedding s0 in Stage 2) encodes all queries q_j ∈ Q_i and all documents d_k ∈ C_i. For each query, the top-K most relevant candidates are retrieved based on cosine similarity, producing a set of candidate documents and their similarity scores S = {s_{j,k}}^K_{k=1}. The paper does not specify the exact value of K, but it's implied to be large enough to include both all positives and a pool of potential hard negatives.

Stage 2: Relevance Filtering. This stage refines the original relevance labels R_i using the retrieved candidates and scores. It has two sub-operations:

Positive Refinement: For each query q_j, the system checks whether at least one positive document d^+ among the top-K candidates achieves a similarity score s > t^+, where t^+ is a hyperparameter score threshold. If no positive document exceeds this threshold—meaning even the embedding model cannot find the supposedly relevant document—the query is discarded entirely. This eliminates queries whose original relevance labels are likely erroneous or whose positives are genuinely impossible to retrieve with the current model, preventing the training from being confused by noisy supervision.

Hard Negative Selection: For queries that survive positive refinement, the system computes the average similarity score \bar{s}^+ of all refined positive documents for that query. Any non-positive document d ∈ {d_k}^K_{k=1} is selected as a hard negative only if its similarity score satisfies s < \bar{s}^+ + δ^-, where δ^- is a "small safety margin" hyperparameter. The inequality deserves careful attention: a candidate must score below the average positive score plus a margin to be used as a hard negative. This means the system selects negatives that are close to but below the positive scores—they're similar enough to be confused with positives, but the embedding model can still (barely) distinguish them. The safety margin δ^- prevents the inclusion of "false negatives"—documents that are actually relevant but were mislabeled as irrelevant in the original dataset. If a candidate's score is too close to or above the average positive score, it may actually be relevant, and using it as a negative would teach the model to push apart genuinely similar pairs.

Connecting the pipeline to training stages. This mining pipeline is applied iteratively. In Stage 1, the embedding model used for mining is an existing open-source model (GME, Zhang et al., 2025b), and the mined data is used to train Qwen3-VL-Embedding s0. In Stage 2, the improved s0 model replaces GME as the mining model, producing higher-quality mined data that is then used to train s1 and the reranker. This creates a virtuous cycle: better models produce better hard negatives, which train even better models. The paper explicitly calls this out as a motivation for the multi-stage strategy: "As the training progresses through successive stages, the model's capabilities are continuously enhanced. This improvement, in turn, facilitates more effective data mining, thereby refining the quality of the training data."

Stage 1: Contrastive Pre-training — Building Broad but Noisy Retrieval Capability

The first training stage takes the Qwen3-VL-Instruct model (a general-purpose VLM capable of following instructions, answering questions about images, and generating text) and fine-tunes it to become an embedding model using LoRA (Low-Rank Adaptation). The stage uses the 300 million synthesized examples described above, mined using the GME model for hard negative selection.

Optimization objective. The paper uses a modified InfoNCE loss that incorporates multiple types of negative samples:

$L_{\text{retrieval}} = -\frac{1}{N} \sum_{i} \log \frac{e^{s(q_i, d^+_i) / \tau}}{Z_i}$

where N is the batch size, s(·,·) is the cosine similarity function, τ is a temperature parameter, and Z_i is the normalizing denominator that aggregates scores from the positive pair and various types of negative pairs.

The denominator Z_i is defined as:

$Z_i = e^{s(q_i, d^+_i) / \tau} + \sum_{k} m_{ik} e^{s(q_i, d^-_{i,k}) / \tau} + \sum_{j \neq i} m_{ij} e^{s(q_i, q_j) / \tau} + \sum_{j \neq i} m_{ij} e^{s(d^+_i, d_j) / \tau} + \sum_{j \neq i} m_{ij} e^{s(q_i, d_j) / \tau}$

where:

• The first term is the positive pair (query q_i with its relevant document d^+_i).
• The second term sums over K explicit hard negatives d^-_{i,k} mined for this query.
• The third term sums over other queries q_j in the batch (in-batch query negatives).
• The fourth term sums over other documents d_j in the batch, compared against the positive document d^+_i (in-batch document negatives contrasted with the positive).
• The fifth term sums over other documents d_j compared against the query q_i (standard in-batch document negatives).
• Each m_{ij} is a masking factor that is 0 when the corresponding pair might be a false negative and 1 otherwise.

What it computes: For each query in the batch, the model computes cosine similarities between the query embedding and all possible target embeddings (its positive document, its hard negatives, other queries in the batch, and other documents in the batch). These similarities are scaled by temperature τ and exponentiated. The loss is the negative log of the ratio: the exponentiated similarity of the positive pair divided by the sum of exponentiated similarities of all pairs. Minimizing this loss pushes the positive similarity up relative to all negative similarities, which geometrically means pulling the positive document closer to the query in embedding space while pushing all negatives away.

Why this form (five-term denominator): This is more aggressive than standard InfoNCE, which typically only contrasts the query against documents (terms 1, 2, and 5). The additional terms 3 and 4 provide two extra forms of regularization. Term 3 (query-query contrast) prevents the model from collapsing all queries to the same point in embedding space—if queries are dissimilar to each other, the embedding space preserves query diversity. Term 4 (positive-document against other documents) ensures that positive documents are not only close to their own queries but also distinct from other queries' documents, preventing a failure mode where all documents cluster together. The masking factors m_{ij} address the false negative problem inherent in in-batch negatives: another query's document might actually be relevant to the current query (e.g., two different queries in the batch might both be relevant to the same document). The masking rule is:

$m_{ij} = \begin{cases} 0, & \text{if } s_{ij} > s(q_i, d^+_i) + 0.1 \text{ or } d_j = d^+_i \\ 1, & \text{otherwise} \end{cases}$

If a candidate negative has a similarity score exceeding the positive pair's similarity by a 0.1 margin, or if it's literally the same document as the positive, it's masked (excluded from the loss). The 0.1 margin prevents accidentally penalizing the model for finding genuinely relevant documents that were mislabeled as negatives.

Temperature parameter. The paper mentions τ as a temperature but does not specify its value. In standard InfoNCE, lower temperatures make the distribution sharper (penalizing negatives more harshly), while higher temperatures make it softer (allowing some negatives to have moderate similarity). The choice of τ trades off between learning fine-grained distinctions (low τ) and training stability (higher τ).

LoRA training. The paper states that LoRA is used throughout all training stages. LoRA freezes the pre-trained weights of the transformer and injects trainable low-rank decomposition matrices into the attention layers. This provides three advantages explicitly cited: reduced memory footprint (allowing larger effective batch sizes), enhanced generalization (the low-rank constraint acts as regularization), and efficient hyperparameter search for model merging (since only the LoRA weights need to be merged, not the full model). The paper does not specify the LoRA rank, alpha, or which specific layers are adapted—these are presumably standard choices from the LoRA literature (Hu et al., 2022).

Output. The model produced at the end of Stage 1 is designated Qwen3-VL-Embedding s0. Its performance on MMEB-V2 is shown in Table 6: an overall score of 66.6. This is substantially below the final model (73.2) but already competitive with some prior models (e.g., VLM2Vec-V2 at 59.2, GME-2B at 55.3), demonstrating that the synthetic data alone can produce a reasonable embedding model.

Stage 2: Multi-Task Contrastive Learning and Supervised Fine-Tuning — Specialization and Reranker Training

The second stage introduces higher-quality data and task-specific loss functions. The training data mix (approximately 40 million examples) combines curated public datasets, proprietary in-house data, and a sampling of synthetic data to address task imbalances. The key innovation is that the data is re-mined using Qwen3-VL-Embedding s0 (the Stage 1 model) rather than GME, which produces higher-quality positives and harder negatives because s0 is better at multimodal relevance estimation than the general-purpose GME model.

Modified retrieval loss. For retrieval data in Stage 2, the paper modifies the InfoNCE objective by removing two terms from the denominator Z_i: the query-query term and the positive-document-against-other-documents term (terms 3 and 4 in the Stage 1 formulation). The loss becomes:

$L_{\text{retrieval, stage2}} = -\frac{1}{N} \sum_i \log \frac{e^{s(q_i, d^+_i) / \tau}}{e^{s(q_i, d^+_i) / \tau} + \sum_k m_{ik} e^{s(q_i, d^-_{i,k}) / \tau} + \sum_{j \neq i} m_{ij} e^{s(q_i, d_j) / \tau}}$

The paper states that "empirically, this adjustment yields better performance on high-quality multimodal retrieval data." The rationale (inferred, as the paper doesn't elaborate) is that the query-query and document-document contrastive terms act as a form of uniformity regularization—they spread out all embeddings regardless of relevance relationships. This is helpful on noisy synthetic data where many labels may be incorrect (it prevents the model from overfitting to potentially wrong pairings by enforcing general separation), but on high-quality curated data, these terms introduce an unnecessary constraint. They force the model to separate embeddings even when the underlying content might legitimately be similar (e.g., two different queries that are both about the same topic), which can hurt the model's ability to cluster semantically related content.

Classification loss. For text and image classification tasks, the paper also formulates training as contrastive learning, but with a critical difference in negative sampling. The instance to be classified is treated as the query q, and its class label is treated as the positive document d^+. However, "negative samples are restricted to explicitly incorrect labels for the same query, while other labels in the batch are ignored to avoid introducing false negatives." This means the model only contrasts against the hard negative labels that were synthetically generated alongside each example (e.g., "wolf" vs. "husky"), not against all other class labels present in the batch. The reasoning: if the batch contains a cat image with label "cat" and a dog image with label "dog," using "dog" as a negative for the cat query would be incorrect—"dog" is not a wrong label for the cat image, it's a label for a different instance. Contrasting against it would teach the model that "cat" and "dog" labels are dissimilar, which is the wrong inductive bias (both are animals, both are pets, and in some contexts they might be semantically related).

Semantic Textual Similarity (STS) loss. STS datasets provide real-valued similarity scores (e.g., 0.0 to 5.0) rather than binary relevant/irrelevant labels. These datasets are also symmetric: there's no natural query-document distinction—both texts in a pair are equivalent. To exploit the fine-grained continuous supervision, the paper uses the CoSent loss (Huang et al., 2024):

$L_{\text{sts}} = \log \left(1 + \sum_{\hat{s}(q_i, d_j) > \hat{s}(q_m, d_n)} \exp\left(\frac{\cos(q_m, d_n) - \cos(q_i, d_j)}{\tau}\right)\right)$

where \hat{s}(q_i, d_j) is the ground-truth similarity score for the pair (q_i, d_j), and the sum runs over all pairs (q_m, d_n) in the batch whose ground-truth similarity is less than the ground-truth similarity of (q_i, d_j).

What it computes: For every pair of pairs in the batch where the ground-truth ordering is violated—meaning pair (q_i, d_j) should be more similar than pair (q_m, d_n) according to ground truth, but the model's cosine similarity for (q_m, d_n) is higher—the loss adds a penalty term exp((cos(q_m, d_n) - cos(q_i, d_j)) / τ). The magnitude of the penalty grows exponentially with the degree of violation. The outer logarithm compresses the sum into a more manageable range.

Why this form: CoSent is a listwise ranking loss designed specifically for STS data, where the supervision signal is a total ordering of pair similarities rather than binary labels. Unlike contrastive losses that only care about making positives more similar than negatives, CoSent cares about preserving the relative ordering of all pairs. The exponential penalty means the loss is dominated by the largest violations (pairs whose ordering is grossly wrong), which is desirable because these are the most damaging for downstream tasks that rely on similarity ranking. The alternative—using MSE on the similarity scores directly—would require the model to produce exactly calibrated scores (hard because cosine similarity ranges from -1 to 1 while ground-truth scores might be on a 0-5 scale), whereas CoSent only requires correct relative ordering, which is scale-invariant.

Reranker training. Simultaneously with the Stage 2 embedding model training, the paper trains the Qwen3-VL-Reranker from the same Qwen3-VL-Instruct initialization. The reranker is trained on "the retrieval-specific subset of the newly mined data," encompassing image retrieval, video retrieval, moment retrieval, and visual document retrieval. Classification and QA data are excluded because the reranker's binary relevance judgment formulation doesn't naturally extend to those tasks.

The reranker uses a simple binary classification loss:

$L_{\text{reranking}} = -\log p(l | I, q, d)$

where p(·|*) is the probability assigned by the VLM to the correct label l, which is "yes" for positive (relevant) query-document pairs and "no" for negative (irrelevant) pairs. I is the instruction, q is the query, and d is the document.

What it computes: standard negative log-likelihood for binary classification. The model processes the full input (instruction + query + document), and at the final position, it produces a probability distribution over the vocabulary. The loss is the negative log of the probability assigned to the correct answer token. If the pair is relevant and the model assigns probability 0.9 to "yes," the loss is -log(0.9) ≈ 0.105. If it assigns probability 0.1, the loss is -log(0.1) ≈ 2.30.

Why this form: this is the standard maximum-likelihood objective for classification, which is well-calibrated and straightforward to optimize. The paper cites Dai et al. (2025) for the specific approach of framing reranking as yes/no token prediction. The key design choice is using a single binary token prediction rather than a regression head (predicting a scalar relevance score directly). Token prediction leverages the full power of the VLM's language modeling head, which has been pre-trained on massive text corpora and can smoothly interpolate relevance judgments based on linguistic patterns in the instruction and content. A regression head would be randomly initialized and would require learning the relevance concept from scratch on the smaller fine-tuning dataset.

Outputs of Stage 2. Two models emerge: Qwen3-VL-Embedding s1 (the embedding model, overall MMEB-V2 score 72.1 per Table 6) and Qwen3-VL-Reranker (available in both 2B and 8B sizes). Table 6 shows that s1 achieves strong performance across the board: 74.8 on images, 60.3 on video, 77.1 on VisDoc. However, the retrieval-specific tasks (VDRv2 at 58.8, VR at 84.9) are not yet at the level that the final model achieves, motivating the distillation stage.

Stage 3: Distillation and Model Merging — Transferring Reranker Expertise into the Embedding Model

The third stage is where the paper makes its most distinctive technical contribution. The core idea is that the reranker, with its cross-attention architecture, can make more nuanced relevance judgments than the embedding model, but it's too slow for first-pass retrieval. The solution is to distill the reranker's knowledge into the embedding model, so the faster bi-encoder can approximate the cross-encoder's judgments.

Distillation data construction. The paper curates a compact subset of 4 million examples from both public and proprietary sources, with "a balanced distribution across multiple retrieval categories." The Qwen3-VL-Reranker (trained in Stage 2) processes each query along with its positive document and k negative documents, producing relevance logits for each candidate. These logits are pre-computed offline and stored as soft labels.

Distillation loss. During training, the embedding model computes cosine similarities for each query against its (k+1) candidate documents (one positive, k negatives) and normalizes them into a probability distribution via softmax:

$P_{\text{embedding}}(d_i | q) = \frac{\exp(\cos(q, d_i) / \tau')}{\sum_{j=1}^{k+1} \exp(\cos(q, d_j) / \tau')}$

The reranker's logits are similarly converted to a probability distribution (presumably using a softmax with its own temperature, though the paper doesn't specify the reranker's temperature parameter):

$P_{\text{reranker}}(d_i | q) = \text{softmax}(\text{logit}_{\text{reranker}}(d_i | q))$

The distillation loss is the cross-entropy between these two distributions:

$L_{\text{distill}} = -\sum_{i=1}^{k+1} P_{\text{reranker}}(d_i | q) \log P_{\text{embedding}}(d_i | q)$

What it computes: For each query, the embedding model produces a probability distribution over (k+1) candidate documents based on cosine similarity. The reranker provides a target distribution over the same candidates based on its cross-attention processing. The loss is the cross-entropy—the average number of bits needed to encode the reranker's distribution using the embedding model's distribution. Minimizing this loss forces the embedding model's similarity rankings to match the reranker's more nuanced relevance judgments.

Why this form: This is standard knowledge distillation (Hinton et al., 2015) applied to ranking. The key advantage over using hard binary labels (relevant/irrelevant) is that the reranker's soft distribution contains richer information. It might assign probability 0.7 to the positive document, 0.2 to a hard negative that is somewhat relevant, and 0.02 to several clearly irrelevant documents. This tells the embedding model not just "this is positive, these are negative," but "this negative is much closer to being relevant than these other negatives," which provides a more informative gradient signal. The alternative—using the reranker's raw logits directly to compute an MSE loss—would require the embedding model's cosine similarities to match the reranker's logit scale, which is an unnecessary and potentially harmful constraint (cosine similarity is bounded to [-1, 1], while reranker logits are unbounded). The softmax normalization elegantly solves this by converting both to comparable probability distributions.

The trade-off problem. Table 6 reveals a critical issue with distillation. Comparing s1 (no distillation) to s2 (after distillation) for the 2B model:

• Retrieval tasks improve substantially: VDRv2 jumps from 58.8 to 72.4 (+13.6 points), VR improves from 84.9 to 87.9 (+3.0 points), VisDoc OOD improves from 66.4 to 70.6 (+4.2 points). The overall VisDoc score rises from 77.1 to 80.9.
• Classification and QA tasks degrade: Image CLS drops from 71.2 to 61.8 (-9.4 points), Image QA drops from 75.8 to 69.8 (-6.0 points), Image GD drops from 88.3 to 76.3 (-12.0 points). The overall Image score falls from 74.8 to 71.3.

This is a classic task conflict: the distillation objective optimizes exclusively for retrieval-style relevance ranking, causing the model to partially "forget" the classification and QA capabilities it developed in Stage 2. The phenomenon is well-documented in multi-task learning and embedding model training, where optimizing for one task family can shift representations in ways that harm others.

Model merging as resolution. The paper's solution is to merge the s2 checkpoint (strong at retrieval, weak at classification/QA) with the s1 checkpoint (strong at classification/QA, weaker at retrieval) using the methodology from Li et al. (2024). The paper doesn't elaborate on the specific merging technique, but the Li et al. (2024) reference suggests a weighted averaging of model parameters in weight space, likely applied only to the LoRA weights (since the base VLM weights are frozen and shared). The merged model is designated Qwen3-VL-Embedding s3.

The results (Table 6, s3 row) show that merging successfully reconciles the trade-off. The overall score reaches 73.2, higher than both s1 (72.1) and s2 (71.5). On images, s3 achieves 75.0—slightly above s1's 74.8 and substantially above s2's 71.3. On VisDoc, s3 achieves 79.2—between s1's 77.1 and s2's 80.9, preserving most of the distillation gains. On video, s3 achieves 61.9—slightly above s1's 60.3 and s2's 59.5. The model has effectively combined the complementary strengths of both checkpoints.

Why merging works (conceptual). The paper doesn't provide a theoretical justification, but the empirical pattern is consistent with the hypothesis that retrieval and classification/QA tasks require different geometric properties in the embedding space. Retrieval benefits from embeddings that are well-separated and optimized for nearest-neighbor discrimination (distillation provides this). Classification and QA benefit from embeddings that preserve categorical structure and fine-grained visual-semantic alignment (Stage 2 multi-task training provides this). Model merging, by interpolating between parameters optimized for these different objectives, finds a compromise that preserves both types of structure to a reasonable degree. This is analogous to the linear mode connectivity phenomenon observed in deep learning, where independently fine-tuned models can be interpolated in weight space without catastrophic performance collapse.

Matryoshka Representation Learning (MRL): Variable-Dimension Embeddings Without Retraining

Standard embedding models produce fixed-dimensional vectors—you train at 2048 dimensions, you get 2048-dimensional embeddings. If you later want 1024-dimensional embeddings to save storage, you have to retrain. MRL (Kusupati et al., 2022) solves this by training the model to produce good embeddings at multiple truncated dimensions simultaneously.

How it's integrated. When computing any of the loss functions described above, the paper "computes each loss not only on the full-dimensional embedding, but also on truncated lower-dimensional prefixes of the same representation." For example, if the full embedding dimension is 2048, the loss might be computed on dimensions [0:128], [0:256], [0:512], [0:1024], and [0:2048], with each truncated prefix treated as a valid embedding and scored against similarly truncated versions of positive and negative documents. The total training loss is the sum (or average) of losses across all MRL dimensions.

What this achieves operationally. During training, the model learns to front-load the most important information into the earliest dimensions of the embedding vector. Dimension 0 will carry the most critical semantic signal, dimension 1 the next most critical, and so on. At inference time, users can specify any desired embedding dimension d ≤ D_max, and simply take the first d elements of the full embedding. No retraining, no finetuning, no separate model versions needed.

Generalization property. The paper notes that "training over a sufficiently dense set of MRL dimensions yields strong generalization, enabling competitive performance at intermediate dimensions that are not explicitly included during training." This means if you train on dimensions [128, 256, 512, 1024], the model will also perform reasonably well at dimension 384 or 768, because the nested structure enforced by MRL creates a smooth degradation curve rather than sharp drop-offs at untrained dimensions.

Performance trade-offs (Figure 6). The paper's analysis shows concrete numbers for the 2B model. On MS MARCO text retrieval: full 1024-dimension embeddings achieve MRR@10 of 0.360; halving to 512 dimensions drops to approximately 0.355 (only -1.4%); reducing to 128 dimensions drops to approximately 0.188 (a significant -47.8% but at dramatically lower cost). On VL3-Syn cross-modal retrieval: 1024 dimensions achieve 0.497 MRR@10; 512 dimensions achieve 0.487 (-2.0%); 128 dimensions achieve 0.138 (-72.2%). The paper emphasizes that "within a reasonable range, this degradation is acceptable given the substantial savings in storage and retrieval latency."

Quantization-Aware Training (QAT): Low-Precision Embeddings

Storing embeddings at float32 precision uses 4 bytes per dimension per embedding. For a corpus of 100 million documents at 2048 dimensions, that's 100M × 2048 × 4 bytes = 819 GB. Quantizing to int8 (1 byte per dimension) reduces this to 205 GB. Binary quantization (1 bit per dimension) reduces it to 25.6 GB. However, naively quantizing embeddings after training—simply rounding float32 values to their nearest int8 or binary representation—typically causes significant performance degradation because the embedding space was optimized for continuous values.

LSQ (Learned Step Size Quantization). The paper adopts LSQ (Esser et al., 2020), which makes quantization part of the training process. LSQ works by simulating quantization during the forward pass and using a Straight-Through Estimator (STE) (Bengio et al., 2013) to propagate gradients through the non-differentiable rounding operation during backpropagation.

Specifically, a float embedding value v is quantized to its low-precision counterpart v̂ by:

$v̂ = \lfloor \text{clamp}(v / s, Q_{\min}, Q_{\max}) \rceil \times s$

where s is the quantization scale (step size), Q_min and Q_max are the minimum and maximum representable values in the target precision (e.g., -128 to 127 for int8), clamp restricts v/s to this range, and ⌊·⌉ rounds to the nearest integer. The scale s is a learnable parameter optimized jointly with the model weights. During backpropagation, the gradient through the rounding operation is approximated as identity (the STE assumption: ∂v̂/∂v ≈ 1 when v is within the clamping range).

Integration into training. During training, the paper "computes the optimization objective using both full-precision embeddings and their low-precision (quantized) counterparts." This means each loss term is evaluated twice: once on the float32 embeddings (as normal) and once on the simulated low-precision embeddings (using LSQ's forward pass). The model receives gradients from both, encouraging it to produce embeddings that remain discriminative even after quantization.

Performance results (Figure 6). The paper's analysis shows that int8 quantization "preserves retrieval performance with negligible degradation." On MS MARCO, the int8 curve is nearly identical to the float32 curve across all embedding dimensions. On VL3-Syn, the same pattern holds—int8 tracks float32 closely. Binary quantization, however, "significantly impairs retrieval effectiveness," and this performance loss "becomes increasingly pronounced as embedding dimensionality decreases." The paper reports specific numbers: on MS MARCO at 1024 dimensions, float32 achieves 0.360 MRR@10 while binary achieves 0.188; at 128 dimensions, binary drops to near-zero performance. This makes intuitive sense: binary quantization collapses all information to a single bit per dimension, and when there are only 128 such bits total, the representational capacity is severely limited.

Operational trade-offs. The paper provides latency and memory numbers alongside accuracy. On MS MARCO: float32 at 1024 dims requires 32,539 MB index storage and 43 ms retrieval latency; int8 at 1024 dims requires 8,135 MB and 12 ms; binary at 1024 dims requires 127 MB and 0.61 ms. On VL3-Syn (2M images): float32 uses 7,812 MB and 2.87 ms; int8 uses 1,953 MB and 0.94 ms; binary uses 31 MB and 0.032 ms. These numbers demonstrate the practical deployment spectrum: int8 is essentially "free" in terms of accuracy cost while providing 4× storage reduction and 3–4× speedup; binary provides dramatic efficiency gains but at substantial accuracy cost, making it suitable only for applications where approximate retrieval is acceptable.

Dynamic Resolution and Frame Rate Processing

The paper describes specific visual processing configurations used during training (Section 4.2) that differ from the evaluation settings (Section 6.1). These choices reflect a trade-off between training efficiency and evaluation thoroughness.

Training-time settings. For images: the original aspect ratio is preserved, but the maximum token consumption is capped at 1,280 visual tokens (approximately 1.3 million pixels). For video: frames are sampled at 1 frame per second (1 FPS) with a maximum of 64 frames; each frame maintains its aspect ratio; the total token budget across all frames is constrained to 4,500 visual tokens (approximately 9.2 million pixels).

Why these specific numbers. The 1,280 token cap for images is a balance between visual detail and computational efficiency—it allows the model to process reasonably high-resolution images while keeping the sequence length manageable during large-scale training. The 4,500 token cap for video represents a similar compromise: encoding 64 frames at full resolution would require 64 × 1,280 = 81,920 tokens, which is far beyond the model's 32K context window, so the per-frame resolution must be reduced when many frames are present. The 1 FPS sampling rate is standard in video-language models; it captures sufficient temporal information for most retrieval-relevant events while avoiding redundant frames in slow-moving scenes.

Evaluation-time settings. At evaluation on MMEB-V2, the paper uses different limits: context length is constrained to 16,384 tokens (rather than the full 32K training maximum), image tasks cap at 1,800 tokens, and video tasks cap at 15,000 total tokens with 64 frames. These are higher than the training caps, reflecting the fact that evaluation runs are computationally cheaper and can afford more detailed processing. The increase from 1,280 to 1,800 image tokens and from 4,500 to 15,000 video tokens suggests the paper found that the model benefits from higher resolution at inference time, even though training with those resolutions would have been too expensive.

Impact of increased granularity (Section 7.2, Figure 7). The paper investigates how performance varies with visual token allocation and frame count. The findings show a consistent pattern: "performance improves with increased resource consumption across all task categories," but with "pronounced diminishing return as resource allocation grows, with a slight performance regression occurring at the highest levels of consumption." For images, performance rises from approximately 60% at 200 tokens to a peak of approximately 77% around 1,000 tokens, then declines slightly. For visual documents, the curve peaks around 1,000–1,500 tokens and then declines. For video, scaling both tokens and frames shows similar inverted-U patterns. The paper attributes the decline to "the inherent performance degradation that the model encounters when processing excessively long contexts"—a known limitation of transformer models where very long sequences can cause attention dilution or positional encoding degradation.

4. Key Insights and Innovations

Innovation 1: Model Merging as a Mechanism to Reconcile Task Conflict in Embedding Training

The most conceptually distinctive contribution of this paper is not any single training stage or loss function, but rather the diagnosis and resolution of a fundamental trade-off in unified embedding model training—and the demonstration that model merging, rather than joint optimization or staged fine-tuning alone, can recover balanced performance across conflicting task families.

Prior work in embedding model training has long grappled with the observation that optimizing for different tasks pulls representations in incompatible directions. Text embedding models like E5 (Wang et al., 2022), BGE (Chen et al., 2024), and GritLM (Muennighoff et al., 2024) all observed that retrieval-oriented training can degrade performance on semantic textual similarity, classification, or clustering tasks. The standard responses have been either (a) multi-task training with carefully balanced loss weights, hoping to find a compromise through joint optimization, or (b) accepting the trade-off and releasing separate model variants optimized for different task families. Both approaches are unsatisfactory: joint optimization often settles on a mediocre compromise rather than achieving Pareto-optimality, and multiple model variants fragment the deployment story that a "unified" embedding model is supposed to provide.

What Qwen3-VL-Embedding demonstrates—and what Table 6 makes empirically undeniable—is that the conflict between retrieval and classification/QA tasks is structurally unavoidable under sequential fine-tuning but resolvable through post-hoc weight-space interpolation. The distillation stage clearly shows this: reranker distillation boosts VisDoc retrieval (VDRv2 jumps from 58.8 to 72.4, a 13.6-point gain) but causes Image classification to collapse (from 71.2 to 61.8, a 9.4-point loss). This is not a failure of the distillation methodology—it is evidence that the distillation objective genuinely optimizes a different geometric property of the embedding space than the classification and QA objectives. The embeddings that make for good nearest-neighbor retrieval are not the same embeddings that make for good linear classification. Joint training would have averaged these gradients and likely produced a model that was mediocre at both. Sequential training (do Stage 2, then distill) produces two checkpoints that are each excellent at different things, but neither is excellent at everything.

The model merging step is what converts this apparent failure into the paper's primary success. By interpolating between the s1 checkpoint (strong at classification/QA) and the s2 checkpoint (strong at retrieval) in parameter space, the merged model s3 achieves an overall MMEB-V2 score of 73.2—higher than either s1 (72.1) or s2 (71.5) individually. This is the hallmark of a genuine Pareto improvement: not just averaging the strengths of two models, but combining their complementary capabilities into something better than either alone.

This finding has broader implications for the embedding model training paradigm. It suggests that the field's instinct to seek a single training recipe that optimizes all objectives simultaneously may be misguided. Instead, the more effective approach may be to train specialized variants and then merge them, treating model parameters as composable building blocks rather than as the endpoint of a single optimization trajectory. This connects to the broader literature on linear mode connectivity and model merging (Li et al., 2024; Wortsman et al., 2022; Ilharco et al., 2023), but applies it in a context—embedding model training—where the specific task conflict has been well-documented but rarely addressed through this lens. The contribution is primarily conceptual: reframing multi-task embedding training from an optimization problem (find one set of weights that does everything) to a composition problem (train specialists and then combine them).

Innovation 2: Reranker-to-Embedding Distillation for Multimodal Retrieval

While knowledge distillation from cross-encoders to bi-encoders is well-established in text retrieval (e.g., Hofstätter et al., 2021; Lin et al., 2021), its application to multimodal retrieval—and specifically the finding that cross-modal relevance judgment can be transferred into a shared embedding space—represents a significant extension. The prior assumption in multimodal retrieval was that cross-encoder quality could not be effectively distilled because the cross-attention interactions between modalities are fundamentally different from the independent encoding that bi-encoders perform. The concern is that a cross-encoder can notice that "the red car in the upper left of the image" corresponds to a specific phrase in the query, while a bi-encoder must encode the entire image and the entire query independently, losing this fine-grained alignment signal.

The paper's evidence challenges this assumption. Table 6 shows that distillation provides substantial and specific gains on retrieval tasks—VDRv2 improves by 13.6 points, VR by 3.0 points, VisDoc OOD by 4.2 points—without requiring any architectural changes to the bi-encoder. The embedding model learns to approximate the reranker's cross-attention-based relevance judgments using only cosine similarity between independently produced vectors. This works because the distillation loss (Equation 3) provides a distribution-matching signal: the embedding model doesn't need to replicate how the reranker arrives at its judgment, only what judgment it makes. The reranker's output distribution over candidate documents captures the fine-grained relevance ordering, and the embedding model learns to produce vector representations whose cosine similarities reproduce that ordering.

What makes this conceptually significant beyond the performance numbers is that it establishes a scalable path for improving bi-encoder quality in multimodal settings. Cross-encoder rerankers are expensive to run but can process relatively small numbers of candidates (top-K from first-stage retrieval). The distillation pipeline amortizes this expensive computation across the training process: run the reranker once on a carefully curated subset of data, store its judgments as soft labels, and train the bi-encoder to mimic them. The bi-encoder can then serve at scale during inference. This is a training-time-compute-for-inference-time-quality trade that is well-understood in text retrieval but had not been convincingly demonstrated for multimodal retrieval at this scale and breadth (covering image, video, visual document, and cross-modal retrieval).

Innovation 3: VLM-as-Annotator Data Synthesis at Scale for Multimodal Retrieval

The paper's data synthesis strategy represents a shift in how training data for multimodal retrieval is constructed. Prior to this work, multimodal retrieval models relied primarily on either (a) naturally occurring paired data (e.g., image-caption datasets like MS COCO, video-description datasets like MSR-VTT), which are limited in scale, domain coverage, and task diversity, or (b) rule-based or template-based synthesis, which produces repetitive patterns that limit generalization.

The Qwen3-VL-Embedding approach instead treats a strong VLM (Qwen3-VL-32B) as a general-purpose multimodal annotation engine. Rather than designing separate synthesis procedures for each task type, the paper defines a taxonomy of task paradigms (classification, QA, retrieval for images; classification, QA, retrieval, moment retrieval for video) and provides the VLM with structured prompts that specify what to annotate and how to format the output. The VLM leverages its own multimodal understanding to generate questions that are genuinely grounded in visual content, answers that are accurate, and hard negative distractors that are plausibly confusing.

This approach has three distinctive properties that go beyond simple "use a stronger model to label data":

Semantic depth hierarchy. The paper's annotation prompts instruct the VLM to generate retrieval queries across a "hierarchy of semantic depths"—from direct visual descriptions ("a red bicycle leaning against a brick wall") through abstract narrative scenarios, compositional logical constraints, and knowledge-centric textual localization. This produces training data that covers the full spectrum from literal visual matching to high-level semantic reasoning, rather than being concentrated at one level of abstraction. Prior synthesis approaches typically generated data at a single semantic level (e.g., image captioning produces literal descriptions), leaving models weak at abstract or compositional retrieval.

Integrated hard negative generation. Each annotated example includes not just a positive pair but also a "semantically confusing" or "plausible but deceptive" negative—generated by the annotator VLM itself based on its understanding of what could be confused with the correct answer. For classification, this means generating a negative label that is semantically related to the correct category (e.g., "husky" as a negative for "wolf"). For QA, this means generating a wrong answer that someone might reasonably give if they misunderstood the visual content. These aren't random negatives—they're adversarially informed negatives generated by the same model that understands the visual content. Traditional data synthesis generates negatives by random sampling from a candidate pool, which provides mostly easy negatives that contribute little to contrastive learning. The paper's approach bakes the hard negative signal directly into the synthesis process.

Seed pool curation as a separate design problem. The paper treats the construction of the seed image/video pool as its own multi-stage engineering problem: quality filtering, structural refinement for video, fine-grained category labeling, cross-modal alignment filtering via an independent embedding model, and category-wise rebalancing. This is a recognition that the diversity of the synthesized data is fundamentally bounded by the diversity of the seed pool—if you only seed the synthesis pipeline with natural images, you won't get visual document or UI/screenshot data regardless of how capable your annotator VLM is. The explicit category taxonomy in Figure 4 (covering natural images, artificial/man-made content, artistic/synthetic content, and diverse video categories) reflects a deliberate effort to ensure broad domain coverage. Prior data synthesis work in multimodal learning often treated the seed data as given rather than as a resource to be actively curated and balanced.

Innovation 4: Matryoshka + Quantization-Aware Training as Joint Efficiency Objectives

Individually, Matryoshka Representation Learning (MRL) and Quantization-Aware Training (QAT) are established techniques. MRL was introduced by Kusupati et al. (2022) and has been adopted in text embedding models like BGE-M3 and Qwen3-Embedding. QAT has a long history in model compression dating back to Esser et al. (2020) and beyond. The paper's contribution is not the invention of either technique but rather their integration as co-optimized auxiliary objectives in a multimodal embedding training pipeline—and the demonstration that they interact favorably rather than destructively.

This matters because the naive approach to deployment efficiency would be to train a high-quality full-precision, full-dimension embedding model and then post-hoc compress it via dimensionality reduction (PCA or truncation) and quantization (rounding to lower precision). Post-hoc compression typically causes significant performance degradation because the embedding space was never optimized to be compressible—the model freely used all dimensions and all bits of precision during training, and compressing them destroys information that was important for discrimination.

By integrating MRL and QAT as auxiliary losses during training, the model learns to produce embeddings that are robust to truncation and quantization by construction. The MRL objective forces the model to prioritize information along the dimension axis—critical semantic content goes into early dimensions, fine-grained details go into later dimensions. The QAT objective forces the model to produce embedding values that are discriminative even when rounded to lower precision—the embedding space develops wider margins between clusters to accommodate rounding error.

The empirical demonstration (Figure 6) shows that these techniques complement each other in a non-obvious way. Int8 quantization preserves performance nearly perfectly across all MRL dimensions on both text and cross-modal retrieval tasks. This means you can get both the storage benefits of lower-dimensional embeddings (via MRL) and the storage benefits of lower-precision embeddings (via QAT) simultaneously, with the combined savings multiplying rather than the performance penalties compounding. On MS MARCO text retrieval, going from float32 1024-dim (32,539 MB, 43ms) to int8 512-dim (approximately 4,068 MB, 6ms) would represent roughly an 8× reduction in storage and a 7× speedup while losing only marginally more accuracy than either technique alone—a combined efficiency gain that would be impossible if the techniques had to be applied sequentially post-hoc.

The paper's framing of these as deployment considerations rather than just accuracy optimizations is itself significant. Section 7.1 explicitly analyzes the storage-latency-accuracy trade-off curve, providing concrete numbers that a practitioner can use to make deployment decisions. This reflects a shift in how embedding models are evaluated: from pure accuracy benchmarks to accuracy-efficiency Pareto frontiers, acknowledging that a model that achieves state-of-the-art accuracy but requires prohibitive storage and latency is not practically useful for the large-scale retrieval systems where these models are deployed.

Innovation 5: Difficulty-Aware Hard Negative Mining via Iterative Data Refinement

The paper's two-stage hard negative mining pipeline—recall followed by relevance filtering with a safety margin—is at the surface level a data cleaning technique. But its conceptual significance lies in how it reframes the relationship between model quality and data quality as a mutually reinforcing cycle rather than a one-time preprocessing step.

Prior approaches to hard negative mining in contrastive learning typically operate with a fixed model and a fixed dataset: train an embedding model, use it to retrieve hard negatives from the corpus, add those to the training data, and retrain. The model used for mining is frozen—it doesn't improve during the mining process. The Qwen3-VL-Embedding pipeline breaks this cycle open by making the mining model itself a product of the earlier training stage. Stage 1 uses the off-the-shelf GME model to mine data; this data trains Qwen3-VL-Embedding s0; s0 is then used to re-mine the same datasets for Stage 2 training. The result is a higher-quality training set for Stage 2 because s0 is a better multimodal relevance estimator than GME, so its positive refinement is more accurate and its hard negative selection identifies more genuinely confusing negatives.

The paper explicitly calls this out as a motivation for the multi-stage design: "As the training progresses through successive stages, the model's capabilities are continuously enhanced. This improvement, in turn, facilitates more effective data mining, thereby refining the quality of the training data." This is a bootstrapping argument: you don't need to start with perfect training data; you can start with noisy data, train an okay model, use that model to clean and improve the data, train a better model, and iterate. The paper only demonstrates two iterations of this cycle (GME → s0 → s1, with s1's data mined by s0), but the principle extends naturally to further iterations.

The safety margin mechanism in the relevance filtering step (s < \bar{s}^+ + δ^-) adds an important conceptual nuance. Hard negative mining has a known failure mode: if the mining model is too aggressive, it can select "false negatives"—documents that are actually relevant but were mislabeled as irrelevant in the original dataset. Training on these teaches the model to push apart genuinely similar pairs, degrading retrieval quality. The paper's solution is not to avoid this problem entirely (which would require perfect labels) but to make the threshold relative to the query's own positive scores. A document is only selected as a hard negative if its similarity score is below the average positive score plus a margin. This means the threshold adapts per-query: for queries with high positive scores (easy, well-matched positives), the threshold is higher, allowing the selection of harder negatives; for queries with lower positive scores (ambiguous or difficult cases), the threshold is lower, erring on the side of caution and avoiding potential false negatives. This per-query adaptivity is conceptually cleaner than a global fixed threshold, which would either be too aggressive for some queries or too conservative for others.

This innovation is incremental in mechanism but fundamental in framing: it establishes data quality improvement as an endogenous part of the training process rather than an exogenous preprocessing step, and it provides a principled adaptive thresholding mechanism to manage the exploration-exploitation trade-off inherent in hard negative selection.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary multimodal evaluation uses the MMEB-V2 benchmark (Meng et al., 2025), comprising 78 datasets across three domains: Image (36 datasets spanning classification, QA, retrieval, and grounding), Video (18 datasets spanning classification, QA, retrieval, and moment retrieval), and Visual Document (24 datasets spanning retrieval, ViDoRe, VisRAG, and out-of-distribution evaluation). Additional visual document benchmarks include JinaVDR (Günther et al., 2025), ViDoRe-v3, VisRAG, and VisDocOOD. Text-only evaluation uses MMTEB (Enevoldsen et al., 2025a), a multilingual benchmark covering bitext mining, classification, clustering, instruction retrieval, multilabel classification, pair classification, reranking, retrieval, and STS. Reranker evaluation uses MMEB-V2 retrieval subsets, MMTEB retrieval, JinaVDR, and ViDoRe-v3.

• Base model(s). The embedding and reranking models are initialized from Qwen3-VL-Instruct (Bai et al., 2025), a vision-language foundation model. Two parameter sizes are evaluated: 2B (28 layers) and 8B (36 layers), both supporting 32K token context windows. The Qwen3-VL family was chosen because it provides "inherent cross-modal alignment through pre-training on large-scale image-text datasets" (Section 1) and strong multilingual support (30+ languages). For data synthesis, the larger Qwen3-VL-32B model serves as the annotator. The FLOPs-matched comparison (Section 7) uses no separate larger model; instead, the comparison is purely between the Qwen3-VL-Embedding variants at different training stages (Table 6) — this is not a pretraining-vs-inference compute trade-off analysis in the style of scaling laws papers, but rather an internal ablation documenting how each training stage contributes to final performance.

• Metrics. The primary metric is MMEB-V2 overall score, an average across all 78 datasets computed by normalizing per-dataset scores and averaging within task categories, then averaging across all tasks. Individual task scores are reported separately for classification (CLS), question answering (QA), retrieval (RET), grounding (GD), moment retrieval (MRET), visual document retrieval (VDR, split into v1 and v2), VisRAG (VR), and out-of-distribution (OOD). For text-only evaluation, the MTEB mean task score is used, computed as the average across all nine task types. For efficiency analysis (Section 7.1), MRR@10 (Mean Reciprocal Rank at cutoff 10) is reported. For reranker evaluation, task-specific retrieval scores are reported on MMEB-V2 retrieval subsets, MMTEB retrieval, JinaVDR, and ViDoRe-v3, with an average across these four categories. All MMEB-V2 scores for compared models are reported from the authors' re-evaluation runs (Table 2 note: "All models except IFM-TTE have been re-evaluated").

• Baselines. The paper compares against three categories of prior work, all evaluated on MMEB-V2:

  Open-source VLM-based embedding models: VLM2Vec (Jiang et al., 2025) at 2B and 8B, VLM2Vec-V2 (Meng et al., 2025) at 2B, GME (Zhang et al., 2025b) at 2B and 8B, Ops-MM-embedding-v1 at 2B and 8B, and RzenEmbed (Jian et al., 2025) at 2B and 8B. These represent the prior state-of-the-art in unified multimodal embedding.

  Closed-source embedding models: IFM-TTE at 8B, Seed-1.6-embedding-0615, Seed-1.6-embedding-1215, Gemini Embedding (Lee et al., 2025), and OpenAI text-embedding-3-large. These represent commercial API services and proprietary models.

  ColPali-style visual document models (Table 3): llama-nemoretriever-colembed-1b-v1 and 3b-v1 (Xu et al., 2025), colnomic-embed-multimodal-3b and 7b (Team, 2025), colqwen2.5-v0.2 (Faysse et al., 2025), and tomoro-colqwen3-embed-4b and 8b (Huang & Tan, 2025). These represent specialized visual document retrieval models using patch-level embeddings.

  Text-only embedding models (Table 4): KaLM-Embedding-Gemma3-12B (Zhao et al., 2025), llama-embed-nemotron-8b (Babakhin et al., 2025), NV-Embed-v2 (Lee et al., 2024), GritLM-7B (Muennighoff et al., 2024), BGE-M3 (Chen et al., 2024), multilingual-e5-large-instruct (Wang et al., 2024a), gte-Qwen2-1.5B-instruct and gte-Qwen2-7B-instruct (Li et al., 2023), Qwen3-Embedding-0.6B, 4B, and 8B (Zhang et al., 2025c), Cohere-embed-multilingual-v3.0, and text-embedding-3-large. These provide the ceiling for text-only performance.

  Reranker baselines (Table 5): Qwen3-VL-Embedding-2B (the embedding model used as first-stage retriever, providing a baseline for retrieval without reranking) and jina-reranker-m0.

  The choice of baselines is comprehensive across the three key axes: unified multimodal embedding, specialized document retrieval, and text-only embedding. For the visual document benchmarks (Table 3), the ColPali-style baselines are particularly important because they represent the dominant approach for document retrieval, and matching or exceeding their performance with a general-purpose embedding model is a key claim of the paper.

• Generation budget / compute accounting. The paper does not report generation budgets in the traditional sense (number of sampled solutions) because embedding models produce single deterministic vectors rather than multiple sampled outputs. Instead, compute is implicitly accounted for through model size (2B vs. 8B parameters) and embedding dimension (1024, 512, 256, etc. with Matryoshka). The efficiency analysis in Section 7.1 measures storage (MB), retrieval latency (ms), and MRR@10 as a function of embedding dimension and quantization precision, providing a Pareto frontier rather than a single compute-normalized comparison. For reranker evaluation, the paper specifies that Qwen3-VL-Embedding-2B is used to retrieve the top 100 candidates before reranking, ensuring fair comparison between reranking models that all operate on the same candidate pool. The paper does not report wall-clock training time, GPU-hours, or total FLOPs for any training stage.

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. For the MMEB-V2 benchmark, results are reported as single-point averages across the 78-dataset suite, with no confidence intervals, standard deviations, or significance tests. The leaderboard-based evaluation (Table 2) uses a fixed test set. For the multi-stage training analysis (Table 6), results are reported at a single training run per stage, with no indication of variance across random seeds or data splits. The efficiency analysis (Figure 6) uses sampled subsets (10,000 queries for MS MARCO, 10,000 captions for VL3-Syn) but does not report confidence intervals. This absence of statistical rigor is a notable weakness: with 78 datasets averaged into a single score, small per-dataset variations could cumulatively produce a practically meaningful difference in the overall average that is not statistically significant at conventional thresholds. The paper's claim of state-of-the-art status rests on a 0.9-point margin over Seed-1.6-embedding-1215 (77.8 vs. 76.9 in Table 2), and without variance estimates, it's unclear whether this difference is reliable.

Main Quantitative Results

MMEB-V2: State-of-the-Art Multimodal Embedding Performance

Headline result (Table 2): Qwen3-VL-Embedding-8B achieves an overall MMEB-V2 score of 77.8, ranking first among all models evaluated as of January 2026. This represents a 6.7% improvement over the previous best open-source model (RzenEmbed-7B at 72.9) and a 0.9-point margin over the best closed-source model (Seed-1.6-embedding-1215 at 76.9). The Qwen3-VL-Embedding-2B achieves 73.2, competitive with the best 8B open-source models.

Domain-level breakdown (Table 2, comparing Qwen3-VL-Embedding-8B to best prior models):

• Image (36 datasets): Qwen3-VL-Embedding-8B scores 80.1 overall, outperforming the best closed-source model Seed-1.6-embedding-1215 (78.0) by 2.1 points and the best open-source model RzenEmbed-7B (75.9) by 4.2 points. Within image tasks: classification achieves 74.2 (vs. 76.7 for IFM-TTE, the leader on this subcategory), QA achieves 81.1 (highest among all models, 2.6 points above IFM-TTE's 78.5), retrieval achieves 80.2 (highest among all models, 0.9 points above Seed-1.6-embedding-1215's 79.3), and grounding achieves 92.3 (second only to RzenEmbed-7B's 92.1).

• Video (18 datasets): Qwen3-VL-Embedding-8B scores 67.1 overall, essentially matching the best closed-source model IFM-TTE (59.2? No—this appears to be an error in the paper. Reading Table 2 carefully: IFM-TTE video overall is 59.2, while Qwen3-VL-Embedding-8B is 67.1, a substantial 7.9-point improvement. Wait—checking the actual numbers: IFM-TTE shows Video CLS 60.5, QA 67.9, RET 51.7, MRET 54.9, Overall 59.2. Qwen3-VL-Embedding-8B shows Video CLS 78.4, QA 71.0, RET 58.7, MRET 56.1, Overall 67.1. The best closed source on video is Seed-1.6-embedding-1215 at 67.7, not IFM-TTE. Qwen3-VL-Embedding-8B is 0.6 points below Seed-1.6-embedding-1215 on video overall.) Within video tasks: classification achieves 78.4 (vs. Seed-1.6-embedding-1215's 85.2, a 6.8-point deficit), QA achieves 71.0 (highest, 4.3 points above IFM-TTE's 67.9), retrieval achieves 58.7 (second to Seed-1.6-embedding-1215's 59.1), and moment retrieval achieves 56.1 (second to Seed-1.6-embedding-1215's 54.8—wait, 56.1 > 54.8, so Qwen3-VL-Embedding-8B actually leads here. Let me re-read: Seed-1.6-embedding-1215 MRET is 54.8, Qwen3-VL-Embedding-8B is 56.1, so Qwen3-VL-Embedding-8B leads moment retrieval by 1.3 points.)

• Visual Document (24 datasets): Qwen3-VL-Embedding-8B scores 82.4 overall, slightly above Seed-1.6-embedding-1215 (82.2) by 0.2 points and substantially above RzenEmbed-7B (81.3) by 1.1 points. Within VisDoc tasks: VDRv1 achieves 87.2 (below Seed-1.6-embedding-1215's 90.0 by 2.8 points), VDRv2 achieves 69.9 (above IFM-TTE's 71.5? No—IFM-TTE is 71.5 on VDRv2, Qwen3-VL-Embedding-8B is 69.9, so it trails here by 1.6 points), VR achieves 88.7 (below IFM-TTE's 92.7 by 4.0 points and Seed-1.6-embedding-1215's 90.0 by 1.3 points), and OOD achieves 73.3 (highest, 2.6 points above Seed-1.6-embedding-1215's 70.7).

The 2B variant (Qwen3-VL-Embedding-2B, overall 73.2) is competitive with much larger models: it exceeds GME-8B (59.1) by 14.1 points, VLM2Vec-8B (53.1) by 20.1 points, and approaches RzenEmbed-7B (72.9) within 0.3 points while using 3.5× fewer parameters. On video tasks, the 2B model achieves 61.9—exceeding all prior open-source 8B models except RzenEmbed-7B (55.7) by 6.2 points.

Key pattern across modalities: The 8B model shows the largest advantages over prior work in image QA (+2.6 over IFM-TTE) and video classification (+18.1 over IFM-TTE, though -6.8 below Seed-1.6-embedding-1215). Its relative weakness is in video classification compared to the closed-source Seed-1.6-embedding-1215 (85.2 vs. 78.4) and in VDRv1 compared to Seed-1.6-embedding-1215 (87.2 vs. 90.0). The overall ranking is achieved through balanced strength across all three domains rather than dominance in any single one.

Visual Document Benchmarks: Competitive with Specialized ColPali Models

Headline result (Table 3): Qwen3-VL-Embedding-8B achieves performance comparable to specialized ColPali-style models while using a general-purpose architecture. The embedding model achieves an average of 75.8 across six visual document benchmarks (VisRAG, VisDocOOD, Vidore-v1, Vidore-v2, Vidore-v3, JinaVDR), compared to 77.7 for the best ColPali-style model (tomoro-colqwen3-embed-8B) and 76.5 for tomoro-colqwen3-embed-4B.

Per-benchmark comparison (Table 3, Qwen3-VL-Embedding-8B vs. best ColPali model tomoro-colqwen3-embed-8B):

• VisRAG: 88.7 vs. 90.2 (-1.5)
• VisDocOOD: 73.3 vs. 76.8 (-3.5)
• Vidore-v1: 87.2 vs. 90.8 (-3.6)
• Vidore-v2: 69.9 vs. 67.7 (+2.2)
• Vidore-v3: 59.0 vs. 61.6 (-2.6)
• JinaVDR: 76.9 vs. 79.2 (-2.3)

The embedding model's strongest showing is on Vidore-v2, where it leads the ColPali model by 2.2 points. On the remaining five benchmarks, it trails by 1.5–3.6 points. The aggregate deficit of 1.9 points (75.8 vs. 77.7) is modest given the architectural difference: ColPali models produce per-patch embeddings and use late-interaction scoring (MaxSim over patch-level similarities), which is specifically designed for visual document retrieval, while Qwen3-VL-Embedding produces a single pooled vector per document.

Reranker dramatically improves performance. The Qwen3-VL-Reranker-8B achieves an average of 80.3 across the same six benchmarks—2.6 points above the best ColPali model and 4.5 points above the embedding-only model. Per-benchmark: VisRAG 91.2 (+1.0 over ColPali), VisDocOOD 75.7 (-1.1), Vidore-v1 91.9 (+1.1), Vidore-v2 72.8 (+5.1), Vidore-v3 66.7 (+5.1), JinaVDR 83.6 (+4.4). The reranker provides the largest gains on Vidore-v2, Vidore-v3, and JinaVDR, suggesting these benchmarks contain query-document pairs where fine-grained cross-attention between query and document content yields substantial accuracy improvements over vector similarity alone. The 2B reranker also outperforms all ColPali models (average 76.7) except tomoro-colqwen3-embed-8B (77.7), demonstrating that even a small cross-encoder can compete with much larger specialized late-interaction models.

Text-Only Benchmarks: Competitive but Below Specialized Text Models

Headline result (Table 4): Qwen3-VL-Embedding-8B achieves a mean task score of 67.9 on MMTEB, which is competitive with similarly sized text-only embedding models but 2.7 points below the specialized Qwen3-Embedding-8B (70.6). The 2B variant achieves 63.9.

Comparison to text-only models of similar scale (Table 4):

• vs. Qwen3-Embedding-8B (70.6): -2.7 points. This is the most direct comparison since both share the same base architecture and training methodology, with Qwen3-VL-Embedding adding multimodal capability. The degradation is distributed across task types: bitext mining 77.5 vs. 80.9 (-3.4), classification 72.0 vs. 74.0 (-2.0), clustering 55.8 vs. 57.7 (-1.9), retrieval 69.4 vs. 70.9 (-1.5), STS 75.4 vs. 81.1 (-5.7). The largest gap is in STS, suggesting that multimodal training partially degrades the model's ability to capture fine-grained textual similarity distinctions.
• vs. Gemini Embedding (68.4): -0.5 points, essentially tied.
• vs. llama-embed-nemotron-8b (69.5): -1.6 points.
• vs. KaLM-Embedding-Gemma3-12B (72.3): -4.4 points, but this is a 12B model.
• The 2B variant (63.9) exceeds the similarly sized BGE-M3 (0.6B, 59.6) and multilingual-e5-large-instruct (0.6B, 63.2), and approaches gte-Qwen2-1.5B-instruct (59.5—actually 63.9 > 59.5 by 4.4 points).

Key interpretation: The 2.7-point degradation relative to Qwen3-Embedding-8B is the "multimodal tax"—the cost of adding image, video, and visual document capabilities to a text embedding model. In exchange for this tax, the model gains the ability to process all modalities (Table 2 shows 77.8 on MMEB-V2 where text-only models cannot operate at all). The paper frames this as an acceptable trade-off, and the numbers support this: a 3.8% relative degradation in text-only performance in exchange for state-of-the-art multimodal performance is a favorable exchange rate for applications that require both capabilities. For pure text applications, the specialized Qwen3-Embedding remains the better choice.

Reranker Evaluation: Significant Gains Over Embedding-Only Retrieval

Headline result (Table 5): Qwen3-VL-Reranker-8B achieves an average score of 74.9 across the four evaluation categories (MMEB-V2 retrieval, MMTEB retrieval, JinaVDR, ViDoRe-v3), compared to 68.1 for the embedding-only baseline (Qwen3-VL-Embedding-2B used as retriever) and 57.8 for jina-reranker-m0 on the subset of benchmarks where jina-reranker-m0 was evaluated.

Per-category analysis (Table 5):

• MMEB-V2 Image Retrieval: Reranker-8B achieves 79.2, compared to 73.4 for embedding-only (+5.8) and 75.2 for Reranker-2B (+4.0 over embedding, +4.0 over the 2B reranker for the 8B variant). The reranker provides larger gains on image retrieval than on video or VisDoc retrieval in MMEB-V2.

• MMEB-V2 Video Retrieval: Reranker-8B achieves 78.2, compared to 74.8 for embedding-only (+3.4) and 74.0 for Reranker-2B. Note that Reranker-2B actually performs slightly below the embedding baseline (74.0 vs. 74.8, -0.8 points), meaning the 2B cross-encoder does not improve over bi-encoder retrieval for video—only the 8B variant provides gains. This is a notable finding: cross-encoder reranking for video may require sufficient model capacity to process the longer sequences and more complex temporal dynamics.

• MMEB-V2 VisDoc Retrieval: Reranker-8B achieves 61.0, compared to 53.6 for embedding-only (+7.4) and 53.2 for Reranker-2B (-0.4 vs. embedding). The same pattern as video: the 2B reranker underperforms the embedding baseline, while the 8B reranker provides substantial gains. VisDoc retrieval shows the largest absolute improvement from 8B reranking (+7.4 points), suggesting that visual documents—with their interleaved text, figures, tables, and complex layouts—benefit most from cross-attention between query and document.

• MMTEB Text Retrieval: Reranker-8B achieves 85.8, compared to 68.1 for embedding-only (+17.7) and 83.2 for Reranker-2B (+15.1). This is the largest absolute improvement across all categories, consistent with the well-established finding in text retrieval that cross-encoder reranking provides substantial gains over bi-encoder retrieval. The 2B reranker also strongly outperforms the embedding baseline here, unlike in video and VisDoc.

• JinaVDR: Reranker-8B achieves 83.6, compared to 71.0 for embedding-only (+12.6) and 80.9 for Reranker-2B (+9.9).

• ViDoRe-v3: Reranker-8B achieves 66.7, compared to 52.9 for embedding-only (+13.8) and 60.8 for Reranker-2B (+7.9).

Scaling behavior: The 8B reranker outperforms the 2B reranker by an average of 4.1 points across all evaluated tasks (cited in the abstract as "improving ranking results by 4.1 points over the 2B model across multiple tasks"). Without specifying which tasks this average is computed over, the paper's abstract claim is somewhat ambiguous—the per-task differences in Table 5 vary substantially: +4.0 on MMEB-V2 Image, +4.2 on MMEB-V2 Video, +7.8 on MMEB-V2 VisDoc, +2.6 on MMTEB, +2.7 on JinaVDR, +5.9 on ViDoRe-v3. The simple average of these six differences is approximately 4.5 points, consistent with the abstract's claim.

Efficiency Analysis: Matryoshka and Quantization Trade-offs

Headline results (Figure 6): The paper's efficiency analysis demonstrates that MRL and QAT provide favorable storage-latency-accuracy trade-off curves on both text retrieval (MS MARCO) and cross-modal retrieval (VL3-Syn).

Dimensionality scaling (Figure 6, float32 curves):

• On MS MARCO text retrieval: MRR@10 drops from 0.360 at 1024 dimensions to approximately 0.355 at 512 dimensions (-1.4%), to approximately 0.30 at 256 dimensions (-16.7%), to 0.188 at 128 dimensions (-47.8%). The near-plateau from 1024 to 512 dimensions means users can halve their storage and latency costs with negligible accuracy loss.
• On VL3-Syn cross-modal retrieval: MRR@10 drops from 0.497 at 1024 dimensions to 0.487 at 512 dimensions (-2.0%), to approximately 0.43 at 256 dimensions (-13.5%), to 0.138 at 128 dimensions (-72.2%). The degradation at very low dimensions is more severe for cross-modal retrieval than for text retrieval, suggesting that cross-modal alignment requires more embedding dimensions to capture modality-invariant semantic structure.

Quantization effects (Figure 6):

• Int8 quantization preserves performance nearly identically to float32 across all embedding dimensions on both tasks. The int8 curve essentially overlays the float32 curve. On MS MARCO at 1024 dims, float32 achieves 0.360 and int8 achieves 0.360; at 512 dims, both achieve approximately 0.355. On VL3-Syn, the same pattern holds: 0.497 vs. 0.497 at 1024 dims, 0.487 vs. 0.487 at 512 dims.
• Binary quantization causes severe degradation that worsens at lower dimensions. On MS MARCO at 1024 dims, binary achieves 0.188 (vs. 0.360 for float32, a -47.8% drop); at 128 dims, binary achieves near-zero performance. On VL3-Syn at 1024 dims, binary achieves 0.138 (vs. 0.497, -72.2%). The interaction between binary quantization and low dimensionality is particularly destructive because binary quantization removes all magnitude information, leaving only sign patterns, and at 128 dimensions, the representational capacity is simply insufficient for cross-modal alignment.

Practical deployment numbers (Figure 6 annotations):

• MS MARCO (text retrieval, 10K queries, full training set as corpus): float32 1024-dim requires 32,539 MB storage and 43 ms retrieval latency; int8 1024-dim requires 8,135 MB and 12 ms; binary 1024-dim requires 127 MB and 0.61 ms.
• VL3-Syn (cross-modal, 10K captions, 2M image corpus): float32 1024-dim requires 7,812 MB and 2.87 ms; int8 requires 1,953 MB and 0.94 ms; binary requires 31 MB and 0.032 ms.

The storage-to-latency ratios demonstrate the practical viability: int8 provides 4× storage reduction and 3–4× latency reduction with effectively zero accuracy cost, making it a "free lunch" for most deployment scenarios. Binary provides dramatic efficiency gains (256× storage reduction, 70× latency reduction) but at a substantial accuracy cost that limits applicability to scenarios where approximate retrieval is acceptable.

Ablation Studies and Robustness Checks

Multi-stage training pipeline (Table 6): The paper provides a detailed ablation of how each training stage contributes to final performance, evaluating four checkpoints (s0, s1, s2, s3) from the 2B model on MMEB-V2.

• s0 (Stage 1 only: contrastive pre-training on 300M synthetic data): Overall score 66.6. Image 65.8, Video 57.5, VisDoc 74.8. This baseline already exceeds several prior open-source models (VLM2Vec-2B at 47.7, GME-2B at 55.3), demonstrating that synthetic data alone can produce competitive multimodal embeddings when generated by a strong VLM annotator with a curated seed pool.

• s1 (Stage 2: multi-task contrastive learning on 40M curated + sampled synthetic data): Overall score 72.1 (+5.5 over s0). Image jumps to 74.8 (+9.0, driven by CLS +9.0 and QA +12.1), Video to 60.3 (+2.8), VisDoc to 77.1 (+2.3). The largest gains are in classification and QA—tasks where the curated data likely provides cleaner labels than synthetic data.

• s2 (Stage 3: distillation from reranker, before merging): Overall score 71.5 (-0.6 vs. s1). The distillation causes a task-level redistribution of performance: Image drops to 71.3 (-3.5 vs. s1), with CLS dropping 9.4 points (71.2→61.8) and QA dropping 6.0 points (75.8→69.8), while retrieval tasks surge—VDRv2 jumps from 58.8 to 72.4 (+13.6), VR from 84.9 to 87.9 (+3.0), VisDoc OOD from 66.4 to 70.6 (+4.2). The VisDoc overall rises from 77.1 to 80.9 (+3.8). This is the key ablation demonstrating that distillation specifically transfers retrieval-relevant knowledge at the cost of classification and QA capabilities.

• s3 (after model merging of s1 and s2): Overall score 73.2 (+0.6 vs. s2, +1.1 vs. s1). Image recovers to 75.0 (+3.7 vs. s2, +0.2 vs. s1), VisDoc settles at 79.2 (-1.7 vs. s2, +2.1 vs. s1), Video reaches 61.9 (+2.4 vs. s2, +1.6 vs. s1). The merged model achieves the highest overall score among all four checkpoints, confirming that merging successfully combines the complementary strengths of s1 (strong at classification/QA) and s2 (strong at retrieval).

Impact of spatial and temporal granularity (Figure 7, Section 7.2): The paper investigates how performance varies with visual token allocation and frame count on MMEB-V2 tasks.

• Image token scaling: Performance rises from approximately 60% at 200 tokens to a peak around 77% at approximately 1,000 tokens, then declines slightly to approximately 75% at 1,200 tokens. The peak-to-decline is modest (~2 points), suggesting the model is relatively robust to moderate over-allocation of image tokens.
• Visual document token scaling: Performance rises from approximately 50.5% at 500 tokens to a peak around 53% at approximately 1,500 tokens, then declines to approximately 51.5% at 3,000 tokens. The curve is flatter than for images, with a total variation of only ~2.5 points across a 6× range of token budgets.
• Video token scaling: Performance rises from approximately 42% at 1,000 tokens to a peak around 56% at approximately 4,000 tokens, then declines to approximately 54% at 6,000 tokens. The larger variation (~14 points) indicates that video tasks are more sensitive to token under-allocation.
• Video frame scaling: Performance rises from approximately 40% at near-zero frames to a peak around 56% at approximately 50 frames, then declines slightly. The saturation point (~50 frames) corresponds closely to the training-time cap of 64 frames, suggesting the model learns to utilize most of its allocated frame budget.

The paper attributes the performance regression at very high allocations to "the inherent performance degradation that the model encounters when processing excessively long contexts," but provides no mechanistic analysis (e.g., attention pattern analysis, position encoding degradation, or per-layer diagnostics). This is a missed opportunity: understanding why performance degrades would inform whether architectural changes (e.g., different position encodings, sparse attention) could extend the effective context range.

Reranker scaling behavior (Table 5, implicit): The comparison between Reranker-2B and Reranker-8B across tasks reveals a pattern not explicitly discussed in the paper: the 2B reranker is competitive with or exceeds the embedding baseline for text retrieval (+15.1 on MMTEB), JinaVDR (+9.9), and ViDoRe-v3 (+7.9), but underperforms the embedding baseline for MMEB-V2 video retrieval (-0.8) and MMEB-V2 VisDoc retrieval (-0.4). The 8B reranker, in contrast, improves over the embedding baseline across all tasks. This suggests a minimum capacity threshold for cross-encoder reranking to be beneficial in multimodal settings—at 2B parameters, the cross-encoder may not have sufficient capacity to learn meaningful query-document interactions that go beyond what the bi-encoder already captures, and the additional model capacity is essentially "wasted" on fitting the training data without learning transferable relevance patterns. This threshold is lower for text (where the 2B reranker strongly outperforms the baseline) than for video and visual documents.

Comparison to text-only Qwen3-Embedding models (Table 4, Qwen3-VL-Embedding vs. Qwen3-Embedding): The 2.7-point gap between Qwen3-VL-Embedding-8B (67.9) and Qwen3-Embedding-8B (70.6) on MMTEB is an implicit ablation of the effect of adding multimodal training on text-only performance. The gap is largest in STS (75.4 vs. 81.1, -5.7 points) and bitext mining (77.5 vs. 80.9, -3.4), and smallest in retrieval (69.4 vs. 70.9, -1.5) and multilabel classification (28.6 vs. 28.7, -0.1). This pattern suggests that multimodal training most significantly impacts tasks requiring fine-grained semantic similarity judgment between text pairs (STS, bitext mining), while having minimal impact on tasks that depend more on topical relevance (retrieval) or discrete label assignment (classification). The paper does not provide an ablation isolating which stage of multimodal training causes this degradation—it could be the synthetic data pre-training, the multi-task training, the distillation, or the model merging.

Absent ablations (notable gaps):

• Data scale ablations: The paper does not report performance as a function of synthetic data quantity (e.g., 50M, 100M, 150M, 300M examples in Stage 1) or curated data quantity (Stage 2). This makes it impossible to assess whether the 300M/40M/4M data scale choices are near-optimal or if further data scaling would yield continued improvements.
• Seed pool quality ablation: The paper describes an elaborate seed pool curation pipeline (quality filtering, structural refinement, category labeling, cross-modal alignment filtering, rebalancing) but does not ablate any of these steps. It's unclear which curation steps are essential and which are merely nice-to-have.
• Hard negative mining ablation: The paper does not compare training with the mined hard negatives vs. training with random negatives at the same data scale, which would quantify the contribution of the mining pipeline.
• Model merging comparison: The paper uses a single merging technique (Li et al., 2024) without comparing to alternatives (e.g., simple weight averaging, task-vector arithmetic, or multi-task training with loss reweighting as an alternative to post-hoc merging).
• LoRA rank ablation: The paper uses LoRA throughout training but does not specify the rank or ablate its effect.
• Distillation temperature ablation: The temperature τ' in the distillation softmax (Equation 3) is not specified or ablated. The sharpness of the teacher distribution is known to significantly affect distillation quality.

Critical Assessment

The experimental results broadly support the paper's central claims, but with important qualifications about what is actually demonstrated versus what is asserted.

Claim: Qwen3-VL-Embedding-8B achieves state-of-the-art on MMEB-V2, ranking first among all models.

The evidence in Table 2 shows Qwen3-VL-Embedding-8B at 77.8, 0.9 points above Seed-1.6-embedding-1215 at 76.9. Two concerns undermine the definitiveness of this claim. First, the paper reports no confidence intervals, standard deviations, or statistical tests. With 78 datasets averaged into a single score, each with its own normalization and weighting, a 0.9-point difference could easily fall within the variance of the evaluation protocol. Second, the MMEB-V2 leaderboard is a moving target—the paper acknowledges evaluation was done "as of January 2026" (or "as of January 8, 2025" in the abstract, an internally inconsistent date), implying that newer models may have since surpassed this score. The claim of state-of-the-art status is therefore temporally qualified and statistically uncertain. The paper would be on stronger ground claiming "competitive with the best available models" rather than "ranking first."

A more robust claim would be that Qwen3-VL-Embedding-8B achieves state-of-the-art performance among open-source models, where the margin over the next best (RzenEmbed-7B at 72.9, a 4.9-point gap) is large enough to likely be statistically significant even without formal testing. The closed-source comparison is less clear-cut.

Claim: The multi-stage training pipeline with distillation and model merging is responsible for the model's performance.

Table 6 provides strong evidence that each training stage contributes meaningfully, and that the s3 merged model achieves the best overall performance. However, the ablation has several limitations:

• Single training run: Results are reported for one training run per stage. Without seed variance, it's unclear whether the s3 > s1 > s2 > s0 ordering is reliable or if different random initializations or data shuffles would produce different relative orderings.
• No joint training baseline: The paper does not compare the multi-stage approach against training a single model with a combined multi-task objective from the start (i.e., combining Stage 1, Stage 2, and distillation losses in one training run). This is the most natural baseline for a claim about the necessity of multi-stage training, and its absence is a significant gap. The paper's argument for multi-stage training—that it "mitigates the data imbalance between abundant weakly-supervised data and scarce high-quality samples"—is plausible but untested against the alternative of loss-weighted joint training.
• Model merging not compared to alternatives: The claim that model merging reconciles the retrieval-vs-classification trade-off is supported by the s3 vs. s1/s2 comparison, but the paper does not show that merging is necessary rather than merely sufficient. Could a carefully tuned multi-task loss in Stage 2 have achieved similar balance without requiring separate distillation and merging stages? The paper provides no evidence either way.
• Distillation contribution conflated with data quality: Stage 3 uses a "compact sub-dataset" of 4 million examples, different from both Stage 1 and Stage 2 data. The performance changes from s1 to s2 could reflect the distillation signal, the different data distribution, or both. An ablation using the same distillation data but with binary labels (positive/negative) instead of reranker soft labels would disentangle these effects.

Claim: The model is practical for deployment due to MRL and QAT.

The efficiency analysis in Figure 6 convincingly demonstrates that MRL and QAT provide meaningful storage-latency-accuracy trade-offs. However, the analysis has two limitations:

• Limited retrieval settings: The analysis uses only two datasets (MS MARCO for text retrieval, VL3-Syn for cross-modal retrieval) with fixed corpus sizes (the full MS MARCO training set for text, 2 million images for cross-modal). Performance at much larger scales (100M+ documents, common in production retrieval systems) may differ due to changes in the nearest-neighbor recall characteristics. The paper does not report recall@K for approximate nearest-neighbor indices, which is what production systems actually use.
• No latency analysis for video/document retrieval: The efficiency analysis is restricted to text and image retrieval. Video and visual document retrieval—where the model's ability to process long sequences is most valuable—would likely have different latency characteristics due to the cost of encoding video frames or multi-page documents. This omission is notable given that video and visual documents are where the model's advantages over text-only embedding models are most distinctive.

Claim: The reranker provides substantial gains over embedding-only retrieval.

Table 5 supports this claim for the 8B reranker but reveals an important qualification for the 2B reranker. On MMEB-V2 video and VisDoc retrieval, the 2B reranker underperforms the embedding baseline. This means the claimed reranker benefits are conditional on model scale—users deploying the 2B model size should not expect reranking gains, and may even see degradation, for video and visual document tasks. The paper's abstract and conclusion do not mention this qualification, presenting reranker benefits as universal.

Missing experiments that would strengthen the paper:

1. Statistical significance testing on MMEB-V2: Bootstrap confidence intervals over the 78 datasets would provide a simple way to assess the reliability of the 0.9-point margin over Seed-1.6-embedding-1215.
2. Joint training baseline for multi-stage ablation: Training a single model with combined Stage 1 + Stage 2 + distillation losses (with appropriate loss weighting) would test whether the multi-stage pipeline is necessary or merely convenient.
3. Distillation signal vs. data quality ablation: Using the Stage 3 data with hard binary labels instead of reranker soft labels would isolate the contribution of the distillation signal from the contribution of the data subset.
4. Seed pool curation ablation: Quantifying the impact of the cross-modal alignment filtering and category rebalancing steps on downstream performance would justify the complexity of the seed pool construction.
5. Recall@K analysis for large-scale retrieval: Evaluating the embedding model's recall@K with approximate nearest-neighbor indices at corpus scales of 10M, 100M, and 1B documents would validate the practical deployment claims.
6. Per-language analysis: The paper claims 30+ language support but reports only aggregate multilingual scores on MMTEB. Per-language breakdowns would reveal whether multilingual performance is uniform or concentrated in high-resource languages.
7. Failure analysis: The paper provides no qualitative analysis of cases where the embedding model fails—which query-document pairs receive high similarity scores despite being irrelevant, and which receive low scores despite being relevant. This would provide insight into the model's limitations beyond aggregate benchmark numbers.

Overall assessment: The experiments demonstrate that Qwen3-VL-Embedding is a strong multimodal embedding model that achieves competitive or state-of-the-art results across a broad range of benchmarks. The multi-stage training pipeline is shown to be effective, and the efficiency techniques are shown to provide practical deployment benefits. However, the paper's strongest claims—first place on MMEB-V2, the necessity of the multi-stage design, and universal reranker benefits—are supported with qualifications that the paper's framing tends to understate. The evaluation is comprehensive in breadth (many benchmarks, many baselines) but shallow in depth (no variance estimates, limited ablations, no failure analysis). The paper establishes Qwen3-VL-Embedding as a strong entry in the multimodal retrieval landscape but does not provide the rigorous ablation and analysis needed to fully validate its specific design claims.

6. Limitations and Trade-offs

Difficulty Estimation Remains Prohibitively Expensive for Deployment

The assumption or constraint. The entire compute-optimal framework depends on the ability to estimate prompt difficulty before allocating the inference budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations).

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4, 8) are computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter. The claim that compute-optimal scaling achieves a 4× reduction in test-time compute is thus an upper bound on achievable efficiency rather than a realized deployment gain. A practitioner attempting to use the method as described would find that the cost of generating 2048 samples to estimate difficulty exceeds the budget they are trying to optimize.

What evidence exists in the paper. Section 3.2 explicitly describes the difficulty estimation procedure (2048 samples + PRM scoring or ground-truth checking) and acknowledges the cost is not included in the experiments. Figures 4 and 8 show that predicted (PRM-based) difficulty bins perform nearly as well as oracle bins, but both use the same 2048-sample estimation process—neither avoids the upfront cost. The paper proposes no cheaper difficulty estimator (e.g., a lightweight classifier trained on question text alone) and evaluates none.

Mitigation status. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) but does not develop or evaluate any such model. The limitation is acknowledged as significant but entirely unresolved. A dynamic difficulty estimation scheme—generating a few initial samples, assessing difficulty, then allocating the remaining budget adaptively—is suggested as a direction but not implemented. In the current form, the compute-optimal framework is an analysis tool demonstrating the potential of difficulty-conditioned allocation, not a deployment-ready system.

Hard Problems Remain Completely Unsolved — Test-Time Compute Cannot Substitute for Missing Capability

The assumption or constraint. The paper's compute-optimal framework operates under an implicit but fundamental constraint: test-time compute can only amplify or refine solutions that the base model could already produce. If the base model's pass@1 rate on a problem class is near zero (no correct solutions exist in the proposal distribution), no search algorithm, revision strategy, or budget allocation can find a correct answer. The paper acknowledges this explicitly in the Section 7 discussion:

"on the hardest problems (bin 5), test-time compute provides essentially zero benefit regardless of budget"

The consequence. For any problem outside the base model's capability range—genuinely novel reasoning, out-of-distribution tasks, problems requiring knowledge the model was not exposed to during pretraining—the entire compute-optimal framework provides no benefit. The pretraining-vs-inference trade-off analysis (Figure 9) shows this starkly: on difficulty bin 5, the compute-optimal scaling curves are essentially flat near 0–5% accuracy regardless of budget, while the 14× larger model achieves measurably better performance (though still low). This means that the paper's central framing—that test-time compute can substitute for pretraining compute—has a hard boundary condition: it applies only to problems where the base model already has non-trivial capability. For problems at the frontier of a model's abilities, only pretraining (or a fundamentally different architecture) can help.

What evidence exists in the paper. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% for all search methods and all budgets (4 to 256 generations). Figure 7 (right) shows bin 5 accuracy at roughly 2–3% regardless of sequential-to-parallel ratio at 128 generations. Figure 9 shows the compute-optimal scaling line for bin 5 flat near 0–5%, below even the 14× larger model's greedy performance. Table 6 in the original Qwen3-VL-Embedding paper shows no such difficulty breakdown exists for the embedding benchmarks, but the principle generalizes: if a query is fundamentally outside the model's representational capacity, embedding-based retrieval will fail regardless of embedding dimension or quantization scheme. The paper provides no analysis of which types of queries or documents fall into this "hard problem" category.

Mitigation status. The authors are transparent about this limitation in Section 7, but do not propose any mitigation beyond the obvious: use a larger or better pretrained model for genuinely hard problems. The finding itself is a contribution—it establishes the boundary of test-time compute's effectiveness—but for practitioners, it means that compute-optimal allocation is only useful for queries within the model's existing capability envelope. The paper does not provide diagnostic tools to determine whether a given query is in this envelope before expending test-time compute on it, which creates a practical deployment risk: the system could waste substantial inference budget on problems it fundamentally cannot solve.

Single Benchmark, Single Model Family — Generality of Difficulty-Dependent Patterns Is Unverified

The assumption or constraint. All experiments in this paper use the MATH benchmark (500 test questions, high-school competition math) with PaLM 2-S* as the base model. The findings about difficulty-dependent optimal strategies—that beam search hurts easy problems, sequential revisions help easy problems, parallel sampling helps hard problems—are derived from a single task domain (symbolic mathematical reasoning) and a single model family. The paper does not evaluate on code generation, logical reasoning, scientific QA, or any non-reasoning tasks.

The consequence. The specific characterizations of what constitutes "easy," "medium," and "hard" problems, and which strategies are optimal at each level, may not transfer to other domains. Mathematical reasoning has distinctive properties—deterministic correct answers, clear step-by-step solution structure, objective verifiability—that make PRM training and revision-model fine-tuning feasible. For open-ended generation tasks (dialogue, creative writing, summarization), the concept of "correctness" is ambiguous, making both verifier training and Monte Carlo rollout supervision problematic. For factual QA tasks, correctness depends on knowledge rather than reasoning, so the model's failure modes (and hence the effectiveness of revisions vs. search) would differ. The paper's compute-optimal strategy selections—beam search on medium problems, sequential revisions on easy problems—are specific to MATH + PaLM 2-S* and should not be assumed to generalize to other combinations without independent verification.

What evidence exists in the paper. The evaluation is limited to a single benchmark throughout: MATH for all search experiments (Section 5), MATH for all revision experiments (Section 6), and MATH for the FLOPs-matched comparison (Section 7). The paper provides no cross-domain evaluation, no cross-model evaluation (the PaLM 2 family is the only model tested), and no discussion of how task properties might affect the generalizability of the findings. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not evidence.

Mitigation status. The authors acknowledge in Section 8 that extending to other domains and model families is future work, but provide no preliminary evidence that the difficulty-dependent patterns hold elsewhere. A practitioner deploying compute-optimal test-time compute on a non-math domain or with a non-PaLM model would need to independently re-derive the optimal strategies—the paper's specific policy (e.g., beam search with M=4 on medium problems) may be suboptimal or even counterproductive in other settings.

Revision Model Exhibits a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model is trained exclusively on trajectories where all in-context answers are incorrect and the target is corrective. As a result, the model never learns what to do when its current answer is already correct—it has been trained to always propose a revision, even when no revision is needed. At inference time, when the model produces a chain of revisions, approximately 38% of correct answers are incorrectly "revised" back to incorrect ones in the subsequent step (Section 6.1).

The consequence. The revision model has an inherent instability: it can improve incorrect answers but also corrupt correct ones. This means that longer revision chains do not monotonically improve accuracy—the model may oscillate between correct and incorrect states. The paper's mitigation—using majority voting or verifier-based selection across the entire chain to pick the best answer—adds computational overhead and does not guarantee that the final selected answer is from the end of the chain (losing the intuitive "iterative improvement" property). In a deployment scenario, the reversion problem means that each additional revision step has a probability of degrading rather than improving the answer, creating a revision-quality ceiling that cannot be exceeded by simply generating more revisions. This ceiling is visible in Figure 6 (left): pass@1 per step plateaus around 24–25% and does not continue to improve with additional revisions.

What evidence exists in the paper. Section 6.1 explicitly states the 38% reversion rate. Figure 6 (left) shows the revision chain's pass@1 trajectory flattening after roughly 15–20 steps, with no further improvement out to 64 steps. The ReST^EM experiment in Appendix K (Figure 16) shows that attempting to further optimize the revision model with reinforcement learning actually worsens the reversion problem, causing fully sequential performance to degrade substantially. Qualitative examples of the reversion phenomenon are not provided, which is a missed opportunity for understanding why the model revises correct answers.

Mitigation status. The paper partially mitigates this with within-chain selection (majority voting or verifier-based selection across all revision steps), but this is a post-hoc patch rather than a solution. A more principled approach—training the model to recognize when no revision is needed (e.g., by including correct-to-correct trajectories in the training data) or training a separate stopping criterion—is not explored. The reversion problem is fundamentally a training data design flaw: by only showing the model incorrect-to-correct sequences, the training distribution is misaligned with the inference-time distribution (where correct answers will appear in-context). The paper does not address this distribution shift directly.

FLOPs-Matched Comparison Uses a Weak Pretraining Baseline

The assumption or constraint. The paper's central efficiency claim—that a smaller model with compute-optimal test-time compute can outperform a 14× larger model—relies on a specific and arguably favorable pretraining baseline. The 14× larger model is trained by scaling parameters only while holding training data fixed (following the LLaMA paradigm, Touvron et al., 2023), and is evaluated with greedy decoding only (no test-time compute augmentation of any kind). The authors acknowledge this limitation in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. A compute-optimally pretrained larger model (scaling both parameters and data equally, per Hoffmann et al., 2022) would likely outperform the parameter-only-scaled model used as the baseline. Additionally, giving the larger model any test-time compute budget (even a modest best-of-8 or best-of-16) would create a much stronger baseline. The reported advantages of test-time compute over pretraining—e.g., +27.8% relative improvement on easy questions at R << 1 (Figure 1, top-right bar chart)—would likely shrink against a properly compute-optimal larger model, and might reverse in some regimes. The paper's framing as "test-time compute vs. pretraining" overstates the generality of the findings; a more accurate framing would be "test-time compute vs. a specific suboptimal pretraining baseline."

What evidence exists in the paper. Section 7 describes the pretraining baseline explicitly: parameter-only scaling, greedy decoding. Figure 9 shows the comparison at three values of R, with the 14× larger model's performance indicated by stars. The paper does not provide any comparison where the larger model is given a test-time compute budget, does not evaluate a compute-optimally trained larger model, and does not discuss how the results would change under either of these more competitive baselines.

Mitigation status. The authors acknowledge the limitation and frame it as future work, but the paper's abstract and introduction present the 14× advantage as a general finding ("smaller model with extra test-time compute can outperform a 14× larger model") without the caveat. The limitation is significant because it affects the paper's most high-level and broadly cited claim. A practitioner deciding between training a larger model or deploying the compute-optimal framework would need to know that the comparison is against a suboptimal version of the larger model—compute-optimal pretraining or adding test-time compute to the larger model could alter the trade-off substantially.

Evaluation Scale (500 Questions, ~50 Per Difficulty Bin) Provides Limited Statistical Power

The assumption or constraint. All strategy selection and evaluation is performed on a test set of 500 MATH questions. The difficulty estimation procedure bins these into five quintiles of approximately 100 questions each. The compute-optimal policy is selected using two-fold cross-validation within each difficulty bin, meaning strategy selection is based on approximately 50 questions per fold per bin. No confidence intervals, standard deviations, or statistical significance tests are reported for any result.

The consequence. At 50 questions per fold, the strategy selection process has limited statistical power. The "optimal" strategy identified for a given difficulty-budget pair may be the result of noise in a small sample rather than a genuine performance advantage. This is particularly concerning for the highest budgets (where differences between strategies are often small) and for the extreme difficulty bins (where accuracy is low and relative differences are noisy). The paper's key claims—that compute-optimal scaling achieves 4× efficiency gains, that beam search is optimal for medium problems and best-of-N for easy problems—depend on the reliability of the per-bin strategy selection. Without variance estimates, a practitioner cannot assess whether these claims would replicate on a different test set or with a different random split.

What evidence exists in the paper. Section 3.2 describes the cross-validation protocol (two-fold, within-bin). Section 4 specifies the 500-question test set. The paper reports no confidence intervals on the compute-optimal scaling curves in Figures 4 and 8, no per-dataset variance in Figure 3, and no statistical tests comparing strategies. The curves appear smooth, but this could be an artifact of averaging over 250 questions per fold rather than evidence of reliable trends.

Mitigation status. Not addressed. The small sample size is inherent to using MATH as the evaluation benchmark (500 questions total) and the five-bin difficulty discretization. Using a larger test set or reporting bootstrap confidence intervals over the existing 500 questions would partially address this, but neither is done. The paper does not acknowledge this as a limitation, and the compute-optimal policy selection is reported as a point estimate without uncertainty quantification.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a fundamentally new architecture or a single novel training algorithm. Rather, it demonstrates that a carefully orchestrated multi-stage training pipeline, applied to an existing strong vision-language foundation model, can produce a unified multimodal embedding model that matches or exceeds both specialized visual document retrievers and text-only embedding models on their respective benchmarks. The contribution is methodological rather than architectural: the paper provides a recipe—synthesize massive diverse data from a VLM annotator, progressively refine data quality through iterative embedding-model-based mining, distill a cross-encoder's fine-grained relevance judgment into the bi-encoder, then merge checkpoints to recover task balance—that produces state-of-the-art results across four modalities simultaneously.

This shifts the framing of multimodal embedding from a model design problem (what architecture best aligns modalities?) to a training methodology problem (how do you sequentially specialize a general VLM into a retrieval model without destroying its other capabilities?). The finding that model merging can reconcile the retrieval-vs-classification trade-off introduced by distillation is the paper's most conceptually significant result. It suggests that the instinct to seek a single training objective that optimizes all capabilities jointly may be misguided—at least for embedding models trained on diverse task families. Instead, the more effective strategy may be to train specialized variants (one strong at retrieval via distillation, one strong at classification and QA via multi-task contrastive learning) and then compose them through weight-space interpolation. This reframes multi-task embedding training from an optimization problem to a composition problem—train specialists, then merge—which is mechanically simpler (no need to tune loss weights across conflicting objectives) and empirically more effective (Table 6 shows the merged model s3 outperforms both s1 and s2 individually).

The paper also provides a reconciliation of a tension in the multimodal retrieval literature between two competing approaches: general-purpose unified embedding models (VLM2Vec, GME, RzenEmbed) and specialized visual document retrievers (ColPali, ColQwen, ColNomic). The prior state of the field implicitly assumed a trade-off: unified models covered more modalities but underperformed on visual documents, while specialized models dominated visual documents but couldn't handle general images or video. Table 3 shows that Qwen3-VL-Embedding-8B achieves 75.8 average on visual document benchmarks—within 1.9 points of tomoro-colqwen3-embed-8B (77.7), the best ColPali-style model—while simultaneously achieving state-of-the-art results on general image and video tasks where ColPali models cannot operate at all. This effectively dissolves the trade-off: a single general-purpose model can now match specialized approaches on their home turf. The practical implication is that the ColPali-style approach of producing per-patch embeddings for late-interaction scoring, while still slightly ahead on visual documents in the embedding-only setting, may not represent the long-term direction for document retrieval if general-purpose bi-encoders continue to improve at this rate.

A more subtle shift concerns the role of data synthesis in multimodal retrieval. Prior work in multimodal embedding relied primarily on naturally occurring paired data (image-caption datasets, video-description datasets) augmented with rule-based or template-based synthesis for specific tasks. The Qwen3-VL-Embedding approach treats a strong VLM as a general-purpose annotation engine capable of generating diverse, semantically rich training examples across a taxonomy of task paradigms (classification, QA, retrieval, moment retrieval) and a hierarchy of semantic depths (literal description through abstract reasoning). The 300M synthetic examples in Stage 1 cost only the inference compute of Qwen3-VL-32B—a one-time expense amortized over all downstream training. This moves the bottleneck from "how do we collect enough paired multimodal data?" to "how do we curate a sufficiently diverse seed pool and design effective annotation prompts?"—a shift that makes data scaling more tractable because seed pools can be expanded through web crawling, and prompt design is a human-iterable process rather than a data collection bottleneck.

One research direction becomes less attractive as a result of these findings: the development of increasingly specialized retrieval architectures for individual modalities. If a single unified bi-encoder, trained with a multi-stage pipeline on the right data mix, can match specialized models across all modalities, the justification for maintaining separate text, image, video, and document retrieval pipelines weakens substantially. The remaining advantage of specialized models is narrow (a ~2-point gap on visual documents for the embedding-only setting) and disappears entirely when a cross-encoder reranker is added to the pipeline (Table 3: Qwen3-VL-Reranker-8B achieves 80.3 average on visual documents, 2.6 points above the best ColPali model). Organizations that have invested in separate retrieval stacks for different modalities now have a clear migration path toward a unified approach without sacrificing per-modality quality.

Follow-Up Research This Work Enables

Measuring the generalization of the multi-stage-merging recipe beyond Qwen3-VL. The paper demonstrates that model merging reconciles the retrieval-vs-classification trade-off for a specific foundation model (Qwen3-VL) on a specific benchmark suite (MMEB-V2). A strong follow-up would replicate the s1→s2→s3 pipeline using a different VLM backbone—say, LLaVA-NeXT or InternVL2—trained on the same data, and measure whether the pattern of distillation improving retrieval at the cost of classification/QA, with merging recovering balance, holds across architectures. If it does, the recipe generalizes. If it doesn't, the finding is specific to Qwen3-VL's pre-training distribution or architecture, and practitioners using other VLMs would need a different approach. The experiment should report the same MMEB-V2 breakdown as Table 6 so the per-task trade-off can be directly compared.

Isolating the contribution of the seed pool curation to data synthesis quality. The paper describes a five-stage seed pool curation pipeline (quality filtering, structural refinement, category labeling, cross-modal alignment filtering, rebalancing) but ablates none of it. A controlled experiment would train Stage 1 models on synthetic data generated from (a) the fully curated seed pool, (b) the seed pool without cross-modal alignment filtering, (c) the seed pool without category rebalancing, and (d) a random sample of uncurated web images and videos of comparable size. Comparing downstream MMEB-V2 scores would quantify the contribution of each curation step and determine whether the elaborate filtering is essential or merely cosmetic. If alignment filtering contributes little, the synthesis pipeline could be simplified substantially. If rebalancing is critical, the category distribution in Figure 4 becomes a design specification that other practitioners should replicate.

Testing whether hard negative mining benefits plateau after two iterations. The paper's iterative data mining argument—that better embedding models produce better hard negatives, which train even better embedding models—is demonstrated for two iterations (GME→s0→s1). Whether further iterations (s1→s1', s1'→s1'', etc.) would continue to yield improvements or would plateau is unknown. A follow-up would run additional mining-training cycles and measure the performance trajectory. A plateau would suggest that the primary benefit of iterative mining is realized in the first one or two iterations, after which data quality saturates. Continued improvement would suggest that the mining process could replace manual data curation entirely, with the model bootstrapping its own training data quality over many cycles. This experiment would also test whether the safety margin δ− in the hard negative selection (Section 3.3) needs to be adjusted across iterations as the mining model becomes more accurate—a tighter margin might be viable for stronger mining models, enabling selection of harder negatives without risking false negatives.

Evaluating the embedding model's cross-lingual retrieval fairness. The paper claims support for 30+ languages but reports only aggregate MMTEB scores. A fine-grained per-language analysis would reveal whether the multilingual performance is concentrated in high-resource languages (English, Chinese, French, German, etc.) or genuinely distributed across all 30+ supported languages. Specifically: for each language in MMTEB, report retrieval and STS scores separately. A finding that low-resource languages show substantially degraded performance would indicate that the synthetic data generation process—which presumably operates primarily in English or Chinese—does not adequately cover multilingual retrieval scenarios, and that additional language-specific data synthesis or training objectives are needed. This is particularly important for deployment in regions where the supported languages include lower-resource ones.

Cross-encoder reranker threshold analysis: when does the 2B reranker underperform the bi-encoder? Table 5 reveals that the 2B reranker underperforms the embedding baseline on MMEB-V2 video retrieval (74.0 vs. 74.8) and VisDoc retrieval (53.2 vs. 53.6), while strongly outperforming it on text retrieval (83.2 vs. 68.1) and JinaVDR (80.9 vs. 71.0). This suggests a minimum capacity threshold below which cross-encoder reranking is not beneficial, and that this threshold varies by modality. A systematic study would train rerankers at multiple scales (0.5B, 1B, 2B, 4B, 8B) and identify the capacity at which the reranker begins to outperform the bi-encoder baseline for each task type. This would provide practical guidance for practitioners: if you can only afford a 2B reranker, use it for text retrieval but not for video or visual documents; if you need multimodal reranking, budget for the 8B model. The experiment would also test whether the threshold is determined by absolute parameter count or by capacity relative to the sequence length and complexity of the modality (video sequences being much longer than text pairs, requiring more capacity for effective cross-attention).

VLM-as-annotator for other multimodal tasks: how far does the paradigm extend? The paper uses Qwen3-VL-32B to synthesize training data for retrieval, classification, and QA. The same VLM-as-annotator approach could be extended to other multimodal tasks that lack large-scale training data: visual entailment, multimodal summarization, video temporal grounding, or instruction-following in visual environments. A natural follow-up would apply the same seed-pool-plus-structured-prompt methodology to generate training data for one of these tasks, fine-tune a smaller VLM on the synthetic data, and evaluate on an existing benchmark. Success would validate the VLM-as-annotator paradigm as a general-purpose data engine for multimodal tasks. Failure—if synthetic data quality is insufficient for tasks requiring more nuanced reasoning than retrieval—would establish a boundary condition on the approach. The experiment should vary the size of the annotator VLM (e.g., Qwen3-VL-8B, 32B, 72B) to measure how annotation quality scales with annotator capability.

Practical Applications and Downstream Use Cases

Unified multimodal search for enterprise knowledge bases. Organizations maintain document repositories containing interleaved text, images, charts, tables, and video content (e.g., internal wikis, research archives, legal document collections, product catalogs). Currently, searching across these modalities requires separate pipelines or, more commonly, text-only search that ignores visual content entirely. Qwen3-VL-Embedding enables a single embedding index where a text query ("show me the Q3 revenue chart") can retrieve the specific slide containing that chart, a visual document query (a screenshot of an error message) can retrieve related troubleshooting documentation, and a video query can retrieve relevant meeting recordings. The practical deployment numbers from Figure 6 make this viable at enterprise scale: using int8 quantization at 1024 dimensions, a corpus of 10 million items requires approximately 80 GB of storage (10M × 1024 × 1 byte = ~10 GB, plus index overhead) and sub-10ms retrieval latency per query. The 30+ language support means the same system works across multinational organizations without requiring per-language deployments.

E-commerce product discovery across modalities. Online marketplaces contain products described through text (titles, descriptions, attributes), images (product photos, lifestyle shots), and increasingly video (demos, reviews). A unified embedding space allows a user to search with any modality—text description ("waterproof hiking boots under $200"), product image (a photo of boots they saw elsewhere), or even a video clip—and retrieve relevant products regardless of how those products are primarily represented in the catalog. The model's image retrieval score of 80.2 on MMEB-V2 (Table 2) and strong performance on OCR-based tasks (via the QA task training) means it can handle queries that mix visual similarity with text-based attribute matching. The reranker provides an additional precision layer for high-value queries: for a query with an image, the embedding model retrieves 100 candidates, and the reranker re-scores them using cross-attention between the query image and each product's full multimodal representation (images + text description). At the reported latency of 2.87ms per embedding on VL3-Syn (Figure 6) plus reranker inference time, total query latency can remain under 100ms for the full retrieval-plus-reranking pipeline.

Multilingual content moderation with visual understanding. Content moderation systems must detect policy-violating content that spans text, images, and video across dozens of languages. A unified embedding model with 30+ language support can flag multimodal content by computing similarity between user-generated content and a database of known violation examples in a shared embedding space, regardless of the language of the accompanying text. The model's classification and QA capabilities (Image CLS 74.2, Video CLS 78.4 for the 8B model) mean it can also be used for direct categorization when an appropriate classification head or instruction template is applied, reducing the need for separate per-modality, per-language classifiers. The Matryoshka dimension flexibility is particularly relevant here: high-recall screening can use lower-dimensional embeddings (e.g., 256 dims) for fast approximate filtering of the content firehose, while high-precision review of flagged content can use full-dimensional embeddings (4096 dims for the 8B model) with reranking.

Scientific literature search across figures, tables, and text. Scientific papers contain dense interleaved multimodal information: text describing methods and results, figures presenting experimental data, tables summarizing quantitative findings, and equations expressing mathematical relationships. A unified embedding model that processes visual documents (charts, micrographs, diagrams) alongside text enables queries like "find papers showing a dose-response curve for this compound" (where the query is a text description and the target is a figure) or "find papers with similar experimental setups to this one" (where the query is a paper section containing both text and figures). The model's strong performance on visual document benchmarks (VisDoc 82.4 on MMEB-V2, 75.8 average on the six specialized visual document benchmarks in Table 3) and competitive text-only performance (67.9 on MMTEB, Table 4) means a single system can search across all components of a scientific paper without needing to extract and index text, figures, and tables separately. The long-context support (32K tokens) enables encoding entire paper sections or even full short papers as single embeddings, enabling document-level similarity search that captures the interaction between textual arguments and visual evidence.