ArXiv: 2509.18095

🎯 Pitch

Current multimodal retrieval forces an impossible choice between fast but information-poor single-vector models and accurate but prohibitively expensive multi-vector methods. MetaEmbed shatters this trade-off by letting users dial retrieval quality up or down at test time—gaining up to 6.6 accuracy points on a single model simply by toggling the number of embedding vectors used.

1. Executive Summary

This paper introduces MetaEmbed, a framework for multimodal retrieval that replaces conventional single-vector or patch-level multi-vector representations with a small set of learnable Meta Tokens whose last-layer hidden states serve as compact, contextualized embeddings for late interaction. The system is trained with Matryoshka Multi-Vector Retrieval (MMR) — a prefix-nested contrastive objective that organizes information across multiple granularities, enabling test-time scaling where users select how many token groups to use for indexing and scoring (e.g., coarse retrieval with 1 query vector and 1 candidate vector vs. fine-grained retrieval with 16 query vectors and 64 candidate vectors). Evaluated on the Massive Multimodal Embedding Benchmark (MMEB) using Qwen2.5-VL backbones, MetaEmbed-7B achieves 76.6% average precision@1 — outperforming the strongest single-vector baseline by over 5 absolute points — while MetaEmbed-32B reaches 78.7%, and the framework scales test-time performance by up to 6.6 points from single-vector to full multi-vector budget on the largest model, establishing that multi-vector late interaction with nested granularity provides compounding accuracy benefits as model capacity grows, but only when the underlying VLM backbone possesses robust cross-modal capabilities.

2. Context and Motivation

The Core Problem: Multimodal Retrieval Is Stuck Between Two Unsatisfactory Extremes

The fundamental problem MetaEmbed addresses is a representation bottleneck in multimodal retrieval systems. When you search for a photo using a text description, or find a relevant document based on an image, a retrieval model must somehow compare two fundamentally different types of data — pixels and words — and determine whether they are semantically related. The central design question is: what representation should the model produce for each input such that similarity computation is both expressive enough to capture fine-grained relevance and efficient enough to run at scale?

This tension manifests as a choice between two extremes, both of which are unsatisfactory:

Single-vector retrieval (the dominant paradigm). Most production multimodal embedding models — CLIP (Radford et al., 2021), SigLIP (Zhai et al., 2023), VLM2Vec (Jiang et al., 2025), and many others — compress an entire input (a full image with its thousands of visual details, or a complex query with nuanced intent) into a single fixed-dimensional vector. Retrieval then reduces to a dot product between two such vectors, which is computationally trivial. This approach is fast and scalable — you can index millions of candidates and search them in milliseconds — but it suffers from a fundamental limitation identified by concurrent theoretical work (Weller et al., 2025): condensing rich multimodal content into one vector inevitably loses fine-grained information. When a query asks about a specific detail in a complex image ("the red bicycle in the background of the street scene"), a single global summary vector may fail to isolate that detail from the surrounding context. The paper cites Yao et al. (2022) and Thrush et al. (2022) as providing empirical evidence that fine-grained cross-modal alignment is lost in single-vector representations.

Full token-level multi-vector retrieval (the expressive but impractical alternative). The opposite approach, pioneered by ColBERT (Khattab and Zaharia, 2020) in text retrieval and extended to vision-language by ColPali and ColQwen (Faysse et al., 2025), retains every token-level embedding from the last transformer layer and computes a late interaction score: for each query token, find the maximum similarity with any candidate token, then sum across query tokens (the "MaxSim" operator in Equation 2). This preserves significantly more contextual information — the query token "bicycle" can directly match against visual patch tokens that represent the bicycle in the image, without that signal being averaged away by a global pooling operation. The problem is scale. Each image is typically encoded into hundreds of patch embeddings (e.g., a 1024-patch visual encoding produces 1024 vectors), and each text query into dozens of token embeddings. For a retrieval corpus of NN candidates, the index stores N×(number of candidate vectors)×DN \times (\text{number of candidate vectors}) \times D floats. The scoring cost per query-candidate pair scales as O(Nq×Nc×D)O(N_q \times N_c \times D), where NqN_q and NcN_c are the number of token-level vectors on each side. This makes retrieval with large indexes computationally expensive in both storage and latency, and it creates a more severe problem the paper emphasizes: multimodal-to-multimodal retrieval becomes effectively infeasible. When both the query and the candidate contain images — say, searching for visually similar documents where both sides have hundreds of image patches — the pairwise token interaction matrix explodes to thousands-by-thousands comparisons. Training with this cost (and doing it at inference time for every candidate) is "prohibitive due to computational demands" (Section 1).

Why This Matters: The Gap Between Research Expressiveness and Production Practicality

This is not merely an academic tradeoff. Multimodal retrieval underpins critical real-world systems: image search engines (Gordo et al., 2016), visual question answering (Hu et al., 2018), visual document retrieval for enterprise search and legal discovery (Faysse et al., 2025; Macé et al., 2025), and retrieval-augmented generation pipelines that ground LLM outputs in visual evidence. These applications have contradictory demands:

  • Accuracy requires representing fine-grained visual and textual details — a medical document retrieval system must distinguish between two visually similar X-rays based on subtle textual annotations, which single-vector representations often fail to capture.
  • Deployability requires that the index fits in memory (GPU or CPU) and that query latency stays within acceptable bounds (tens of milliseconds for interactive applications, not seconds). Full multi-vector methods like ColPali produce indexes that are 64-128× larger than single-vector indexes, which may not scale to production corpus sizes of millions or billions of items.

The gap between these demands is widening because foundation vision-language models (VLMs) are getting dramatically better at understanding rich multimodal content — Qwen2.5-VL, Llama-3.2-Vision, and PaliGemma can all process high-resolution images with nuanced text prompts — but the retrieval community has no good way to translate this improved understanding into a representation that is simultaneously expressive and compact. The paper's framing makes this explicit: "condensing the entire query and candidate into a single vector is not an optimal choice, as fine-grained details are lost between modalities" (Section 1), yet the alternative of "hundreds of patch embeddings" is "prohibitive." The field is stuck.

Prior Approaches and Their Specific Limitations

The paper situates its contribution against a landscape of existing methods, each of which addresses only part of the problem:

Single-vector VLM-based embedders (strong but bottlenecked). Recent methods like VLM2Vec (Jiang et al., 2025), GME (Zhang et al., 2024b), mmE5 (Chen et al., 2025b), and MoCa (Chen et al., 2025a) have pushed state-of-the-art multimodal retrieval by fine-tuning pretrained VLMs with contrastive objectives, extracting the last-token hidden state or a pooled representation as the retrieval vector. These models achieve impressive results — MoCa-7B reaches 71.5% on MMEB, mmE5 achieves 69.8% (Table 1) — but they remain fundamentally limited by the single-vector bottleneck. As the paper notes, scaling to larger VLMs produces diminishing returns for these methods: "the improvement of the single-vector baseline from 7B to 32B is no longer statistically significant" (Section 4.3, discussing Figure 4a), suggesting that the single-vector representation has saturated and cannot absorb the additional capacity of larger models. The information bottleneck is not a matter of model quality; it is a matter of representation dimensionality.

Token-level multi-vector methods (expressive but unscalable). ColPali (Faysse et al., 2025) and its variants (ColQwen, Llama Nemoretriever Colembed from Xu et al., 2025) adopt the ColBERT late-interaction framework directly for vision-language retrieval: they encode every visual patch and every text token as separate embeddings, store all of them, and score via MaxSim. These methods perform strongly on visual document retrieval tasks (ColPali achieves 54.5% NDCG@5 on ViDoRe v2, Table 2) where fine-grained alignment between text queries and document image regions is essential. However, they have three specific weaknesses that MetaEmbed identifies:

  1. Index size explosion: With hundreds of patch embeddings per image, the index scales linearly with the number of visual tokens, making large-scale deployment expensive. The paper quantifies this in Table 3: even MetaEmbed's own (16, 64)-vector configuration requires 42.72 GiB of index memory for 100,000 candidates. A ColPali-style encoding with, say, 1024 patch vectors per image would require roughly 1024/64≈16×1024/64 \approx 16\times more — close to 683 GiB for the same corpus, which would not fit on a single A100 GPU.

  2. Multimodal-to-multimodal infeasibility: When both query and candidate contain images, the pairwise token comparison matrix has size (Pq+Tq)×(Pc+Tc)(P_q + T_q) \times (P_c + T_c) where PP denotes patch tokens and TT denotes text tokens. With hundreds of tokens on each side, this becomes computationally prohibitive for both training and inference — the paper explicitly states that "naive ColBERT-style methods... do not support multimodal-to-multimodal retrieval" (Section 2, Related Work).

  3. No flexibility: Once you encode at full token granularity, you must store and score at full granularity. There is no mechanism to degrade gracefully under tighter computational constraints — you cannot say "use a coarse approximation for this query" or "store a compressed index for infrequently accessed candidates."

Matryoshka Representation Learning (MRL) — applied only to single vectors, not multi-vector retrieval. MRL (Kusupati et al., 2022) introduced the idea that a single vector can be trained to be informative at multiple prefix lengths: the first d1d_1 dimensions form a coarse representation, the first d2>d1d_2 > d_1 dimensions add detail, and so on, with each prefix independently useful. This has been adopted in text-only single-vector retrieval models (Zhang et al., 2025; Günther et al., 2025) where users can choose the embedding dimensionality at query time. However, MRL has not been applied to multi-vector retrieval — that is, to the number of vectors rather than the dimension of a single vector. The paper notes that prior work (Cai et al., 2025) applied Matryoshka-style methods to VLM generation token budgeting, but "to the best of our knowledge, MetaEmbed presents the first work that leverages such a framework for multi-vector retrieval and achieves successful test-time scaling" (Section 2). This gap is critical because the number of vectors (not their dimensionality) is the primary scalability knob for multi-vector methods: reducing from 1024 vectors to 1 vector per image compresses the index by 1024×, whereas MRL-style dimensionality reduction typically achieves only 2–8× compression.

How MetaEmbed Positions Itself

MetaEmbed's positioning is best understood as occupying the empty middle ground between single-vector and full-token multi-vector retrieval, while adding a new axis of flexibility that neither existing approach provides. The paper's contributions address each identified gap:

Against single-vector methods: MetaEmbed claims that a small set of learned Meta Tokens (e.g., 16 for queries, 64 for candidates) provides sufficient expressive capacity to capture fine-grained cross-modal alignment without requiring hundreds of patch-level embeddings. The key insight is that these tokens are not derived from the input tokens via pooling or partitioning (which would inherit the single-vector bottleneck), but are instead learned parameters that attend to the full input during the transformer forward pass and thus can selectively extract relevant information into a compact set of vectors. This is analogous to how [CLS] tokens in BERT or register tokens in vision transformers learn to aggregate information, but extended to produce multiple complementary vectors rather than one.

Against full multi-vector methods: MetaEmbed replaces hundreds or thousands of patch-level vectors with a fixed, small number of learned vectors (16–64), making the approach practical for large-scale indexing and enabling multimodal-to-multimodal retrieval where both sides are image-based. The computational savings are dramatic: MetaEmbed's scoring cost for a (16, 64) configuration involves 16×64=1,02416 \times 64 = 1,024 pairwise comparisons, versus potentially 1,024×1,024≈1061,024 \times 1,024 \approx 10^6 for full patch-level interaction. The paper's efficiency analysis (Table 3) shows that even at the full (16, 64) budget, scoring latency is only 6.25 ms per 100,000 candidates on an A100 — fast enough for production use. Encoding an image query dominates at 788 ms anyway, so the scoring cost is negligible by comparison.

A new axis: test-time scalable multi-vector retrieval. The novel contribution that distinguishes MetaEmbed from both extremes is the Matryoshka Multi-Vector Retrieval (MMR) training framework. By training with parallel contrastive objectives across nested prefix groups — (1,1)(1, 1), (2,4)(2, 4), (4,8)(4, 8), (8,16)(8, 16), (16,64)(16, 64) — the model learns to organize its Meta Embeddings hierarchically: the first vector captures global semantics, the first two capture somewhat finer detail, and so on, with each progressive group size adding discriminative power. This enables something neither single-vector nor full multi-vector methods can do: test-time adaptive scaling, where the retrieval system can operate at different accuracy-efficiency operating points depending on the query load, latency requirements, or available index memory, without any retraining or model changes (Figure 2). A user can query with coarse (1, 1) budgets for fast filtering, then rerank top candidates with a larger budget, or permanently index at (4, 8) for storage-constrained deployments while serving queries at their original (16, 64) budget. This flexibility is the paper's central technical novelty.

Positioning within the broader training-inference tradeoff landscape. The paper's framing — test-time scaling in retrieval — connects conceptually to a broader trend in machine learning where inference compute is treated as a variable resource rather than a fixed cost. Just as the LLM literature has explored compute-optimal test-time strategies (e.g., scaling inference compute via beam search or best-of-N sampling to substitute for larger pretrained models), MetaEmbed proposes a retrieval-specific analog: scale the number of embedding vectors used for late interaction at test time to trade off between accuracy and computational cost. This positions the work as part of a shift toward systems where deployment decisions (accuracy vs. latency vs. memory) can be made dynamically without retraining, which is particularly valuable for multimodal retrieval where different use cases (real-time search vs. offline batch processing vs. long-term archival indexing) have wildly different resource constraints.

A Critical Subtlety: The VLM Backbone Matters — and Not All Architectures Are Equal

The paper's experimental design reveals an important constraint that it is upfront about but which is easy to overlook: MetaEmbed is a training recipe that can be applied to any VLM, but the quality of the resulting embeddings is heavily dependent on the underlying model's cross-modal capabilities. The Llama-3.2-Vision-based MetaEmbed-11B achieves only 65.1 on MMEB, despite being a strong model in absolute terms, because its VQA score drops to 42.1 — "more than 32 points lower than the Qwen2.5-VL-initialized 7B model" (Section 4.2). The paper observes that "if the underlying base model itself struggles on some domains when used as a generative model, such a weakness directly propagates into MetaEmbed as an embedding model." This is not a flaw in MetaEmbed but a design reality: the Meta Tokens synthesize information from the VLM's internal representations, so if those representations are weak for certain modalities or tasks, the Meta Embeddings will inherit that weakness. This contrasts with some prior single-vector methods that train heavily from scratch with massive contrastive data (e.g., CLIP trained on 400M pairs) and can partially overcome backbone limitations through data scale. MetaEmbed's lighter-weight fine-tuning (LoRA with rank 32, trained on MMEB-train + ViDoRe-train only) preserves the backbone's strengths and limitations more faithfully. This means that selecting the right backbone VLM is a first-order decision when deploying MetaEmbed, not an afterthought — the paper implies that Qwen2.5-VL is the recommended choice given its balanced cross-modal performance.

Summary of the Motivation

The paper addresses a clear, well-motivated gap: multimodal retrieval systems must choose between compact but lossy single-vector representations and expressive but unscalable multi-vector representations, with no good intermediate option and no flexibility to adapt to varying resource constraints. MetaEmbed proposes to fill this gap by replacing input-derived token embeddings with a small set of learned Meta Tokens, training them with a nested granularity objective (MMR) so that information is organized hierarchically, and thus enabling the first test-time scalable multi-vector retrieval system where users can dial the accuracy-efficiency tradeoff at deployment time without retraining. The empirical context makes the motivation concrete: the best single-vector method (MoCa-7B at 71.5% on MMEB) is saturating as models scale, while full multi-vector methods (ColPali) are too expensive for large indexes and cannot handle multimodal queries. MetaEmbed attempts to resolve both problems simultaneously.

3. Technical Approach

3.1 Reader Orientation

MetaEmbed is a training recipe applied to a pretrained vision-language model (VLM) that teaches the model to produce a small set of compact, flexible embedding vectors suitable for multi-vector retrieval with late interaction. The system solves the problem of multimodal retrieval being stuck between two unsatisfactory extremes — single-vector representations that are fast but lose fine-grained detail, and full token-level multi-vector representations that are expressive but prohibitively expensive to store and score — by introducing a configurable number of learned "Meta Tokens" whose contextualized outputs form a nested hierarchy of representations, enabling users to select how many vectors to use at test time depending on their accuracy-versus-efficiency needs, without any model retraining.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

  1. Pretrained VLM Backbone — a vision-language model (e.g., Qwen2.5-VL, PaliGemma, Llama-3.2-Vision) that processes interleaved image and text inputs through its transformer layers. It provides the foundational multimodal understanding that MetaEmbed builds upon, generating hidden-state representations for all input tokens (visual patches, text tokens, and the learnable Meta Tokens). The VLM is frozen in its core parameters and adapted only through low-rank (LoRA) fine-tuning.

  2. Learned Meta Tokens — a small, fixed number of trainable embedding vectors (e.g., 16 for queries, 64 for candidates) that are appended to the input sequence before processing by the VLM. Unlike patch or text tokens, these tokens carry no input-specific information initially; they learn during training to attend to the full multimodal input and extract relevant information into their final-layer hidden states. The Meta Tokens on the query side and candidate side are separate parameter sets, each optimized for their respective roles.

  3. Matryoshka Multi-Vector Retrieval (MMR) Module — a training objective that applies contrastive learning across multiple nested prefix groups of the Meta Embeddings simultaneously. For example, the first 1 query vector and first 1 candidate vector form group 1, the first 2 query vectors and first 4 candidate vectors form group 2, continuing up to the full (16, 64) group. Each group computes its own late-interaction score and contrastive loss, enforcing that the embeddings are organized from coarse to fine granularity. This is the mechanism that enables test-time flexibility.

  4. Test-Time Retrieval Engine — at inference, the user selects a retrieval budget $(r_q, r_c)$ specifying how many Meta Embeddings to use on the query side and candidate side. The system computes a late-interaction MaxSim score between the selected query vectors and the selected candidate vectors, producing a relevance score. The same trained model serves all budget choices without reconfiguration. For indexing, the system stores only the first $r_c$ vectors for each candidate; for scoring, it uses only the first $r_q$ query vectors.

Information flows as follows: a multimodal query (e.g., text + image) enters the system → the query's Meta Tokens are appended to the input → the VLM processes the combined sequence through all transformer layers → the last-layer hidden states at the Meta Token positions are extracted as the query's Meta Embeddings → independently, each candidate in the corpus is encoded the same way and its Meta Embeddings are stored in the index → at retrieval time, the selected $(r_q, r_c)$ prefix of query and candidate Meta Embeddings are compared via MaxSim late interaction → a relevance score is produced, and candidates are ranked.

3.3 Roadmap for the Deep Dive

  • First, the formal definition of the multimodal retrieval problem (Equation 1) and the late interaction operator (Equation 2), since these are the mathematical foundations that MetaEmbed builds upon and modifies.
  • Second, the MetaEmbed architecture — how Meta Tokens are integrated into the VLM input, how the model produces Meta Embeddings, and why learned tokens are used instead of input-derived representations — because this is the core structural innovation.
  • Third, the Matryoshka Multi-Vector Retrieval (MMR) training framework, including the nested group structure, the group-specific late-interaction scoring (Equations 3–4), and the contrastive loss (Equations 5–7) — since this is the mechanism that enables test-time scaling and is the paper's main algorithmic contribution.
  • Fourth, the test-time scaling mechanism itself — how the trained model supports arbitrary retrieval budgets, what choosing a budget means operationally, and the efficiency-accuracy tradeoff that becomes possible — to show how the training design translates into deployment flexibility.
  • Fifth, the training configuration details (LoRA, batch size, optimizer, learning rate, data) and design choices that make the recipe work in practice, including important implementation decisions like gradient checkpointing and distributed training strategy.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that a small set of learned Meta Tokens, trained with a nested multi-granularity contrastive objective (MMR), can replace both single-vector pooling and full patch-level multi-vector encoding in multimodal retrieval, providing a flexible middle ground that scales at test time.

Formal Problem Definition and Late Interaction

The paper frames multimodal retrieval as a maximum-similarity search problem defined by Equation 1:

c⋆=arg⁡max⁡c∈{c1,…,cN}s(q,c)c^\star = \arg\max_{c \in \{c_1, \ldots, c_N\}} s(q, c)

where $q$ is a multimodal query (which can be text $q_t$, an image $q_i$, or a combination $(q_t, q_i)$), $\{c_1, \ldots, c_N\}$ is a set of $N$ retrieval candidates of arbitrary modality, $s(q, c)$ is a similarity function that measures semantic relevance between $q$ and $c$, and $c^\star$ is the top-ranked (most relevant) candidate.

What it computes: the identity of the candidate that maximizes the similarity score with the query. The model searches over all $N$ candidates, computes $s(q, c_i)$ for each, and returns the one with the highest score.

Why this form: this is the standard definition of the retrieval task — a ranking problem where the similarity function $s(\cdot, \cdot)$ is what the embedding model defines. The challenge (and what MetaEmbed addresses) is entirely in how $s(q, c)$ is computed from the raw inputs, because that computation must balance expressiveness (capturing fine-grained cross-modal relevance) against efficiency (scoring $N$ candidates quickly for large $N$).

The late interaction mechanism that MetaEmbed uses for scoring is defined by Equation 2:

LI(q,d)=∑i=1Nqmax⁡j∈[1,Nd]⟨Eq(i),Ed(j)⟩\text{LI}(q, d) = \sum_{i=1}^{N_q} \max_{j \in [1, N_d]} \left\langle E^{(i)}_q, E^{(j)}_d \right\rangle

where $E_q \in \mathbb{R}^{N_q \times D}$ is the multi-vector representation of the query (consisting of $N_q$ vectors, each of dimension $D$), $E_d \in \mathbb{R}^{N_d \times D}$ is the multi-vector representation of the candidate (consisting of $N_d$ vectors of dimension $D$), $E^{(i)}_q$ is the $i$-th query vector, $E^{(j)}_d$ is the $j$-th candidate vector, $\langle \cdot, \cdot \rangle$ denotes the dot product (cosine similarity when vectors are L2-normalized), and $\text{LI}(q, d)$ is the scalar relevance score.

What it computes: for each query vector, find the candidate vector that is most similar to it (the $\max_{j}$ operation over all $N_d$ candidate vectors), take that maximum similarity score, and sum these maximum similarities across all $N_q$ query vectors. The result is a single number: higher means the query's information is better aligned with some aspect of the candidate. This is the "MaxSim-then-sum" operator that defines ColBERT-style late interaction.

Why this form: the max operation allows each query vector to independently find its best match in the candidate, regardless of where that match occurs. This is crucial for cross-modal alignment because a query token like "bicycle" can match against whichever visual patch in the candidate image contains bicycle-related features, without being forced to align positionally. The sum over query vectors ensures that all parts of the query contribute to the score — the model cannot ignore query terms. The asymmetry (sum over query vectors, max over candidate vectors) is deliberate: the query drives the retrieval, so every aspect of the query should be accounted for, while the candidate can be partially relevant (only some regions match). An alternative symmetric form (e.g., summing over both sides) would penalize candidates that contain the relevant information but also contain unrelated content, which is inappropriate for retrieval where distractors are common.

A critical practical note: in MetaEmbed, the vectors $E_q$ and $E_d$ are not derived from the input tokens (patches or text tokens). They are produced by the learned Meta Tokens. This substitution is the key architectural difference from ColBERT-style methods, and it changes the scaling properties dramatically: $N_q$ and $N_d$ are small constants (e.g., 16 and 64) chosen by the designer, not variable quantities that grow with input length (hundreds of patches, dozens of text tokens). The late interaction formula is identical; what changes is what the vectors represent and how many there are.

The MetaEmbed Architecture: Meta Tokens and Meta Embeddings

The central architectural innovation in MetaEmbed is the introduction of learnable Meta Tokens — a small, fixed set of embedding vectors that are appended to the input sequence before it enters the VLM, and whose contextualized outputs after the full transformer forward pass become the retrieval representations.

Input construction. Given a multimodal input consisting of text $x = [x_1, \ldots, x_n]$ (where $n$ is the number of text tokens) and associated images $I$, the system first tokenizes the text into token embeddings and the images into visual patch embeddings (using the VLM's standard vision encoder and patch projection). Let $v \in \mathbb{R}^{P \times D}$ be the $P$ visual patch tokens and $t \in \mathbb{R}^{n \times D}$ be the $n$ text tokens, each of dimension $D$ (the VLM's hidden size). The paper introduces two sets of learnable parameters:

Mq∈RRq×D,Mc∈RRc×DM_q \in \mathbb{R}^{R_q \times D}, \quad M_c \in \mathbb{R}^{R_c \times D}

where $R_q$ is the number of Meta Tokens on the query side (e.g., 16), $R_c$ is the number on the candidate side (e.g., 64), and each row is a $D$-dimensional embedding initialized randomly and trained. The complete input sequence to the transformer is the concatenation:

z(0)=[v  ;  t  ;  Mq  ;  Mc]∈R(P+n+Rq+Rc)×Dz^{(0)} = [v \;;\; t \;;\; M_q \;;\; M_c] \in \mathbb{R}^{(P + n + R_q + R_c) \times D}

where $[ \cdot ; \cdot ]$ denotes concatenation along the sequence dimension. This means the VLM sees the visual patches, then the text tokens, then the query Meta Tokens, then the candidate Meta Tokens, as a single flat sequence of token embeddings at layer 0.

Forward pass and extraction. The transformer network $F_\theta$ (where $\theta$ denotes the VLM parameters, adapted via LoRA) processes this input through all its layers:

H=Fθ(z(0))∈R(P+n+Rq+Rc)×DH = F_\theta(z^{(0)}) \in \mathbb{R}^{(P + n + R_q + R_c) \times D}

The matrix $H$ contains the last-layer hidden state for every position in the sequence. The critical extraction step is to isolate the hidden states at the positions corresponding to the Meta Tokens. Since the Meta Tokens were appended at known positions (after the visual patches and text tokens), their final representations are:

Emeta(q)=H[positions of Mq]∈RRq×DE^{(q)}_{\text{meta}} = H[\text{positions of } M_q] \in \mathbb{R}^{R_q \times D} Emeta(c)=H[positions of Mc]∈RRc×DE^{(c)}_{\text{meta}} = H[\text{positions of } M_c] \in \mathbb{R}^{R_c \times D}

These extracted matrices are the Meta Embeddings — the multi-vector representations used for retrieval. Each row is a $D$-dimensional vector that has been contextualized by attending to all visual patches, all text tokens, and all other Meta Tokens through the full depth of the transformer. The paper applies L2 normalization to each vector after extraction (making the dot product equivalent to cosine similarity in the late interaction computation). The query and candidate are encoded in separate forward passes — the query is processed with its Meta Tokens to produce $E^{(q)}_{\text{meta}}$, and each candidate is processed independently (possibly offline, at indexing time) with its Meta Tokens to produce $E^{(c)}_{\text{meta}}$. The concatenation $[v; t; M_q; M_c]$ is the input format for both passes, but at inference time, only the query-side or candidate-side Meta Embeddings are retained depending on which pass is being performed.

Why learned Meta Tokens instead of input-derived representations? The paper identifies three specific problems with using the input tokens (visual patches, text tokens) directly as multi-vector representations, and the Meta Token design addresses all three:

  1. Cardinality control: Input-derived representations have variable and potentially large cardinality — an image might have 1,024 patches, a long document might have hundreds of text tokens. Meta Tokens fix the cardinality to a small designer-chosen constant $(R_q, R_c)$, making the index size and scoring cost predictable and bounded regardless of input complexity. This is what makes multimodal-to-multimodal retrieval feasible: both query and candidate sides are represented by at most $R_q$ and $R_c$ vectors respectively, even when both contain images.

  2. Learned information aggregation: Unlike patch tokens (which represent local image regions) or text tokens (which represent individual words or subwords), Meta Tokens have no predetermined semantic role. Through training, they learn to attend to different aspects of the input and extract complementary information. The paper's nested training (MMR, described next) further encourages this: the first Meta Token learns to capture a global summary, subsequent tokens add progressively finer detail. This is fundamentally different from taking the mean of input token segments (the "split" baseline in Appendix B, Table 5), which cannot learn which information to preserve and which to discard.

  3. Parameter efficiency: The Meta Tokens are a tiny fraction of the total parameters. For MetaEmbed-7B with $D = 3,584$, the query and candidate Meta Tokens together contribute $(16 + 64) \times 3,584 = 286,720$ parameters — negligible compared to the 7 billion backbone parameters. The system relies on the VLM's pretrained multimodal understanding to provide rich input features; the Meta Tokens only need to learn how to extract and organize these features, not how to process vision and language from scratch.

Comparison with alternatives considered. The paper explicitly tests a naive multi-vector baseline (Appendix B, "split-(16, 64)") where the query-side last-layer hidden states are evenly partitioned into 16 segments and averaged into 16 vectors, and similarly for 64 candidate-side vectors. This baseline introduces no additional parameters and serves as a fixed-length multi-vector method. MetaEmbed outperforms it substantially — for MetaEmbed-7B on MMEB, the split baseline achieves 70.1% versus MetaEmbed's 76.6% (Table 5). The gap demonstrates that learned aggregation (Meta Tokens) is significantly more effective than static pooling for producing discriminative multi-vector representations. The Meta Tokens learn to allocate representational capacity where it matters; average pooling treats all input regions equally regardless of relevance.

Matryoshka Multi-Vector Retrieval (MMR): The Nested Training Objective

MMR is the training framework that gives MetaEmbed its test-time scalability. The core idea is to train the model such that the Meta Embeddings are organized in a prefix-nested hierarchy of increasing granularity: the first vector alone is a usable coarse representation, the first two vectors together provide more detail, the first four even more, and so on, with each successive prefix building on the information captured by earlier ones. This is achieved by computing multiple late-interaction scores and contrastive losses in parallel during training, one for each prefix group.

Nested group structure. The paper defines $G$ groups (e.g., $G = 5$), each specified by a pair of sizes $(r^{(g)}_q, r^{(g)}_c)$ for $g = 1, \ldots, G$, satisfying:

1≤rq(1)<rq(2)<⋯<rq(G)=Rq1 \leq r^{(1)}_q < r^{(2)}_q < \cdots < r^{(G)}_q = R_q 1≤rc(1)<rc(2)<⋯<rc(G)=Rc1 \leq r^{(1)}_c < r^{(2)}_c < \cdots < r^{(G)}_c = R_c

The specific groups used in experiments are $G = 5$ with:

{(rq,rc)}g=15={(1,1),(2,4),(4,8),(8,16),(16,64)}\{(r_q, r_c)\}_{g=1}^5 = \{(1, 1), (2, 4), (4, 8), (8, 16), (16, 64)\}

The first group (1, 1) operates as a single-vector retrieval system: one query vector, one candidate vector, dot product similarity. The second group (2, 4) uses the first 2 query vectors and first 4 candidate vectors. The pattern continues to the full budget (16, 64). Critically, these groups are prefixes: group $g$ always uses the first $r^{(g)}_q$ query vectors and first $r^{(g)}_c$ candidate vectors. This enforces that earlier vectors in the sequence must carry information that is useful on its own (for small groups) while later vectors add complementary detail (for larger groups). The prefixes are not disjoint — vector 1 is used in all groups, vector 2 in groups 2–5, vector 16 only in group 5.

Group-specific late-interaction scoring. For a given group $g$, the query's group-$g$ representation is the first $r^{(g)}_q$ rows of $E^{(q)}_{\text{meta}}$:

E(q,g)=Emeta(q)[1:rq(g),:]∈Rrq(g)×DE^{(q, g)} = E^{(q)}_{\text{meta}}[1:r^{(g)}_q, :] \in \mathbb{R}^{r^{(g)}_q \times D}

and similarly for the candidate:

E(c,g)=Emeta(c)[1:rc(g),:]∈Rrc(g)×DE^{(c, g)} = E^{(c)}_{\text{meta}}[1:r^{(g)}_c, :] \in \mathbb{R}^{r^{(g)}_c \times D}

The group-specific late-interaction score is then computed identically to Equation 2 but restricted to the group's prefix vectors (Equation 4):

s(g)(q,c)=∑i=1rq(g)max⁡j∈[1,rc(g)]⟨Eq(g,i),Ec(g,j)⟩s^{(g)}(q, c) = \sum_{i=1}^{r^{(g)}_q} \max_{j \in [1, r^{(g)}_c]} \left\langle E^{(g,i)}_q, E^{(g,j)}_c \right\rangle

where $E^{(g,i)}_q$ is the $i$-th row (vector) of $E^{(q,g)}$, $E^{(g,j)}_c$ is the $j$-th row of $E^{(c,g)}$, and $\langle \cdot, \cdot \rangle$ is the dot product between L2-normalized vectors.

What it computes: for each group size, independently compute a late-interaction relevance score between the query and candidate using only the first $r^{(g)}_q$ query vectors and first $r^{(g)}_c$ candidate vectors. Group 1 (single-vector) computes a single dot product. Group 5 (full budget) computes the MaxSim sum over 16 query vectors against 64 candidate vectors. Each group produces its own scalar score that uses a different amount of the available representational capacity.

Why this nested prefix structure: the design ensures that the representations are coherent across granularities. Because vector 1 must be useful in group (1,1) and in group (16,64), it is forced to encode information that is both globally discriminative (good enough on its own for coarse matching) and complementary to later vectors (not redundant with what vectors 2–16 add). If the groups were non-nested (e.g., group 1 used vectors 1–2, group 2 used vectors 3–6), there would be no guarantee that a small prefix is useful on its own — you might get a model where the first few vectors only make sense in the context of later vectors. The nesting property is what enables test-time truncation: you can drop later vectors and the remaining prefix is still a coherent, trained representation.

Contrastive training objective. For each group $g$, the paper applies an InfoNCE-style contrastive loss (Oord et al., 2018) with in-batch negatives and one explicit hard negative per query. Let a minibatch consist of $B$ triplets $(q^{(b)}, c^{(b)}, c^{(b,-)})$ where $c^{(b)}$ is the positive (correct) candidate for query $b$, and $c^{(b,-)}$ is an additional hard negative candidate. For group $g$, define the scaled similarity between query $u$ and candidate $v$ (Equation 5):

Su,v(g)=1τs(g)(q(u),c(v))S^{(g)}_{u,v} = \frac{1}{\tau} s^{(g)}\left(q^{(u)}, c^{(v)}\right)

where $\tau > 0$ is a temperature hyperparameter that controls the sharpness of the softmax distribution (set to $\tau = 0.03$ in experiments). The InfoNCE loss for group $g$ is then (Equation 6):

LNCE(g)=−1B∑u=1Blog⁡exp⁡(Su,u(g))exp⁡(Su,u(g))+∑v≠uexp⁡(Su,v(g))+exp⁡(1τs(g)(q(u),c(u,−)))\mathcal{L}^{(g)}_{\text{NCE}} = -\frac{1}{B} \sum_{u=1}^{B} \log \frac{\exp(S^{(g)}_{u,u})}{\exp(S^{(g)}_{u,u}) + \sum_{v \neq u} \exp(S^{(g)}_{u,v}) + \exp\left(\frac{1}{\tau} s^{(g)}(q^{(u)}, c^{(u,-)})\right)}

where $S^{(g)}_{u,u}$ is the scaled similarity between query $u$ and its correct candidate, $\sum_{v \neq u} \exp(S^{(g)}_{u,v})$ sums over all other candidates in the batch (in-batch negatives), and the final term $\exp(\frac{1}{\tau} s^{(g)}(q^{(u)}, c^{(u,-)}))$ is the contribution from the explicit hard negative candidate for query $u$.

What it computes: for each query in the batch, the loss encourages the model to assign a high similarity to its positive candidate and low similarities to all negatives (both in-batch and the explicit hard negative). The softmax over similarities converts the scores into a probability distribution over which candidate is correct, and the negative log-likelihood penalizes the model when it assigns significant probability to incorrect candidates. The hard negative term is added to the denominator, meaning the model is explicitly penalized if the hard negative's similarity is high — this focuses learning on difficult distinctions that in-batch negatives alone might not cover.

Why this form: the InfoNCE loss with both in-batch and explicit hard negatives is standard in modern contrastive retrieval training. In-batch negatives provide computational efficiency (reusing the candidates already in the batch as negatives, no separate negative sampling needed) and a large number of negatives per query ($B - 1$ in-batch negatives plus one hard negative). The explicit hard negative is important because random in-batch negatives may be trivially distinguishable from positives — the model would not learn fine-grained distinctions. The hard negative, presumably selected to be superficially similar to the correct candidate but semantically distinct (the paper uses hard negatives from Chen et al., 2025a), forces the model to attend to subtle differences that matter for retrieval accuracy.

Combined MMR loss. The final training objective is a weighted sum of the group-specific losses (Equation 7):

Lfinal=∑g=1GwgLNCE(g)\mathcal{L}_{\text{final}} = \sum_{g=1}^{G} w_g \mathcal{L}^{(g)}_{\text{NCE}}

where $w_g$ is a group-specific importance weight. The paper sets all $w_g = 1$ following Kusupati et al. (2022), giving equal weight to each granularity level.

What it computes: a single scalar loss that is the sum of five independent contrastive losses, one for each prefix group. Each group's loss is computed from the same forward pass — the Meta Embeddings for the full (16, 64) budget are extracted once, and then each group's prefix is sliced out to compute its own late-interaction score and loss. The gradients from all five losses flow back through the shared Meta Token parameters and the LoRA-adapted VLM layers, so the model learns to satisfy all granularity constraints simultaneously.

Why equal weighting: the paper follows the original MRL convention of uniform weights. The intuition is that no granularity level is prioritized during training; the model must make each prefix independently discriminative. An unequal weighting (e.g., higher weight on the full budget) would risk the model optimizing for the full representation at the expense of the coarse prefixes, undermining test-time scalability. The equal weighting forces a compromise: the model must allocate representational capacity such that the first vector alone is a strong single-vector retriever, the first two together improve on that, and so on up to the full set.

Why this nested multi-loss design works for test-time scaling: the key insight is that the training objective and the deployment use case are directly aligned. At test time, a user selects a group $g$ and computes $s^{(g)}(q, c)$ using only that group's prefix. Because the model was trained to make that exact prefix discriminative (via $\mathcal{L}^{(g)}_{\text{NCE}}$), the truncated representation performs well. Without MMR — i.e., training only on the full (16, 64) budget — truncating to a smaller prefix at test time causes a dramatic performance drop because the model had no incentive to make early vectors useful on their own. The paper quantifies this in Figure 4b: without MMR, dropping from (16, 64) to (1, 1) causes a 9.0-point NDCG@5 drop on ViDoRe v1. With MMR, the same truncation causes a much smaller drop, and the model at (1, 1) with MMR substantially outperforms the model at (1, 1) without MMR.

A subtle consequence of the nested design: the Meta Embeddings at different positions develop different "roles" through training. The first few vectors tend to capture global, coarse-grained semantics (since they must work in small groups where they are the only vectors available). Later vectors capture finer details that complement the coarse information. This is not explicitly enforced by any loss term — it emerges from the optimization because the early vectors are penalized by all group losses (small and large), while later vectors are penalized only by the larger groups. Early vectors must therefore carry information that is useful across all granularities, which naturally pushes them toward global features.

Test-Time Scaling Mechanism

The nested training directly enables the deployment flexibility shown in Figure 2. At test time, the system operates as follows:

Indexing phase (offline). Each candidate in the retrieval corpus is encoded through a forward pass of the VLM with its candidate-side Meta Tokens appended. The full set of $R_c$ candidate Meta Embeddings $E^{(c)}_{\text{meta}} \in \mathbb{R}^{R_c \times D}$ is produced. The system then stores only the first $r^{(g)}_c$ vectors in the index, where $g$ is chosen based on the available storage budget. For example, indexing at (1, 1) stores one vector per candidate (requiring $N \times 1 \times D$ floats, or 0.68 GiB for 100,000 candidates with $D = 3,584$ at bfloat16 precision, per Table 3). Indexing at (16, 64) stores 64 vectors per candidate (42.72 GiB for the same corpus). The choice of indexing budget $r^{(g)}_c$ is a one-time decision that determines the permanent index size; it can be increased later by re-encoding and storing additional vectors.

Query phase (online). A query is encoded similarly, producing $E^{(q)}_{\text{meta}} \in \mathbb{R}^{R_q \times D}$. The system selects a query budget $r^{(g)}_q$ (which may differ from the index budget $r^{(g)}_c$) and computes the late-interaction score against each candidate using $s^{(g)}(q, c)$ from Equation 4, where $r^{(g)}_q$ query vectors interact with $r^{(g)}_c$ candidate vectors. The candidates are ranked by this score, and the top-$k$ are returned.

The accuracy-efficiency knob. The retrieval budget $(r_q, r_c)$ controls a three-way tradeoff:

  • Accuracy: larger budgets use more vectors, capturing finer-grained alignment information. Figure 3 shows this empirically: MetaEmbed-7B improves from 71.3% (1, 1) to 76.6% (16, 64) on MMEB, a +5.3-point gain.
  • Index storage: the index stores $r_c$ vectors per candidate, scaling as $O(N \times r_c \times D)$. Table 3 quantifies this: index memory ranges from 0.68 GiB at (1, 1) to 42.72 GiB at (16, 64) for 100,000 candidates.
  • Scoring latency: each query-candidate comparison involves $r_q \times r_c$ dot products plus $r_q$ max operations and a sum. Table 3 shows scoring latency remains nearly flat (1.66–1.67 ms) for budgets up to (4, 8), rises slightly to 1.92 ms for (8, 16), and jumps to 6.25 ms for (16, 64), for 100,000 candidates. The latency is dominated by memory bandwidth rather than compute for small budgets, explaining the flat region.

Important asymmetry: the indexing budget $r_c$ and query budget $r_q$ can be chosen independently. The paper's experiments always pair them as a single group (e.g., (4, 8) together) for evaluation, but in principle, one could index at (16, 64) for maximum accuracy and serve queries at (4, 8) for lower latency, or index at (4, 8) to save storage and serve queries at (16, 64) (though the latter would be limited by the coarser index). This flexibility is a direct consequence of the MMR training: each prefix is independently trained, so any combination of valid prefixes is well-formed.

Difference from dimensionality-based MRL. Traditional Matryoshka Representation Learning (Kusupati et al., 2022) operates on the embedding dimension: you train so that the first $d_1$ dimensions of a single vector are useful, the first $d_2 > d_1$ are more useful, etc. MetaEmbed's MMR operates on the number of vectors, not the dimension of each vector. The embedding dimension $D$ is fixed (e.g., 3,584 for the 7B model); what scales is the count of vectors used in the late interaction. This is a more natural axis for multi-vector retrieval because the dominant cost is the $O(r_q \times r_c)$ pairwise comparisons, not the per-vector dimension. Reducing from 64 vectors to 1 vector compresses the index by 64×; reducing the vector dimension from 3,584 to, say, 1,792 would only compress by 2×. The MetaEmbed approach targets the primary bottleneck (vector count) rather than a secondary one (per-vector dimension).

Training Configuration and Design Choices

The paper provides specific training details that are essential for understanding and reproducing MetaEmbed.

Base models and adaptation. MetaEmbed is implemented as a fine-tuning recipe on top of pretrained VLMs. Three backbone architectures are tested to demonstrate generality:

  • Qwen2.5-VL (Bai et al., 2025) at 3B, 7B, and 32B scales — a unified multimodal architecture that processes text and vision through a shared transformer. This is the primary backbone and achieves the strongest results.
  • PaliGemma (Beyer et al., 2024) at 3B scale — another unified architecture, tested as MetaEmbed-3B^Gemma^.
  • Llama-3.2-Vision (Grattafiori et al., 2024) at 11B scale — a cross-attention-based architecture where visual features are integrated into the language model through cross-attention layers rather than being concatenated as tokens. This architectural difference turns out to matter (see below).

All models are fine-tuned using LoRA (Low-Rank Adaptation) (Hu et al., 2022) with rank 32 and scaling factor $\alpha = 32$. LoRA inserts trainable low-rank decomposition matrices $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times k}$ into the weight matrices of the VLM, where $r = 32$ is the rank and $\alpha = 32$ scales the contribution. The pretrained weights remain frozen; only the LoRA matrices and the Meta Token embeddings are updated during training.

Why LoRA instead of full fine-tuning: adapting a 7B or 32B parameter VLM with full fine-tuning would be prohibitively expensive in memory and compute. LoRA reduces the number of trainable parameters dramatically (from billions to tens of millions) while preserving most of the pretrained knowledge. This is particularly important because MetaEmbed's training data (MMEB-train + ViDoRe-train) is relatively small compared to the pretraining corpora — full fine-tuning on such limited data could cause catastrophic forgetting of the VLM's multimodal understanding. LoRA acts as a regularizer, keeping the model close to its pretrained state while adapting the representations for the embedding task. The rank of 32 was chosen empirically; the paper does not ablate over rank values.

Training data. The paper uses only two datasets, keeping the recipe deliberately simple:

  • MMEB-train (Jiang et al., 2025): the training split of the Massive Multimodal Embedding Benchmark, covering diverse tasks across classification, VQA, retrieval, and visual grounding.
  • ViDoRe-train (Faysse et al., 2025): the training split for visual document retrieval.

For each query, the paper includes one explicit hard negative from Chen et al. (2025a). Hard negatives are candidate items that are superficially similar to the positive candidate (e.g., visually similar images, documents on related topics) but not actually correct for the query. Including them forces the model to learn fine-grained distinctions rather than relying on coarse modality-level or topic-level signals that would accidentally separate in-batch negatives.

Why this minimal data recipe: the paper positions MetaEmbed as a training recipe that should work without massive proprietary datasets. Many strong baselines (mmE5, MegaPairs, MoCa) use extensive synthetic data generation pipelines or additional pretraining stages. MetaEmbed shows that strong results can be achieved with publicly available training data and straightforward contrastive fine-tuning, provided the architecture (Meta Tokens + MMR) is well-designed. This also means that the quality of the underlying VLM backbone is critical — MetaEmbed cannot compensate for a weak backbone through data scale, as evidenced by the MetaEmbed-11B (Llama) results.

Optimization hyperparameters. Training details vary by model size (Table 4):

  • MetaEmbed-3B (Qwen2.5-VL): global batch size 2,048, learning rate $1 \times 10^{-4}$, trained for 23 hours on 32 H100 SXM5 96GB GPUs, embedding dimension $D = 2,048$.
  • MetaEmbed-7B (Qwen2.5-VL): global batch size 1,536, learning rate $1 \times 10^{-4}$, trained for 30 hours on 32 H100 GPUs, embedding dimension $D = 3,584$, trained for 3,500 steps.
  • MetaEmbed-32B (Qwen2.5-VL): global batch size 1,536, learning rate $1 \times 10^{-5}$ (note: 10× lower than the smaller models), trained for 25 hours on 64 H100 GPUs, embedding dimension $D = 5,120$.
  • MetaEmbed-11B (Llama-3.2-Vision): global batch size 1,024, learning rate $1 \times 10^{-4}$, trained for 10 hours on 32 H100 GPUs, embedding dimension $D = 4,096$.
  • MetaEmbed-3B^Gemma^ (PaliGemma): global batch size 2,048, learning rate $1 \times 10^{-4}$, trained for 14 hours on 32 H100 GPUs, embedding dimension $D = 2,048$.

Temperature: the contrastive temperature $\tau$ is set to 0.03 for all variants. This is a relatively low temperature, meaning the softmax over similarities is sharp — the model is penalized heavily for assigning even moderate probability to negatives. Low temperatures are common in contrastive retrieval training because they encourage tight clustering of positives and clean separation from negatives.

Validation and early stopping: 1% of training data from each subset of MMEB-train is held out as an evaluation split. Training is early-stopped when evaluation loss stops decreasing.

Group sizes: the chosen $G = 5$ groups $\{(1, 1), (2, 4), (4, 8), (8, 16), (16, 64)\}$ represent an exponential scaling pattern in both query and candidate vector counts. The candidate side grows faster (doubling from 1 to 4 to 8 to 16 to 64) than the query side, with the final jump from 16 to 64 candidate vectors being the largest. This asymmetry reflects the typical retrieval scenario where candidates contain more information than queries (an image or document has more content than a short text query), so more candidate-side vectors are needed to capture all potentially relevant details. The paper does not ablate over group size choices but states the chosen values are empirical.

Why these specific group sizes: the exponential spacing (powers of 2) provides a wide range of operating points from single-vector to full multi-vector, with each step roughly doubling the computational cost. The final group (16, 64) is the full budget used for reporting main results. The query side maxes out at 16 rather than 32 or 64 because queries are typically shorter or less complex than candidates, and 16 vectors are sufficient to capture query intent while keeping scoring cost manageable ($16 \times 64 = 1,024$ comparisons per candidate, versus $64 \times 64 = 4,096$ if query also had 64 vectors).

Implementation infrastructure. Training uses PyTorch 2.6.0 with CUDA 12.4, FlashAttention 2.0 (Dao, 2024) for efficient attention computation, and gradient checkpointing (Chen et al., 2016) to reduce GPU memory usage (trading compute for memory by recomputing activations during the backward pass rather than storing them). For the 3B models, standard Distributed Data Parallel (DDP) (Li et al., 2020) is used. For 7B and larger models, Fully Sharded Data Parallel (FSDP) v2 (Zhao et al., 2023) is used, which shards model parameters, gradients, and optimizer states across GPUs to fit large models in memory. A notable implementation detail: to prevent distributed hanging when training samples contain no images under FSDP, the paper pads those samples with a placeholder image to ensure the visual encoder is activated on every GPU. This is a pragmatic fix for a distributed training edge case.

Why these infrastructure choices: the combination of FlashAttention, gradient checkpointing, and FSDP is standard for training large transformer models on multi-GPU setups. FlashAttention reduces the $O(n^2)$ memory cost of attention to $O(n)$ by computing attention in tiles without materializing the full attention matrix. Gradient checkpointing reduces activation memory by ~70% at the cost of ~20% more compute. FSDP enables training models that are too large to fit on a single GPU by distributing the model state across devices, synchronizing only when needed. Together, these techniques are what make training a 32B-parameter VLM feasible on 64 GPUs.

Design Choices: Why This Approach Over Alternatives

The paper makes several deliberate design decisions that distinguish MetaEmbed from other possible approaches, each with a specific rationale:

1. Learned Meta Tokens vs. using pretrained special tokens (e.g., [CLS]). Many VLMs already have special tokens that are designed to aggregate information (e.g., a vision-language connector token or an end-of-sequence token used for pooling). The paper could have used such existing tokens rather than introducing new learnable Meta Tokens. The decision to use separate learned tokens provides two advantages: (a) the number of vectors is fully configurable — you can have 16 or 64 Meta Tokens regardless of how many special tokens the pretrained VLM has; (b) the Meta Tokens are trained from scratch for the specific purpose of multi-vector retrieval, whereas pretrained special tokens were optimized for the VLM's original pretraining objective (typically next-token prediction or image-text matching), which may not align with producing complementary multi-vector representations.

2. MMR on vector count vs. MRL on embedding dimension. The paper could have applied traditional Matryoshka embedding on the per-vector dimension $D$ (so that truncating dimensions reduces storage and compute) while using a fixed number of vectors. This would have been more similar to existing work. The choice to apply the nesting to the vector count instead targets the dominant scaling factor: reducing from 64 vectors to 1 vector compresses the index by 64×, whereas reducing the dimension from $D$ to $D/2$ compresses by only 2×. For multi-vector retrieval, the number of vectors is the primary cost driver, so nesting on that axis provides more impactful test-time scaling. The two approaches could potentially be combined (nested both in vector count and per-vector dimension), but MetaEmbed focuses on vector count.

3. Separate forward passes for query and candidate vs. joint encoding. ColBERT-style methods encode queries and candidates independently (two forward passes) and then compute late interaction. An alternative would be to encode the query and candidate jointly in a single forward pass (cross-encoding), which allows richer interaction but requires reprocessing the candidate for every query — impossible to pre-index. MetaEmbed preserves the independent encoding property, which is essential for scalable retrieval: candidates are encoded once offline (at indexing time), and only the query is encoded online. The late interaction provides cross-modal alignment without requiring joint processing at query time.

4. LoRA fine-tuning vs. full model training from scratch. Training a VLM-sized retrieval model from scratch with contrastive objectives (as CLIP did) would require enormous datasets (400M+ pairs) and compute budgets. MetaEmbed's approach leverages the VLM's pretrained multimodal understanding — the model already knows how to relate images and text — and only needs to learn how to produce good retrieval representations from that understanding. LoRA provides a parameter-efficient way to adapt the model while retaining its pretrained knowledge. This is particularly important because the training datasets (MMEB-train + ViDoRe-train) are orders of magnitude smaller than VLM pretraining corpora.

5. Single hard negative per query vs. multiple hard negatives or no hard negatives. The paper uses exactly one explicit hard negative per query, in addition to the $B - 1$ in-batch negatives. This is a minimal approach — some methods use multiple hard negatives or mine negatives dynamically during training. The choice of one hard negative balances training cost (each hard negative requires an additional forward pass for scoring within the loss) against the benefit of learning to distinguish subtle differences. The paper inherits the hard negatives from Chen et al. (2025a) rather than mining them independently, suggesting that the quality of hard negatives matters more than the quantity.

4. Key Insights and Innovations

Innovation 1: Recasting the Multi-Vector Retrieval Problem as a Learned Compression Task

The dominant assumption in multi-vector retrieval since ColBERT has been that the vectors used for late interaction should be derived directly from the input tokens — image patches, text subwords — because those tokens faithfully represent the raw information in the input. MetaEmbed challenges this assumption at its foundation by proposing that the vectors should instead be learned aggregators that synthesize information from the full input during the forward pass and produce a compact set of embeddings whose cardinality is a design choice, not a function of input length.

This is a conceptual reframing, not just an architectural tweak. Prior work treats multi-vector retrieval as an information preservation problem: we have hundreds of patch embeddings that contain fine-grained visual information, and we want to keep as much of that information as possible when comparing with the query. The problem is then one of efficiency — how to compress or approximate the full token-level interaction (e.g., Plaid for ColBERT, Santhanam et al., 2022a; Muvera, Jayaram et al., 2024). MetaEmbed treats it instead as an information extraction problem: the VLM sees the full input, processes it through its transformer layers, and a small set of learned Meta Tokens selectively attend to whatever information is most relevant for determining semantic similarity. The tokens are not compressing the patch embeddings; they are constructing their own representations from the transformer's intermediate features.

The significance of this shift is that it decouples representational capacity from input complexity. An image with 1,024 patches and one with 256 patches both produce exactly 64 candidate-side Meta Embeddings — the representation cost is fixed regardless of how much visual information the VLM extracts. This makes the multi-vector approach practical for the first time in regimes where it was previously impossible: multimodal-to-multimodal retrieval, where both query and candidate sides contain images. Under the input-derived paradigm, a query with 1,024 patches matched against a candidate with 1,024 patches requires ~10^6 pairwise comparisons per candidate. Under MetaEmbed, it requires at most 16 × 64 = 1,024 comparisons, independent of input resolution. The paper does not just make multi-vector retrieval more efficient — it makes an entire retrieval modality (image-to-image via late interaction) feasible where it was not before.

The "split" baseline in Appendix B (Table 5) validates that this is not a trivial benefit of simply having multiple vectors. When you take the VLM's last-layer hidden states and average-pool them into 16 and 64 segments (the "split-(16, 64)" baseline), you get a multi-vector representation without additional parameters. But MetaEmbed-7B outperforms this baseline by +6.5 points on MMEB (76.6 vs. 70.1), demonstrating that the learned aggregation of Meta Tokens extracts substantially more discriminative information than static pooling of the same VLM features. The Meta Tokens are not just a compression mechanism — they learn to produce better representations than the raw token features they replace.

This reframing is fundamental rather than incremental because it changes what problem a multi-vector retrieval researcher thinks they are solving. Before MetaEmbed, the question was: "How can we make full token-level late interaction faster?" After MetaEmbed, the question becomes: "How many learned vectors do we need, and how should we structure them, to capture the semantic information relevant for retrieval?" The efficiency bottleneck shifts from scoring cost (which is now bounded by a small constant) to encoding quality (how well the Meta Tokens extract information from the VLM), which is a very different research direction.

Innovation 2: Test-Time Scaling as a First-Class Design Goal via Nested Multi-Vector Granularity

Prior work on flexible retrieval representations — specifically Matryoshka Representation Learning (MRL, Kusupati et al., 2022) and its adoption in text embedding models — achieved test-time scalability by nesting along the embedding dimension: the first d dimensions of a single vector form a coarse representation, and additional dimensions add detail. MetaEmbed pioneers a different axis: nesting along the number of vectors used in late interaction. This is not a straightforward analog — it requires solving a different structural problem because vectors in a multi-vector set do not have a natural ordering the way dimensions within a vector do.

The intellectual contribution is the recognition that prefix nesting of multi-vector sets can create the same kind of graceful degradation that dimension nesting provides for single vectors, if the training objective enforces that the first k vectors are independently discriminative at each k. The Matryoshka Multi-Vector Retrieval (MMR) loss — computing separate contrastive objectives for (1,1), (2,4), (4,8), (8,16), and (16,64) prefixes simultaneously — is the mechanism that enforces this structure. But the key insight is not the loss function itself (which adapts the standard MRL weighted-sum approach); it is the observation that multi-vector retrieval has a natural coarse-to-fine axis that is orthogonal to, and more impactful than, per-vector dimensionality.

To appreciate why this is a significant conceptual move: a ColBERT-style multi-vector representation with 1,024 patch vectors cannot be "truncated" to 64 vectors in any principled way because the model was never trained to make any particular 64-vector subset discriminative on its own. The representation is monolithic — all vectors must be present for the similarity computation to be meaningful. MetaEmbed's nested training creates a representation that is compositional along the vector-count axis: each prefix is a valid, trained representation in its own right, and larger prefixes add complementary detail rather than being necessary for coherence.

This enables a deployment model that was previously impossible: one trained model serves every point on the accuracy-efficiency Pareto frontier, from single-vector (1,1) to full multi-vector (16,64). Figure 4b provides the critical evidence that MMR is what enables this: without MMR training, using (1,1) at test time causes a 9.0-point NDCG@5 drop on ViDoRe v1 compared to the full budget. With MMR, the drop is much smaller. This means the flexibility is not an emergent property of any multi-vector model — it must be explicitly trained for, and MMR is the training framework that achieves it.

The significance beyond raw performance is that this introduces a new dimension of deployment-time adaptability into retrieval system design. A single MetaEmbed model can serve real-time queries with tight latency budgets at (2,4) while simultaneously powering an offline batch reranking pipeline at (16,64), without maintaining separate models or indexes. The index can be built once at (16,64) and queried at any smaller budget; or the index can be built at (4,8) to save storage while queries are served at their original budget. This operational flexibility has no analog in prior single-vector or multi-vector retrieval systems, where the accuracy-efficiency operating point is fixed at training time.

The comparison with MRL-on-dimension highlights why vector-count nesting is the right axis for multi-vector retrieval. Reducing from 64 vectors to 1 vector compresses the index by 64×. Reducing the per-vector dimension from 3,584 to, say, 896 compresses by only 4×. The dominant scaling factor for multi-vector methods is the number of vectors (which drives both index storage and the O(r_q × r_c) scoring cost), so nesting on that axis provides the most impactful test-time scaling knob. MetaEmbed could have applied traditional MRL on top of its vector-count nesting (reducing dimensions as well), but it focuses on the primary bottleneck first.

This is a fundamental contribution rather than incremental because it generalizes the Matryoshka concept to a new structural domain (sets of vectors rather than dimensions of a single vector) and shows it is both feasible and highly effective. It opens the door to other compositional representation structures where different axes of granularity can be nested and independently controlled.

Innovation 3: Empirical Evidence That Multi-Vector Retrieval Scales Better with Model Size Than Single-Vector Retrieval

One of the paper's most consequential empirical findings is that the benefit of multi-vector representations compounds as the underlying VLM grows larger, while single-vector methods saturate. Figure 4a and the associated discussion in Section 4.3 quantify this: MetaEmbed's gain over single-vector retrieval is +2.1 points at 3B, +6.0 points at 7B, and +7.5 points at 32B on MMEB. Moreover, "the improvement of the single-vector baseline from 7B to 32B is no longer statistically significant while MetaEmbed still holds a noticeable gain" (Section 4.3).

This is a diagnostic finding with significant implications for how the field should think about scaling retrieval models. The conventional wisdom from single-vector methods like CLIP and its successors has been that scaling model size improves retrieval quality — larger vision encoders, larger text encoders, larger batch sizes. But MetaEmbed's results suggest that single-vector representations have an information ceiling: beyond a certain model capacity, the bottleneck is not the model's ability to understand the input, but the representational limitation of compressing everything into one vector. The 32B single-vector model understands the inputs better than the 7B model, but it cannot express that improved understanding in a single 5,120-dimensional vector any more effectively — the additional capacity is wasted.

Multi-vector representations break through this ceiling because they provide more output bandwidth: the model can distribute its improved understanding across multiple complementary vectors, each capturing different aspects of the input's semantics. The Meta Tokens at a 32B scale can learn more sophisticated information decomposition — perhaps one vector captures object-level semantics, another captures spatial relationships, a third captures textual details — because the underlying VLM provides richer features to attend to. The single-vector model is forced to average all of these signals into one representation, losing the decomposition.

This finding positions multi-vector retrieval not just as a "more expressive alternative to single-vector" (which was already known from ColBERT-style work) but as the necessary representational form for continued scaling of retrieval models. If single-vector performance saturates at 7B–32B parameters, then pushing retrieval accuracy further — to 100B+ parameter models — will require multi-vector representations that can absorb the additional capacity. MetaEmbed's recipe (learned Meta Tokens + MMR) provides a concrete path for doing so without the index explosion of full token-level methods.

The evidence for this claim comes from the trend across model sizes in Table 5 and Figure 3. The gain from (1,1) to (16,64) grows monotonically with model size: +3.3 points at 3B, +5.3 points at 7B, +6.6 points at 32B on MMEB. This is not guaranteed — it could have been that larger models produce better single vectors and the multi-vector gain stays constant or shrinks. The fact that the gain grows suggests there is a genuine interaction between model capacity and representational bandwidth: larger models produce richer internal features, and more Meta Embedding vectors are needed to externalize those features into a retrieval-compatible format.

This is a fundamental empirical insight that changes the scaling roadmap for multimodal retrieval. It is not an incremental improvement over existing methods; it redefines what "scaling up" means for retrieval by showing that output dimensionality (in the form of vector count) must scale jointly with model parameters to realize the full benefit of larger backbones. This has direct implications for model architecture design and resource allocation in both research and production.

Innovation 4: Distinguishing VLM Architectural Quality from Retrieval Recipe Effectiveness

MetaEmbed's multi-backbone evaluation — testing on Qwen2.5-VL, PaliGemma, and Llama-3.2-Vision at different scales — produces a clean diagnostic result that the paper does not overstate but which is methodologically important: the MetaEmbed recipe faithfully preserves the strengths and weaknesses of its underlying VLM, and cannot compensate for fundamental cross-modal deficiencies in the backbone. MetaEmbed-11B (Llama-3.2-Vision) achieves only 65.1 on MMEB, with its VQA score dropping to 42.1 — more than 32 points below the 7B Qwen2.5-VL variant (74.2 VQA), despite being a larger model.

This might seem like a negative result ("some VLMs work better than others") but it is actually a diagnostic contribution that clarifies the relationship between pretraining and fine-tuning for retrieval. Prior work on VLM-based embedders often presents the retrieval recipe as the primary source of performance, with the backbone treated as interchangeable. MetaEmbed's results demonstrate that the backbone's cross-modal understanding — particularly in tasks like visual question answering that require deep integration of visual and linguistic semantics — is the dominant factor, and the retrieval fine-tuning amplifies rather than corrects backbone deficiencies.

The conceptual significance is that it separates the problem of "good retrieval" into two independently solvable subproblems: (1) the backbone VLM must have strong, balanced multimodal understanding (which is a pretraining problem), and (2) the retrieval recipe must effectively extract that understanding into a compact, searchable representation (which is MetaEmbed's contribution). The field can now pursue these independently — improve VLMs for better multimodal understanding, apply MetaEmbed to convert that understanding into embeddings — rather than conflating them.

This is a methodological contribution rather than a technical one: it provides a framework for evaluating future retrieval methods by explicitly varying the backbone and measuring whether the recipe amplifies or compensates for backbone quality. If a new recipe claims to improve retrieval, does it improve Qwen2.5-VL-based and Llama-based variants equally? If it improves the Llama variant more, it may be fixing a specific weakness of that architecture rather than generally improving retrieval. MetaEmbed's results — where the relative ordering of backbones is preserved — suggest it is a "clean" recipe that adds value orthogonally to backbone quality.

The evidence is in the contrast between Table 1's MetaEmbed rows: Qwen2.5-VL-based variants achieve balanced scores across Classification, VQA, Retrieval, and Grounding; the Llama-3.2-Vision variant (MetaEmbed-11B) achieves strong Grounding (91.6) and Retrieval (74.3) but collapses on VQA (42.1). This pattern mirrors the known properties of the underlying VLMs — Huang et al. (2025) note Llama-3.2-Vision-11B's weakness in zero-shot VQA — suggesting MetaEmbed is not introducing new failure modes or biases. This transparency is valuable for practitioners: choosing a backbone for MetaEmbed means choosing which tasks to prioritize, and the paper provides the data to make that decision.

This is an incremental contribution in the sense that it does not propose a new technique, but it is a foundational empirical finding that should influence how the community designs and evaluates retrieval systems. Future papers that propose new retrieval recipes should follow MetaEmbed's lead in testing across multiple backbones and reporting per-task breakdowns, to distinguish recipe quality from backbone quality. The failure to do this in much prior work may explain why some methods appear strong on specific benchmarks but fail to generalize — they were implicitly benchmarked on backbones that happened to align well with the evaluation tasks.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on two established benchmarks. The Massive Multimodal Embedding Benchmark (MMEB) (Jiang et al., 2024) consists of 36 tasks across four categories — Classification, Visual Question Answering (VQA), Retrieval, and Visual Grounding — with tasks including ScienceQA (Lu et al., 2022), VizWiz (Gurari et al., 2018), ChartQA (Masry et al., 2022), Visual News (Liu et al., 2021), FashionIQ (Wu et al., 2021), OvenWiki (Hu et al., 2023), COCO (Lin et al., 2014), and RefCOCO (Kazemzadeh et al., 2014; Yu et al., 2016). The Visual Document Retrieval Benchmarks (ViDoRe) v2 (Macé et al., 2025) covers 7 visual document retrieval tasks spanning human-annotated, synthetic, multilingual, and biomedical domains; it was introduced to mitigate performance saturation in the original ViDoRe (Faysse et al., 2025) by including more generalized settings and multilingual subsets. Training uses MMEB-train (Jiang et al., 2025) and ViDoRe-train (Faysse et al., 2025), with one explicit hard negative per query from Chen et al. (2025a). Evaluation on MMEB uses its standard test split; ViDoRe v2 is evaluated on its full benchmark.

  • Base model(s). Three VLM families are tested to demonstrate architectural generality. Qwen2.5-VL (Bai et al., 2025) at 3B, 7B, and 32B scales serves as the primary backbone; it uses a unified multimodal architecture where visual patches and text tokens are processed jointly through a shared transformer. PaliGemma (Beyer et al., 2024) at 3B is another unified architecture, tested as MetaEmbed-3B^Gemma^. Llama-3.2-Vision (Grattafiori et al., 2024) at 11B represents a cross-attention-based design where visual features are integrated through cross-attention layers rather than token concatenation. The Qwen2.5-VL family is the primary focus because it achieves balanced performance across all MMEB task categories, enabling clean measurement of MetaEmbed's contribution without backbone-induced task biases.

  • Metrics. For MMEB, the primary metric is Precision@1 (%) — the fraction of queries for which the top-ranked candidate is correct — averaged across all 36 tasks, with per-category breakdowns (Classification, VQA, Retrieval, Grounding) and aggregated metrics (IND for in-domain average, OOD for out-of-domain average, Overall for the full benchmark average). For ViDoRe v2, the metric is average NDCG@5 (%) — Normalized Discounted Cumulative Gain at rank 5, averaged across all 7 evaluation tracks (ESG_Human, Eco_Mul, Bio, ESG_Syn, ESG_Syn_Mul, Bio_Mul, Eco), which measures ranking quality with higher weight on top positions. All MetaEmbed results in main tables are reported at the full retrieval budget of (r_q, r_c) = (16, 64) unless otherwise specified.

  • Baselines. The paper compares against a comprehensive set of methods. Single-vector baselines: CLIP (Radford et al., 2021, 428M), MagicLens (Zhang et al., 2024a, 613M), UniIR (Wei et al., 2024, 428M), ABC (7B), MM-EMBED (Lin et al., 2024, 7B), GME (Zhang et al., 2024b, 7B), VLM2Vec (Jiang et al., 2025, 7B), VLM2Vec-V2 (Meng et al., 2025, 2B), MMRet (Zhou et al., 2024, 7B), mmE5 (Chen et al., 2025b, 11B), MoCa-3B and MoCa-7B (Chen et al., 2025a), B3-7B (Thirukovalluru et al., 2025), SigLIP (Zhai et al., 2023, 652M), and VisRAG-Ret (3B). Multi-vector baselines (ViDoRe v2 only): ColPali (Faysse et al., 2025, 3B) and ColQwen2 (2B). All baseline metrics are taken directly from the corresponding original papers or from Chen et al. (2025a). Appendix C provides short introductions to each baseline method.

  • Generation budget / compute accounting. The concept of a "generation budget" does not apply here as it would for LLM sampling; instead, compute is measured through the retrieval budget $(r_q, r_c)$ — the number of Meta Embedding vectors used on the query side ($r_q$) and candidate side ($r_c$). The five groups trained with MMR are {(1,1), (2,4), (4,8), (8,16), (16,64)}. At test time, selecting a smaller group reduces index storage (proportional to $r_c$ vectors per candidate), scoring FLOPs (proportional to $r_q \times r_c$ dot products per query-candidate pair), and scoring latency. Efficiency is measured in Table 3 via scoring FLOPs (in GFLOPs), scoring latency (ms per 100,000 candidates on an A100 GPU with bfloat16 precision and scoring batch size 1,000, reported as mean ± std over 10 runs), and index memory (GiB for 100,000 candidates). Query encoding latency (which dominates at 788 ms for a 1024-token image query) and index generation latency are reported as context but are identical across retrieval budgets.

  • Cross-validation / statistical protocol. The paper uses 1% of training data from each subset of MMEB-train as a held-out evaluation split for early stopping; training stops when evaluation loss ceases to decrease (Section 4.1, Appendix A). No explicit cross-validation for hyperparameter selection is described — the group sizes {(1,1), (2,4), (4,8), (8,16), (16,64)}, temperature τ = 0.03, LoRA rank 32 with α = 32, and group weights w_g = 1 are fixed a priori based on prior work (Kusupati et al., 2022). The paper does not report confidence intervals or statistical significance tests for the main benchmark results beyond the latency standard deviations in Table 3, though the claim about single-vector performance saturation from 7B to 32B is explicitly qualified as "no longer statistically significant" (Section 4.3). Test set sizes are fixed by the benchmarks: 36 tasks in MMEB (the number of queries per task is not specified in the paper but is determined by each task's standard test split) and 7 tracks in ViDoRe v2.

Main Quantitative Results

Overall Multimodal Retrieval Performance on MMEB (Table 1)

The headline result is that MetaEmbed-7B achieves 76.6% overall Precision@1 on MMEB, outperforming the strongest single-vector baseline (B3-7B at 72.0%) by 4.6 absolute points and the strongest comparable-scale multi-vector method (ColPali at 54.5% on ViDoRe, but ColPali does not report MMEB scores). MetaEmbed-32B pushes the state of the art to 78.7%, exceeding all baselines at any scale.

Breaking down by model scale in Table 1:

  • At 3B: MetaEmbed-3B (Qwen2.5-VL) achieves 69.1% overall, surpassing MoCa-3B (67.5%) by +1.6 points and VLM2Vec-V2 (2B, 64.9%) by +4.2 points. MetaEmbed-3B^Gemma^ (PaliGemma) achieves 65.4%, demonstrating the recipe works across architectures but with lower absolute performance due to backbone quality differences.

  • At 7B: MetaEmbed-7B achieves 76.6% overall, improving over MoCa-7B (71.5%) by +5.1 points, mmE5 (11B, 69.8%) by +6.8 points, and B3-7B (72.0%) by +4.6 points. The gain is not uniform across task categories: MetaEmbed-7B leads most strongly in VQA (74.2% vs. 66.5% for B3-7B, a +7.7-point gap) and Retrieval (78.7% vs. 74.1%), while Grounding (85.4%) trails MoCa-7B (92.4%) by 7.0 points — the only category where a baseline outperforms MetaEmbed at comparable scale.

  • At 32B: MetaEmbed-32B achieves 78.7% overall, with particularly strong VQA (78.6%) and Retrieval (78.9%). The Classification score (73.7%) shows the smallest gain over 7B (+2.4 points), while VQA gains +4.4 points from 7B and Grounding gains +2.7 points.

  • At 11B (Llama-3.2-Vision): MetaEmbed-11B achieves 65.1% overall but with extreme task imbalance: Grounding reaches 91.6% (competitive with the best baselines), Classification is 66.4%, but VQA collapses to 42.1% — more than 32 points below MetaEmbed-7B (Qwen2.5-VL) on the same task. This stark difference isolates backbone quality as the dominant factor for cross-modal reasoning tasks.

The per-category breakdown reveals where MetaEmbed's multi-vector representation provides the most benefit. On Retrieval tasks (where fine-grained cross-modal alignment matters most), MetaEmbed-7B (78.7%) leads all baselines including the 11B mmE5 (71.0%) by +7.7 points. On VQA, MetaEmbed-7B (74.2%) leads B3-7B (66.5%) by +7.7 points. On Classification — which often relies more on global image-text alignment that single vectors can capture — MetaEmbed-7B (71.3%) trails B3-7B (70.0%) by only +1.3 points, suggesting the multi-vector advantage is smaller when coarse semantic matching suffices.

Visual Document Retrieval on ViDoRe v2 (Table 2)

On visual document retrieval, MetaEmbed-7B achieves 61.3% average NDCG@5, and MetaEmbed-3B achieves 60.3%. When compared against single-vector and multi-vector baselines:

  • Against single-vector methods: MetaEmbed-7B (61.3%) outperforms the strongest single-vector baseline GME (7B, 59.3%) by +2.0 points and MoCa-3B (59.8%) by +1.5 points. MetaEmbed-3B (60.3%) already surpasses GME-7B and is competitive with MoCa-7B (58.8%). The gain is particularly large on multilingual and biomedical domains: on Bio_Mul, MetaEmbed-7B achieves 61.9% vs. GME's 55.1% (+6.8 points); on Eco_Mul, MetaEmbed-7B achieves 54.2% vs. GME's 56.2% (−2.0 points, the only multilingual track where MetaEmbed trails).

  • Against multi-vector methods: MetaEmbed-3B (60.3%) outperforms ColPali (3B, 54.5%) by +5.8 points and ColQwen2 (2B, 57.5%) by +2.8 points. MetaEmbed-7B (61.3%) extends this lead to +6.8 points over ColPali. The improvement is consistent across all 7 tracks: MetaEmbed-7B leads ColPali on every track, with the largest gaps on ESG_Human (+11.8 points) and Eco (+9.3 points). The consistent outperformance is notable because ColPali uses full patch-level token interaction (hundreds of vectors per image), while MetaEmbed uses only 64 candidate-side vectors — yet MetaEmbed achieves higher accuracy. This suggests that learned Meta Token aggregation extracts more discriminative information than raw patch embeddings for document retrieval, where semantic content (text, layout, structure) matters more than pixel-level detail.

  • A notable finding: MetaEmbed performs strongly on multilingual and biomedical domains despite not being trained on multilingual data (Section 4.2): "This is especially noteworthy given that no explicit multilingual data was included during training, suggesting that MetaEmbed effectively retains and leverages cross-lingual capabilities from its backbone." On ESG_Syn_Mul, MetaEmbed-7B achieves 61.1% vs. 56.7% for GME and 55.7% for ColPali. On Bio, MetaEmbed-7B achieves 65.0% vs. 64.0% for GME and 59.7% for ColPali.

Test-Time Scaling Behavior: Accuracy vs. Retrieval Budget (Figure 3)

Figure 3 presents the core test-time scaling curves for all four MetaEmbed variants (3B, 7B, 32B Qwen2.5-VL; 11B Llama-3.2-Vision), showing MMEB Precision@1 as a function of retrieval budget from (1,1) to (16,64). The dashed green lines mark the best single-vector retrieval performance using identical training settings.

  • MetaEmbed-7B (Figure 3b): At (1,1) — equivalent to single-vector retrieval — the model achieves 71.3%. This increases to 72.0% at (2,4), 72.9% at (4,8), 74.3% at (8,16), and 76.6% at (16,64). The total gain from (1,1) to (16,64) is +5.3 percentage points. The single-vector baseline (dashed green line) is at 71.6%, meaning MetaEmbed at (2,4) already surpasses single-vector performance, and the full (16,64) budget delivers a +5.0-point advantage over the single-vector model. Importantly, the curve shows diminishing returns: +0.7 points from (1,1) to (2,4), +0.9 from (2,4) to (4,8), +1.4 from (4,8) to (8,16), and +2.3 from (8,16) to (16,64). The largest absolute jump occurs at the final, most expensive step — suggesting that the full 64 candidate vectors provide qualitatively different information than the earlier, coarser prefixes.

  • MetaEmbed-32B (Figure 3c): The scaling curve shows the largest total gain: from 72.4% at (1,1) to 78.7% at (16,64), a +6.6-point improvement. The single-vector baseline is at 72.1%, meaning the (1,1) configuration already matches single-vector performance, and the full multi-vector budget provides a substantial +6.6-point advantage. The curve is notably steeper than for the 7B model: +0.8 points from (1,1) to (2,4), +1.0 from (2,4) to (4,8), +1.6 from (4,8) to (8,16), and +2.9 from (8,16) to (16,64). The gain at every step is larger than the corresponding step for the 7B model, demonstrating that multi-vector representations extract proportionally more benefit from the larger backbone's richer internal features.

  • MetaEmbed-3B (Figure 3a): The scaling gain is more modest: from 65.4% at (1,1) to 69.1% at (16,64), a +3.3-point improvement. The single-vector baseline is at 65.6%, so the (1,1) configuration slightly underperforms single-vector (−0.2 points), and the full budget provides a +3.5-point advantage. The smaller gain suggests that the 3B backbone's representational capacity limits how much additional discriminative information the multi-vector representation can extract — the model is saturated at a lower level of expressiveness.

  • MetaEmbed-11B (Llama-3.2-Vision, Figure 3d): Despite the lower absolute performance (65.1% at full budget), the scaling pattern holds: from 60.0% at (1,1) to 65.1% at (16,64), a +5.0-point gain. The curve shape (steep initial gains, continuing improvement at larger budgets) mirrors the Qwen2.5-VL variants, suggesting that the MetaEmbed recipe reliably produces scalable multi-vector representations even when the backbone has task-specific weaknesses.

The red arrows in each subplot quantify the absolute gain between MetaEmbed at (16,64) and the best single-vector retrieval method: +3.3 points at 3B, +5.0 points at 7B, +6.0 points at 32B (note: Figure 3c shows +6.6 from (1,1) to (16,64) while the red arrow shows +6.0 vs. single-vector baseline, suggesting the single-vector baseline at 32B is slightly higher than the (1,1) point). The monotonic growth of this gain with model size (Figure 4a quantifies it as +2.1, +6.0, +7.5 for 3B, 7B, 32B respectively) is one of the paper's key empirical findings.

Efficiency Analysis (Table 3)

Table 3 provides the quantitative efficiency characterization for MetaEmbed-7B across all five retrieval budgets, measuring scoring on 100,000 candidates per query with an A100 GPU:

BudgetScoring FLOPs (G)Latency (ms)Index Memory (GiB)MMEB Acc (%)
(1, 1)0.711.67 ± 0.130.6871.3
(2, 4)5.731.66 ± 0.122.6772.0
(4, 8)22.941.67 ± 0.125.3472.9
(8, 16)91.751.92 ± 0.1210.6874.3
(16, 64)733.896.25 ± 0.0742.7276.6

Key observations from this data:

  • Scoring latency is not FLOPs-bound for moderate budgets. From (1,1) to (4,8), measured latency is essentially flat (1.66–1.67 ms) despite FLOPs increasing 32× (from 0.71 to 22.94 GFLOPs). This indicates the scoring kernel is memory-bandwidth-bound at these sizes — the GPU's compute units are underutilized because data movement dominates. Only at (16,64) does latency jump to 6.25 ms, a 3.7× increase, while FLOPs increase by 1,033× relative to (1,1). The absolute latency remains low: even at the most expensive budget, scoring 100,000 candidates takes 6.25 ms, which is negligible in the context of a retrieval pipeline where encoding a single 1024-token image query takes 788 ms.

  • Index memory scales linearly with r_c. The index size grows from 0.68 GiB at r_c = 1 to 42.72 GiB at r_c = 64, a 63× increase (slightly less than the expected 64× due to metadata overhead). For production deployments with millions of candidates, this linear scaling becomes the primary constraint — indexing 1 million candidates at (16,64) would require approximately 427 GiB, exceeding typical GPU memory. The paper notes this challenge and suggests mitigation through balanced retrieval budgets (e.g., indexing at (4,8) would require only 53.4 GiB for 1M candidates) or CPU offloading.

  • Query encoding dominates total latency. The paper reports that encoding an image query of 1,024 tokens requires 42.72 TFLOPs and 788 ms — orders of magnitude more than the scoring costs in Table 3. This means efficiency optimization efforts should target the VLM encoder rather than the scoring kernel for most practical deployments. The MetaEmbed approach of fixing the number of candidate-side vectors (rather than scaling with input tokens) ensures that scoring cost remains a small fraction of total pipeline latency.

Scaling Model Size: Multi-Vector vs. Single-Vector (Figure 4a)

Figure 4a directly compares MetaEmbed at (16,64) against the single-vector retrieval method across the 3B, 7B, and 32B Qwen2.5-VL backbones, with both trained under identical settings. The key finding:

  • Single-vector performance saturates: from 3B to 7B, single-vector improves from 65.6 to 71.6 (+6.0 points). From 7B to 32B, it improves only to 72.1 (+0.5 points), which the paper states is "no longer statistically significant" (Section 4.3).

  • MetaEmbed continues to improve: from 3B to 7B, MetaEmbed improves from 69.1 to 76.6 (+7.5 points). From 7B to 32B, it improves further to 78.7 (+2.1 points). While the gain from 7B to 32B is smaller than from 3B to 7B (diminishing returns are expected), it is still substantial and — critically — larger than the single-vector gain over the same scale jump (+2.1 vs. +0.5).

  • The multi-vector advantage grows with scale: the gap between MetaEmbed and single-vector is +3.5 points at 3B, +5.0 points at 7B, and +6.6 points at 32B. This is the empirical basis for the claim that multi-vector retrieval scales better with model size than single-vector retrieval. The green numbers in Figure 4a annotate the gain compared to the preceding model size, making the saturation of single-vector and continued growth of MetaEmbed visually apparent.

Ablation Studies and Robustness Checks

Matryoshka Multi-Vector Retrieval (MMR) vs. no MMR (Figure 4b, Table 5): The most critical ablation tests whether the nested training objective is necessary for test-time scalability. Figure 4b compares MetaEmbed-3B trained with and without MMR on ViDoRe v1, measuring average NDCG@5 across all five retrieval budgets. Without MMR, performance at (1,1) drops to 68.9% — a 9.0-point decrease from the full-budget performance (the figure shows NoMMR-(16,64) at approximately 77.9%, though the exact values are read from the plot). With MMR, the (1,1) performance is substantially higher (the curve shows approximately 78.5% at (1,1) with MMR, nearly matching the NoMMR full-budget performance). As the budget increases, the gap narrows: at (16,64), the MMR model reaches approximately 88.7% vs. 88.4% for the NoMMR variant — a negligible difference, confirming that MMR does not sacrifice full-budget quality. Table 5 provides complementary MMEB data: NoMMR-(16,64) achieves 69.3% at 3B vs. MetaEmbed's 69.1% (∆ = +0.2) and 77.0% at 7B vs. 76.6% (∆ = +0.4), again showing statistically indistinguishable full-budget performance with or without MMR. The crucial takeaway is that MMR provides test-time flexibility with essentially zero cost to peak performance.

Single-vector baselines (Table 5, Appendix B): To isolate whether MetaEmbed's gains come from the multi-vector representation or from the training recipe itself, the paper compares against two single-vector baselines trained identically. single-last uses the last-token hidden state from the final layer as the retrieval vector (standard practice for VLM-based embedders). single-mean applies average pooling over all last-layer hidden states. At 7B, single-last achieves 71.6%, single-mean achieves 71.2% (−0.4), and MetaEmbed-(16,64) achieves 76.6% (+5.0 over single-last). At 32B, single-last achieves 72.1%, single-mean achieves 71.1% (−1.0), and MetaEmbed-(16,64) achieves 78.7% (+6.6). The single-mean underperforming single-last suggests that the last token's hidden state (which in causal VLMs has attended to the entire sequence) is a better summary than uniform pooling, consistent with standard practice. MetaEmbed's gains are not attributable to training data or optimization — they come from the multi-vector representation.

Naive multi-vector baseline: split-(16, 64) (Table 5): To test whether any multi-vector representation (not just learned Meta Tokens) provides gains, the paper implements a baseline that evenly partitions the VLM's last-layer hidden states into 16 query-side segments and 64 candidate-side segments, taking the mean of each segment as a vector. This introduces no additional parameters and serves as a fixed-length multi-vector method. At 7B, split-(16,64) achieves 70.1% — not only below MetaEmbed-(16,64) (76.6%, −6.5 points) but also below the single-vector single-last baseline (71.6%, −1.5 points). The fact that a naive multi-vector partition underperforms even single-vector retrieval demonstrates that arbitrary multi-vector representations are not inherently better — the vectors must be learned to extract discriminative information, which is what Meta Tokens provide. At 3B, split-(16,64) achieves 64.2% vs. single-last at 65.6% and MetaEmbed at 69.1%. At 32B, split-(16,64) achieves 70.5% vs. single-last at 72.1% and MetaEmbed at 78.7%. The split baseline consistently underperforms single-vector, while MetaEmbed consistently outperforms both.

VLM backbone architecture (Table 1, Figure 3d): MetaEmbed is tested on three VLM families to assess robustness to architectural differences. The Qwen2.5-VL variants (3B, 7B, 32B) achieve the strongest and most balanced results. PaliGemma-based MetaEmbed-3B^Gemma^ achieves 65.4% overall on MMEB — competitive with MetaEmbed-3B (Qwen2.5-VL) at 69.1%, suggesting the recipe transfers across unified-architecture VLMs with moderate performance variation attributable to backbone quality. The Llama-3.2-Vision-based MetaEmbed-11B achieves 65.1% overall but with the severe VQA deficit (42.1%) discussed earlier. Figure 3d confirms that test-time scaling still functions (60.0% to 65.1%, +5.0 points) despite the backbone's cross-modal limitations, demonstrating that the MMR mechanism and Meta Token design are architecture-agnostic. The recommendation implicit in these results is that Qwen2.5-VL is the preferred backbone for balanced multimodal retrieval, but MetaEmbed can be applied to any VLM.

Model size scaling with MMR disabled (Table 5, NoMMR rows): The NoMMR variants at full budget (16,64) achieve: 69.3% at 3B, 77.0% at 7B, 79.1% at 32B, and 66.2% at 11B (Llama). Comparing to the corresponding MetaEmbed (with MMR) rows: 69.1% vs. 69.3% (3B, ∆ = −0.2), 76.6% vs. 77.0% (7B, ∆ = −0.4), 78.7% vs. 79.1% (32B, ∆ = −0.4), 65.1% vs. 66.2% (11B, ∆ = −1.1). The differences are small (≤1.1 points) and likely within noise, confirming that MMR does not degrade peak performance. The 11B variant shows the largest gap (−1.1), which may reflect the Llama backbone's training dynamics differing from Qwen2.5-VL under the MMR objective, though the paper does not investigate this.

Training data and hard negatives: The paper trains on only MMEB-train and ViDoRe-train with one explicit hard negative per query from Chen et al. (2025a). This is deliberately minimalist compared to baselines like mmE5 and MegaPairs that use large-scale synthetic data generation. The strong results (76.6% at 7B, surpassing mmE5-11B at 69.8%) suggest that architectural improvements (Meta Tokens + MMR) can substitute for data scale when the backbone VLM is strong. However, the paper does not ablate over: different numbers of hard negatives, training without hard negatives, training data scale, or the effect of adding synthetic data. These missing ablations mean we cannot determine whether MetaEmbed's gains would be even larger with more data, or whether the minimalist data recipe is optimal.

Group size configuration: The paper uses G = 5 groups at {(1,1), (2,4), (4,8), (8,16), (16,64)} and states these are "empirically chosen" (Section 4.1). There is no ablation over: the number of groups G, alternative size progressions (e.g., geometric vs. linear vs. custom spacing), different maximum budgets (e.g., (32, 128)), or asymmetric configurations where query and candidate budgets are selected independently. This is a significant gap because the group sizes directly determine the available operating points and the granularity of the accuracy-efficiency tradeoff. A practitioner wanting different tradeoff points (e.g., (1,8) for fast query with detailed index, or (32,32) for symmetric retrieval) would need to retrain or extrapolate from the provided curves.

LoRA rank and training hyperparameters: The paper fixes LoRA rank to 32 with α = 32, temperature τ = 0.03, group weights w_g = 1, and learning rates as specified in Table 4. None of these are ablated. Given that MetaEmbed is presented as a training recipe, the sensitivity of results to these hyperparameters is unknown. The 10× lower learning rate for the 32B model (1 × 10⁻⁵ vs. 1 × 10⁻⁴ for smaller models) suggests some hyperparameter tuning occurred, but the paper does not discuss the tuning process or report sensitivity.

Embedding dimension and precision: All experiments use bfloat16 precision (Wang and Kanwar, 2019) for storage and scoring. The embedding dimensions (Table 4) match each backbone's native hidden size: 2,048 for 3B models, 3,584 for 7B, 5,120 for 32B, 4,096 for 11B. The paper does not ablate over reduced precision (e.g., int8 quantization for index compression), dimension reduction (e.g., applying traditional MRL on the per-vector dimension in addition to vector-count nesting), or the interaction between embedding dimension and retrieval budget.

Critical Assessment

Does MetaEmbed enable test-time scaling that balances accuracy against efficiency?

Supported with qualifications. Figure 3 and Table 3 together demonstrate that accuracy increases monotonically with retrieval budget across all model sizes, and Table 3 quantifies the corresponding efficiency costs. The core mechanism works: a single trained model serves multiple accuracy-efficiency operating points. However, the paper characterizes this as "test-time scaling" but only demonstrates the accuracy half — the efficiency measurements in Table 3 are for a specific hardware configuration (A100 GPU, 100,000 candidates, bfloat16, batch size 1,000) and a single model size (7B). We do not see efficiency curves for 3B, 32B, or 11B models, nor for different candidate corpus sizes, different GPU architectures, or different precision formats. The claim that scoring latency is "not compute-bounded until the extreme case of (16, 64)" (Section 5) is based on a single data point and may not generalize to larger corpora (where memory bandwidth pressure increases) or different hardware (where the compute/memory balance differs).

More importantly, the paper does not compare MetaEmbed's accuracy-efficiency Pareto frontier against alternatives that could achieve similar flexibility. For instance, a single-vector model with traditional MRL on embedding dimension could also provide multiple operating points (by truncating dimensions), and the paper does not benchmark against this. A ColBERT-style model with vector pruning or clustering-based compression could also offer budget-controlled retrieval. Without these comparisons, we cannot assess whether MetaEmbed's Pareto frontier is actually superior to what existing methods could achieve with their own test-time scaling mechanisms.

Does MetaEmbed achieve state-of-the-art retrieval performance?

Supported on MMEB, with caveats. MetaEmbed-7B (76.6%) and MetaEmbed-32B (78.7%) are clearly the highest reported numbers on MMEB in Table 1. However, state-of-the-art claims in rapidly moving benchmarks are inherently temporal — the paper's baselines are taken from prior publications, and concurrent or subsequent work may have exceeded these numbers. The paper acknowledges this by including very recent baselines (MoCa, B3, mmE5), strengthening the claim.

The more important caveat is that MetaEmbed's results are at the full (16,64) budget, which is substantially more expensive at indexing time than single-vector methods. Table 3 shows that the index for 100,000 candidates at (16,64) requires 42.72 GiB — 63× more than the single-vector (1,1) index at 0.68 GiB. The single-vector baselines (MoCa-7B at 71.5%, B3-7B at 72.0%) achieve their scores with 1 vector per candidate. MetaEmbed-7B at (2,4) — which uses 4 candidate vectors, a 4× larger index than single-vector — achieves 72.0%, exactly matching B3-7B. At (4,8) with an 8× larger index, it achieves 72.9%, barely edging ahead. The paper's headline 76.6% uses a 64× larger index than single-vector methods. Whether this is a fair comparison depends on the deployment context: if index storage is cheap and retrieval accuracy is paramount, the 64× overhead is justified; if index storage is the bottleneck, MetaEmbed's advantage shrinks dramatically. The paper is transparent about these numbers (Table 3) but does not explicitly discuss this storage-accuracy tradeoff in interpreting the headline results.

Does MetaEmbed scale robustly to 32B parameters with minimal diminishing returns?

Partially supported. Figure 3c shows MetaEmbed-32B improving to 78.7% from 72.4% at (1,1), a +6.6-point gain, which is the largest absolute gain across all model sizes. Figure 4a shows the multi-vector advantage growing from +3.5 at 3B to +6.6 at 32B. These data support the claim that multi-vector representations benefit from larger models.

However, the claim of "minimal diminishing returns" is harder to assess. The gain from 7B to 32B is +2.1 points for MetaEmbed vs. +0.5 points for single-vector. While +2.1 is larger than +0.5, it still represents a significant slowdown relative to the +7.5-point gain from 3B to 7B. Scaling from 7B to 32B (a 4.6× parameter increase) yields less improvement than scaling from 3B to 7B (a 2.3× increase). This is textbook diminishing returns, just less severe than for single-vector methods. The paper's framing — "minimal diminishing returns as models grow larger" — is relative to the more severe saturation of single-vector methods, not an absolute claim of linear scaling. This distinction is important but could be lost on readers who see "minimal diminishing returns" and expect near-linear improvement.

Additionally, the paper only evaluates at three scale points (3B, 7B, 32B) and one architecture family (Qwen2.5-VL). Three points are insufficient to characterize a scaling trend reliably — interpolation between 7B and 32B might hide inflection points, and we have no data at intermediate scales (e.g., 14B) that would reveal whether the improvement is smooth or stepwise. The 11B Llama variant cannot serve as an intermediate point because it uses a different architecture with known task-specific weaknesses, making it incomparable for scaling analysis.

Does MetaEmbed's performance depend critically on backbone VLM quality?

Strongly supported. The MetaEmbed-11B (Llama-3.2-Vision) results in Table 1 and Figure 3d provide compelling evidence: VQA drops to 42.1% (vs. 74.2% for 7B Qwen2.5-VL) while Grounding reaches 91.6%, mirroring the known strengths and weaknesses of the underlying Llama model. This is the cleanest demonstration that MetaEmbed does not compensate for backbone deficiencies — it amplifies existing capabilities but cannot create new ones. The PaliGemma variant (65.4% vs. 69.1% for Qwen2.5-VL at 3B) provides additional confirmatory evidence with a different architecture. The paper's transparency about this limitation is a strength: it clarifies the boundary of what MetaEmbed can and cannot do.

However, this finding also limits the generality of the paper's headline results. All the strongest numbers come from Qwen2.5-VL backbones, which may have properties (training data, architecture, pretraining objectives) that make them particularly suitable for MetaEmbed's fine-tuning recipe. The paper does not investigate what specific backbone properties correlate with MetaEmbed performance — is it VQA capability? Cross-modal alignment quality? Representation dimensionality? Understanding this would help practitioners select backbones and guide future VLM development for retrieval.

Critical gaps and missing experiments

Several experiments would have significantly strengthened the paper's claims:

  1. Efficiency curves at multiple model sizes: Table 3 only covers MetaEmbed-7B. We need latency and memory data for 3B and 32B to assess whether the efficiency-accuracy tradeoff shifts with scale (e.g., does the 32B model's scoring remain memory-bandwidth-bound?).

  2. Comparison against dimensionality-based MRL on single-vector models: A single-vector model trained with traditional MRL (nesting on embedding dimension) could provide test-time scaling without multi-vector overhead. How does MetaEmbed's (1,1) performance compare to a single-vector MRL model at equivalent storage? How does MetaEmbed's (16,64) compare to a single-vector model with 16× the embedding dimension (if that were feasible)?

  3. Ablation over group size configurations: The fixed {(1,1), (2,4), (4,8), (8,16), (16,64)} progression is never justified. Does geometric spacing matter? Would (1,2), (2,8), (4,32), (8,64) work as well? What if G=3 instead of G=5? The sensitivity of results to this design choice is unknown.

  4. Retrieval latency at scale: Table 3 uses 100,000 candidates — a moderate corpus size. Production systems may index millions or billions of candidates. How does scoring latency scale to 1M, 10M, 100M candidates? At what corpus size does the scoring cost become dominant over query encoding?

  5. Multimodal-to-multimodal retrieval evaluation: The paper motivates MetaEmbed partly by noting that full multi-vector methods "do not support multimodal-to-multimodal retrieval" (Section 2). Yet neither MMEB nor ViDoRe v2 includes image-to-image retrieval tasks where both query and candidate are images. This capability — which is a key claimed advantage — is never directly evaluated.

  6. Training data ablation: The paper uses a deliberately minimalist data recipe (MMEB-train + ViDoRe-train). Would adding more data (synthetic, larger corpora) improve MetaEmbed further, or has the architecture saturated? Would MetaEmbed benefit from data beyond what single-vector methods use?

  7. Candidate corpus size ablation for ViDoRe: ViDoRe v2 results are reported as NDCG@5, but the corpus size for each track is not specified in the paper. Retrieval metrics depend on corpus size (harder to rank correctly in larger corpora), making cross-method comparisons potentially confounded if different papers evaluate on different corpus sizes.

These gaps do not invalidate the paper's contributions but mean that several of its claims — particularly around efficiency, scaling behavior, and generality — are supported more narrowly than the framing suggests. The paper convincingly demonstrates that MetaEmbed works well on MMEB and ViDoRe v2 with Qwen2.5-VL backbones, and that MMR enables multi-granularity retrieval from a single model. Whether it "charts a path toward multimodal retrieval systems that are both accurate and deployable at scale" (Section 6) depends on factors — very large corpus efficiency, multimodal-to-multimodal performance, robustness to backbone choice — that are partially but not fully validated.

6. Limitations and Trade-offs

Limitation 1: The Full Retrieval Budget That Achieves State-of-the-Art Requires a Dramatically Larger Index Than Single-Vector Methods

The assumption or constraint. The paper's headline results on MMEB — MetaEmbed-7B at 76.6% and MetaEmbed-32B at 78.7% — are reported at the full retrieval budget of (16, 64), meaning 64 candidate-side vectors are stored in the index for every item in the retrieval corpus. Single-vector baselines like MoCa-7B (71.5%) and B3-7B (72.0%) achieve their scores with 1 vector per candidate. Table 3 quantifies this explicitly: for 100,000 candidates at bfloat16 precision, the (1,1) index requires 0.68 GiB while the (16,64) index requires 42.72 GiB — a 63× increase in index storage. The paper acknowledges that "index memory consumption can grow proportionally with the retrieval budget" and notes this "can present challenges for large deployments" (Section 5), but does not fold this overhead into how it frames the accuracy comparisons.

The consequence. The state-of-the-art claims are made at an operating point with fundamentally different resource requirements than the baselines they are compared against. At retrieval budgets with comparable or modestly larger index sizes, MetaEmbed's advantage over single-vector methods narrows substantially. MetaEmbed-7B at (2,4) — a 4× larger index than single-vector — achieves 72.0% on MMEB, effectively tied with B3-7B (72.0%) and only marginally ahead of MoCa-7B (71.5%). At (4,8) with an 8× larger index, it reaches 72.9%. The bulk of MetaEmbed's advantage (72.9% → 76.6%, a +3.7-point gain) comes from the most expensive index configurations — (8,16) with a 16× larger index and (16,64) with a 64× larger index. For production deployments where index storage is the binding constraint (e.g., billion-scale web search or document retrieval), the relevant comparison is MetaEmbed at small-to-moderate budgets against single-vector methods, where the accuracy advantage is ~0–2 points, not 5–7 points. A practitioner reading the headline "outperforms by over 5 absolute points" without consulting Table 3 would make deployment decisions based on a resource mismatch.

What evidence exists in the paper. Table 3 provides the storage numbers directly, and Figure 3 shows the per-budget accuracy for MetaEmbed-7B. The single-vector baseline is plotted as a dashed green line in Figure 3b at 71.6%, which MetaEmbed crosses at (2,4) with 72.0%. So the paper provides all the data needed to make the storage-accuracy tradeoff explicit, but the narrative framing in Section 4.2 and the abstract emphasizes the full-budget numbers without equivalently emphasizing the storage cost. The ViDoRe v2 comparisons in Table 2 further illustrate this: MetaEmbed-7B (61.3%) leads GME-7B (59.3%) by only 2.0 points, and MetaEmbed-3B (60.3%) leads MoCa-3B (59.8%) by 0.5 points, but the index overhead is not reported for ViDoRe, making it unclear whether these gains justify the storage cost in document retrieval contexts where corpora are typically large.

Mitigation status. The paper suggests two mitigations in Section 5: "using a balanced retrieval budget" (e.g., indexing at a smaller r_c) and "more frequent offloading of index data to CPU memory." The first mitigation is effectively an acknowledgment that the full-budget advantage is not always achievable — you trade accuracy for storage, and the paper provides the data to make that trade. The second mitigation (CPU offloading) is standard for large-scale retrieval systems but introduces its own latency costs that are not measured. The paper does not propose a principled method for selecting the optimal storage budget given a corpus size and latency constraint, nor does it benchmark MetaEmbed against single-vector methods at equal index memory (e.g., by comparing MetaEmbed-(1,4) against single-vector with 4× the embedding dimension, if both consume comparable memory). This leaves the storage-accuracy Pareto frontier partially characterized — we know the MetaEmbed points on it, but not where single-vector methods would sit at equivalent storage.

Limitation 2: The Hardest Multimodal Reasoning Tasks Remain Unsolved, and MetaEmbed's Gains Vanish Where the VLM Backbone Struggles

The assumption or constraint. MetaEmbed is a training recipe applied to a pretrained VLM; it operates entirely by fine-tuning the backbone's existing representations into a retrieval-compatible format. The paper is explicit that this means MetaEmbed inherits the backbone's fundamental limitations: "if the underlying base model itself struggles on some domains when used as a generative model, such a weakness directly propagates into MetaEmbed as an embedding model" (Section 4.2). The MetaEmbed-11B (Llama-3.2-Vision) results make this concrete: VQA performance collapses to 42.1%, more than 32 points below the 7B Qwen2.5-VL variant's 74.2%. This is not a matter of insufficient multi-vector capacity — the model has 16 query vectors and 64 candidate vectors available — but of the underlying VLM failing to cross-modally reason about visual questions in the first place.

The consequence. There exists a class of multimodal retrieval tasks — specifically those requiring deep integration of visual and linguistic semantics, as measured by VQA on MMEB — where MetaEmbed cannot help regardless of retrieval budget if the backbone VLM is not strong on those tasks. The paper's scaling analysis (Figure 3) shows that accuracy improves with retrieval budget for all model sizes, but only within the performance envelope defined by the backbone. MetaEmbed-11B improves from 60.0% at (1,1) to 65.1% at (16,64) — a +5.0-point gain comparable to the 7B Qwen2.5-VL model's +5.3-point gain. But the absolute ceiling is 65.1%, far below the 76.6% of the 7B Qwen variant. The multi-vector representation amplifies what the backbone can do but cannot create cross-modal understanding where it is absent. This means that deploying MetaEmbed on a backbone with task-specific weaknesses (e.g., a VLM that is strong on natural images but weak on charts and diagrams, or strong on English but weak on multilingual content) will produce embeddings with corresponding blind spots. The paper's ViDoRe v2 results hint at a more optimistic picture for multilingual transfer — MetaEmbed performs well on multilingual tracks despite no multilingual training data — but this likely reflects the Qwen2.5-VL backbone's inherent multilingual capabilities rather than any property of MetaEmbed itself.

What evidence exists in the paper. Table 1 provides the per-category breakdown that isolates this limitation. The MetaEmbed-11B row shows Classification at 66.4%, VQA at 42.1%, Retrieval at 74.3%, and Grounding at 91.6% — extreme variance across tasks that mirrors the known profile of Llama-3.2-Vision (Huang et al., 2025). Figure 3d shows the scaling curve for this model is well-behaved (monotonic improvement with budget), confirming that the problem is not with MetaEmbed's mechanism but with the starting point. The paper does not report per-category scaling curves (e.g., does VQA improve with budget at the same rate as Retrieval for the Llama variant?), so we cannot assess whether the multi-vector representation partially compensates for backbone weaknesses on specific task types or simply amplifies all task types equally.

Mitigation status. The paper is transparent about this limitation and effectively treats it as a feature rather than a bug: MetaEmbed faithfully preserves backbone quality, which means practitioners should select backbones carefully based on their target task distribution. The paper's multi-backbone evaluation (Qwen2.5-VL, PaliGemma, Llama-3.2-Vision) is presented partly as a diagnostic tool for backbone selection. However, the paper does not provide guidance on how to predict backbone suitability for MetaEmbed without running the full fine-tuning and evaluation pipeline. Is strong VQA performance in the base VLM predictive of strong MetaEmbed VQA? Are there specific architectural features (unified vs. cross-attention, training data mixture, pretraining objectives) that correlate with MetaEmbed performance? Without such guidance, a practitioner must train and evaluate MetaEmbed on each candidate backbone, which is expensive at the 7B–32B scale.

Limitation 3: Multimodal-to-Multimodal Retrieval Is a Motivating Use Case That Is Never Evaluated

The assumption or constraint. One of the paper's central motivations for MetaEmbed is that full token-level multi-vector methods (ColPali, ColQwen) "do not support multimodal-to-multimodal retrieval, since introducing hundreds of image tokens on the query side renders both training and inference computationally prohibitive" (Section 2). MetaEmbed's fixed small cardinality (at most 16 query-side vectors, 64 candidate-side vectors) is positioned as the solution: it makes late interaction between two image-containing inputs feasible because the pairwise comparison cost is bounded at $r_q \times r_c = 1,024$ regardless of how many visual patches each input contains. This capability is presented as a key advantage over existing multi-vector methods and as a core motivation for the Meta Token design.

The consequence. The claim that MetaEmbed enables multimodal-to-multimodal retrieval is not validated. Both MMEB and ViDoRe v2 are evaluated on the standard retrieval setup where queries are primarily text or text+image, and candidates are images or documents. The paper does not report a single result on image-to-image retrieval, video-to-video retrieval, or any task where both query and candidate contain substantial visual content. While the architectural argument is sound (replacing 1,024 patch tokens with 16 learned tokens reduces the interaction matrix from ~10^6 to ~10^3), architectural arguments without empirical validation are hypotheses, not results. It is possible that the Meta Tokens, when both sides of the retrieval pair are visual, fail to capture the fine-grained visual correspondences that patch-level late interaction would detect — the learned tokens might default to coarse global image matching that underperforms even single-vector methods. Alternatively, they might work perfectly. We do not know because the experiment was not conducted. Given that this capability is one of the primary differentiators from existing multi-vector methods, its absence from the evaluation is a significant gap between the paper's claimed scope and its demonstrated scope.

What evidence exists in the paper. None. The MMEB benchmark includes tasks across Classification, VQA, Retrieval, and Visual Grounding, but the Retrieval tasks are predominantly text-to-image or text-to-text. The ViDoRe v2 benchmark evaluates text queries against visual documents. There is no image-to-image retrieval task in either benchmark. The paper does not construct a custom evaluation or use an existing image-to-image retrieval benchmark (e.g., the retrieval track of MMEB appears to use text queries against image candidates based on the task descriptions in Section 4.1). The efficiency analysis (Table 3) only measures text-to-candidate scoring; the query encoding cost for an image query is mentioned (788 ms for 1,024 tokens) but never benchmarked in a retrieval setting with image queries against image candidates.

Mitigation status. The paper does not acknowledge this gap. Section 2 presents multimodal-to-multimodal retrieval as an unsolved problem that MetaEmbed addresses; Section 6 (Conclusion) claims MetaEmbed "bridg[es] the gap between fine-grained expressiveness and large-scale deployability." But the deployability of multimodal-to-multimodal retrieval with MetaEmbed is never tested. The architecture supports it in principle — (16, 64) late interaction between two image-encoded inputs — but whether the learned Meta Tokens produce representations that are sufficiently expressive for this harder setting is an open question. A minimal evaluation on even a small image-to-image retrieval dataset would have substantially strengthened this claim.

Limitation 4: The Group Size Configuration Is Fixed and Unexplored, Yet It Defines the Available Operating Points

The assumption or constraint. The Matryoshka Multi-Vector Retrieval training relies on a predefined set of nested prefix groups. The paper uses $G = 5$ groups at $\{(1,1), (2,4), (4,8), (8,16), (16,64)\}$ for all experiments and states these were "empirically chosen" (Section 4.1) without further justification or ablation. The maximum query-side budget (16) and candidate-side budget (64), the exponential spacing, the asymmetry between query and candidate growth rates, and the number of groups (5) are all design choices that directly determine: (a) the granularity of the accuracy-efficiency tradeoff available at test time; (b) the computational cost of training (five parallel contrastive losses); and (c) the inductive bias on how information is organized across vector positions.

The consequence. The paper presents test-time scaling as a continuous-spectrum capability — "users can balance retrieval quality against efficiency demands by selecting the number of tokens used" (abstract) — but in practice, users are restricted to exactly five discrete operating points. If a deployment needs a budget of (1, 8) for fast query processing with a moderately detailed index, or (4, 4) for symmetric retrieval, or (8, 32) for a specific GPU memory constraint, the trained model does not support it. The user must approximate with the nearest available budget, which may be suboptimal. More importantly, because the group sizes are never ablated, we do not know whether the observed performance at each budget is near-optimal or whether alternative configurations would yield a better accuracy-efficiency Pareto frontier. Could a denser set of groups (e.g., G=9 with (1,1), (1,2), (2,4), (4,8), (6,12), (8,16), (10,32), (12,48), (16,64)) provide finer-grained control without degrading peak performance? Would different spacing (e.g., linear rather than exponential) produce better results at intermediate budgets? Would training with asymmetric groups where query and candidate budgets are varied independently (rather than paired) enable the full cross-product of budget combinations at test time? These questions are not addressed, and the fixed group configuration is presented as an inherent part of the method rather than a hyperparameter to be tuned.

What evidence exists in the paper. The only evidence about group size sensitivity is indirect: Figure 4b shows that MMR training (with the chosen groups) substantially outperforms no-MMR training at small budgets while matching at full budget, and Table 5 confirms the full-budget equivalence is consistent across model sizes. This tells us MMR works, but not whether the chosen configuration is optimal. The paper does not ablate over G (number of groups), spacing, symmetry, or maximum budgets. Appendix B's Table 5 provides the only configuration-adjacent data by comparing against single-vector baselines, but these are single-group configurations trained separately, not alternative MMR configurations.

Mitigation status. The paper does not acknowledge the sensitivity of results to group size configuration as a limitation. The configuration is presented as a fixed design choice. A simple ablation — say, training with G=3 groups {(1,1), (4,8), (16,64)} and comparing the scaling curve to G=5 — would have shown whether the intermediate groups (2,4) and (8,16) provide meaningful additional operating points or mainly serve as regularization. The paper's claim that group sizes are "empirically chosen" implies some tuning was done, but no tuning results or alternative configurations are reported. This makes it impossible for a practitioner to adapt the group configuration to their specific deployment needs without retraining and evaluating multiple variants from scratch.

Limitation 5: Indexing and Query Costs Are Measured Only at Moderate Scale with a Single Hardware Configuration, Limiting Extrapolation to Production Settings

The assumption or constraint. The efficiency analysis in Table 3 measures scoring latency and index memory for MetaEmbed-7B with exactly 100,000 candidates on a single A100 GPU with bfloat16 precision and a scoring batch size of 1,000. The three key claims about efficiency — that scoring is not compute-bounded until extreme budgets, that query encoding dominates total latency, and that index memory is the primary scaling constraint — are all derived from this single configuration. The paper states that "these findings suggest that MetaEmbed is efficient in practice" (Section 5), generalizing from the 100K-candidate measurement to "practice" writ large.

The consequence. Production retrieval systems operate across a wide range of corpus sizes: enterprise document search might involve 100K–1M documents; web-scale image search involves 100M–1B+ images; retrieval-augmented generation might query a 10M-document index. The efficiency characteristics measured at 100K candidates do not necessarily extrapolate. Specifically:

  • Scoring latency scales at least linearly with corpus size if brute-force search is used (as Table 3 assumes — all 100K candidates are scored per query). At 1M candidates, the (16,64) scoring latency would be approximately 62.5 ms (10× the 6.25 ms for 100K). At 100M candidates without approximate nearest neighbor search, it would be 6,250 ms — completely prohibitive for interactive use. The paper does not discuss whether approximate search (e.g., using the coarse prefixes for filtering, then reranking with larger budgets) is compatible with MetaEmbed's multi-vector late interaction.

  • Memory bandwidth saturation may occur at larger corpus sizes or different hardware. The finding that latency is flat from (1,1) to (4,8) despite 32× FLOPs increase is attributed to memory-bandwidth-bound kernels on the A100. On a GPU with different memory bandwidth (e.g., H100 with higher bandwidth, or an inference-optimized card with lower bandwidth), the crossover point where latency becomes compute-bound may shift, changing the practical tradeoff curve.

  • Index memory for production corpora at full budget becomes infeasible: 100M candidates at (16,64) with bfloat16 and $D = 3,584$ would require approximately 42,720 GiB — over 40 TB. Even at (4,8), it would be ~5,340 GiB. This is not a "can present challenges" scenario (as the paper describes it); it is a hard requirement for aggressive index compression, CPU offloading, or sharding that the paper does not design or evaluate. The proposed mitigations (balanced budget, CPU offloading) are mentioned but not measured — we do not know the latency penalty of CPU offloading for MetaEmbed's multi-vector scoring, nor whether the accuracy of small-budget retrieval is sufficient for the production use case.

What evidence exists in the paper. Table 3 provides the 100K-candidate measurements for MetaEmbed-7B only. No scaling curves for latency vs. corpus size, no measurements on hardware other than A100, no measurements at other precision formats (e.g., int8 quantization, which could reduce index memory by 2× or more), no measurements for the 3B or 32B model variants (which have different embedding dimensions and may have different compute-vs-memory tradeoffs), and no approximate search integration results.

Mitigation status. The paper acknowledges the index memory challenge in passing ("this can present challenges for large deployments") and gestures at mitigations (balanced budget, CPU offloading) without implementing or evaluating them. The efficiency analysis is positioned as a preliminary characterization rather than a production deployment guide, which is reasonable for a research paper introducing a new method. However, the paper's claims about deployability ("charting a path toward multimodal retrieval systems that are both accurate and deployable at scale," Section 6) outpace the evidence. A more defensible framing would present the 100K-candidate measurements as a proof of concept and explicitly call out production-scale efficiency as future work. The absence of even approximate search integration is particularly notable: all practical large-scale retrieval systems use approximate nearest neighbor search or inverted indexing to avoid brute-force scoring over the full corpus, and whether MetaEmbed's MaxSim late interaction is compatible with standard approximate search indices (which are designed for single-vector or fixed-cardinality representations) is an open question.

Limitation 6: All Evaluations Are on English-Centric Benchmarks with Ground-Truth Relevance Judgments, Leaving Open-Domain and Multilingual Retrieval Unexplored

The assumption or constraint. MetaEmbed is evaluated on MMEB and ViDoRe v2, both of which are curated benchmarks with: (a) predominantly English content (ViDoRe v2 includes multilingual tracks, but MMEB is English-centric based on the task descriptions); (b) clean, pre-defined query-candidate relevance pairs with ground-truth labels; and (c) a closed candidate set where the correct answer is always present. This setup evaluates embedding quality for the standard retrieval-as-ranking problem, but it does not capture several dimensions of real-world retrieval: open-domain queries where relevant content may not exist in the corpus, queries with ambiguous or multi-faceted intent, retrieval from noisy or adversarial corpora, and retrieval in languages or domains not represented in training.

The consequence. The paper's claims about generality — "advancing the pursuit of generality, efficiency, and flexibility" (Section 1) — are based on evaluation within a specific, controlled paradigm. We do not know whether MetaEmbed's Meta Token representations generalize to: (a) truly out-of-domain retrieval tasks (e.g., medical images with radiology reports, legal document retrieval with domain-specific terminology, e-commerce product search with attribute-heavy queries); (b) multilingual retrieval where the query and candidate languages differ and the backbone's multilingual capabilities are strained beyond what ViDoRe v2's multilingual tracks measure; (c) adversarial or distribution-shifted queries where the VLM's pretraining biases may be surfaced through the Meta Token aggregation; or (d) open-set retrieval where queries may have zero relevant candidates and the model must produce calibrated low scores (a retrieval model that overestimates similarity on irrelevant pairs would perform poorly). The ViDoRe v2 multilingual results are encouraging — MetaEmbed-7B achieves 61.1% on ESG_Syn_Mul vs. 56.7% for GME — but these are synthetic document retrieval tasks, not natural multilingual queries from diverse languages and domains. The paper notes that "no explicit multilingual data was included during training" and attributes the multilingual performance to backbone capabilities, but does not measure how this performance degrades for languages that the backbone handles poorly.

What evidence exists in the paper. MMEB covers 36 tasks across four categories, providing some domain diversity (news images, fashion, Wikipedia entities, charts, science QA, COCO, RefCOCO). However, all tasks share the common structure of closed-set ranking with ground-truth relevance. The paper does not report on any zero-shot retrieval benchmark outside MMEB and ViDoRe, any open-domain retrieval task, or any robustness evaluation (e.g., performance under query paraphrasing, image perturbations, or distractor candidates). The per-category MMEB breakdown in Table 1 shows MetaEmbed's performance varies across categories (e.g., 7B achieves 71.3% Classification, 74.2% VQA, 78.7% Retrieval, 85.4% Grounding), which is expected, but we cannot infer from these numbers how the model would perform on a held-out domain not represented in any MMEB category.

Mitigation status. The paper does not frame its evaluation scope as a limitation — it presents MMEB as "a comprehensive suite of retrieval tests covering images, text and visual documents" (Section 1) and uses the results to support generality claims. The ViDoRe v2 multilingual and biomedical tracks partially address domain diversity, but the fact that MetaEmbed performs well on them despite no domain-specific training is attributed to backbone quality rather than MetaEmbed's domain generalization ability per se. The absence of open-domain retrieval evaluation is not discussed. A more cautious framing would acknowledge that MMEB and ViDoRe v2 measure retrieval capability within a specific paradigm (closed-set ranking with curated negatives) and that deployment in open-domain, multilingual, or adversarial settings may reveal failure modes not captured by these benchmarks — particularly if the Meta Tokens learn to rely on dataset-specific heuristics that do not transfer. This is a standard limitation of benchmark-based retrieval evaluation, but it is worth stating explicitly given the paper's emphasis on generality and deployability.

7. Implications and Future Directions

How This Work Changes the Landscape

MetaEmbed introduces a new axis of flexibility for multimodal retrieval that did not previously exist: the ability to deploy a single trained model at multiple accuracy-efficiency operating points by varying the number of vectors used in late interaction, without retraining. This is not a paradigm shift in the sense of replacing the dominant retrieval architectures — single-vector and full multi-vector methods remain viable for their respective sweet spots — but it is a meaningful reframing of how the field should think about the relationship between representational expressiveness and deployment cost.

The conceptual contribution is the recognition that multi-vector retrieval can be made composable along the vector-count axis, analogous to how Matryoshka Representation Learning made single-vector retrieval composable along the embedding-dimension axis. The paper demonstrates that this composability is not an emergent property of any multi-vector model — it must be explicitly trained for via the nested MMR objective — and that training for it incurs negligible cost to peak performance (Table 5: NoMMR-(16,64) achieves 77.0% vs. MetaEmbed's 76.6% at 7B, a statistically indistinguishable difference). This means the field now has a recipe for making multi-vector retrieval flexible by default, with no accuracy penalty for doing so. Future work on multi-vector retrieval should treat test-time scalability as a standard feature, not an optional add-on — the same way that modern text embedding models now routinely support MRL on embedding dimension.

The paper resolves a tension that has been implicit in the multi-vector retrieval literature: how to get the expressiveness benefits of multi-vector representations without being locked into the full computational cost at deployment time. Prior work addressed this through post-hoc compression (Plaid for ColBERT, Santhanam et al., 2022a), learned compression (CITADEL, Li et al., 2023), or approximate search (DESSERT, Engels et al., 2023) — all of which accept some accuracy degradation relative to the full multi-vector model. MetaEmbed takes a different approach: bake the flexibility into the training objective itself, so that every prefix of the representation is a valid, trained representation in its own right. This shifts the burden from the retrieval engine (which previously had to approximate or compress a monolithic multi-vector representation) to the training procedure (which now produces a naturally truncatable representation). This is a cleaner separation of concerns: the model handles representation quality across granularities; the retrieval engine simply selects the appropriate granularity at query time.

An important reframing the paper achieves is decoupling the representation cardinality from the input complexity. In ColBERT-style methods, the number of vectors grows with input length — more text tokens produce more vectors, higher-resolution images produce more patch vectors. The retrieval cost is therefore input-dependent, which complicates latency budgeting and index sizing. MetaEmbed fixes the cardinality to a small designer-chosen constant — (16, 64) in the paper's configuration — regardless of whether the input is a 5-word query or a 1,024-patch image. This makes retrieval cost predictable and bounded, which is a first-order requirement for production systems with latency SLAs. The fact that the learned Meta Tokens achieve this while outperforming the full patch-level interaction of ColPali on ViDoRe v2 (MetaEmbed-3B: 60.3% vs. ColPali-3B: 54.5%, Table 2) suggests that the learned aggregation is not just compressing the input — it is extracting more discriminative information than the raw patch tokens provide. This finding should redirect some research attention away from compression-of-patch-tokens (an information preservation problem) toward learned-aggregation (an information extraction problem), since the latter appears to be both more accurate and more efficient.

The multi-backbone evaluation (Qwen2.5-VL, PaliGemma, Llama-3.2-Vision) provides a diagnostic contribution that should influence experimental methodology in the multimodal retrieval literature. The paper demonstrates that retrieval recipe quality and backbone VLM quality are separable variables: MetaEmbed-11B (Llama) collapses on VQA (42.1%) because the backbone collapses on VQA, not because MetaEmbed's recipe fails for VQA tasks. This means future papers proposing new retrieval fine-tuning methods should follow MetaEmbed's lead in evaluating across multiple backbones and reporting per-task breakdowns, to avoid conflating "our recipe is good" with "our chosen backbone happens to be strong on this benchmark." The single-backbone evaluation common in prior work (e.g., VLM2Vec on Phi-3.5-V, GME on Qwen2) makes it difficult to assess whether reported gains would transfer to other model families — MetaEmbed's results suggest they might not, or at least that the magnitude of gain depends heavily on backbone choice.

The finding that single-vector performance saturates at scale while multi-vector continues to improve (Figure 4a: single-vector gains from 7B to 32B are +0.5 points and "no longer statistically significant," while MetaEmbed gains +2.1 points over the same scale jump) has implications for the scaling roadmap of multimodal retrieval. If this trend holds at larger scales (70B, 100B+), it implies that future improvements in retrieval accuracy will require multi-vector representations to absorb the additional capacity of larger backbone VLMs. The single-vector paradigm — which has dominated since CLIP — may be approaching a fundamental information bottleneck that cannot be overcome by simply training larger models. This parallels the finding in the LLM literature that certain reasoning capabilities require test-time compute (chain-of-thought, search) rather than just larger pretrained models. For retrieval, the analog is that certain fine-grained cross-modal alignment requires multi-vector late interaction rather than just larger single-vector encoders. This positions MetaEmbed's approach not as an alternative to single-vector retrieval, but as the necessary next step for continued scaling.

Finally, the paper's efficiency analysis (Table 3) provides a useful calibration for the community: scoring latency for 100K candidates at (16, 64) is 6.25 ms on an A100 — fast enough that it is not the bottleneck in most retrieval pipelines, where query encoding (788 ms for a 1,024-token image) dominates. This means that multi-vector retrieval with a moderate number of learned vectors is already practical for production use on moderate-scale corpora, without requiring approximate search or aggressive compression. The primary constraint is index storage (42.72 GiB for 100K candidates at (16, 64)), not query-time latency. This should focus future efficiency work on index compression (quantization, learned sparse representations, CPU offloading strategies) rather than on scoring kernel optimization, since scoring is already fast enough for most use cases.

Follow-Up Research This Work Enables

Scaling MetaEmbed to larger backbone VLMs (70B+) to test whether the multi-vector advantage continues to grow monotonically with model capacity. Figure 4a shows the multi-vector advantage growing from +3.5 points at 3B to +6.0 at 7B to +7.5 at 32B (using MMEB overall scores). The paper only evaluates three scale points within one architecture family (Qwen2.5-VL). A critical open question is whether this trend continues — does a 70B Qwen2.5-VL-based MetaEmbed achieve a +10-point multi-vector advantage, or does the curve bend toward saturation at some scale? And does the single-vector baseline flatline completely, or does it resume improving at larger scales? A strong follow-up would train MetaEmbed on a 70B+ VLM (if available) or on a scaling series with more intermediate points (e.g., 3B, 7B, 14B, 32B, 70B) and fit scaling laws for both single-vector and multi-vector retrieval accuracy as a function of backbone parameters. The result would determine whether multi-vector representations are a permanent feature of the scaling landscape or a temporary advantage that narrows at larger scales. The paper's efficiency analysis would also need to scale: scoring cost at (16, 64) is independent of backbone size (the vectors are D-dimensional, and D grows with backbone size — 5,120 at 32B — so FLOPs and memory would increase), so a 70B model's efficiency characteristics need measurement.

Direct evaluation of multimodal-to-multimodal retrieval with MetaEmbed, closing the gap between the paper's motivating claim and its empirical coverage. The paper argues that MetaEmbed enables multimodal-to-multimodal retrieval because the fixed small cardinality (16 query vectors, 64 candidate vectors) makes late interaction between two image-containing inputs computationally feasible, while full token-level methods cannot handle this case (Section 2). This capability is never evaluated. A direct test would construct or use an existing image-to-image retrieval benchmark — for instance, the image-to-image retrieval tasks from the DeepFashion or SOP datasets, or a visual document-to-document retrieval task where queries and candidates are both page images — and compare MetaEmbed against: (a) single-vector methods (CLIP, SigLIP, MetaEmbed at (1,1)); (b) full token-level multi-vector methods (ColPali, ColQwen) where feasible (small corpora may allow brute-force comparison despite the cost); and (c) MetaEmbed at various retrieval budgets to show test-time scaling. A negative result — where MetaEmbed underperforms even single-vector methods on image-to-image retrieval — would reveal that the Meta Token aggregation loses fine-grained visual correspondence needed for this task, which would constrain the method's applicability. A positive result would validate the architectural claim and open up new application areas (visual search, duplicate detection, multimodal RAG with image queries).

Combining MetaEmbed's vector-count nesting with traditional Matryoshka embedding on the per-vector dimension, producing a two-axis test-time scaling mechanism. The paper nests on the number of vectors (r_q, r_c) but does not nest on the embedding dimension D. Table 3 shows that at (16, 64) with D = 3,584, the index requires 42.72 GiB for 100K candidates. Applying traditional MRL on D — training so that the first 896 dimensions of each vector are independently useful, the first 1,792 are more useful, etc. — would add a second test-time scaling knob: reduce D for storage-constrained deployments, reduce (r_q, r_c) for latency-constrained deployments, or reduce both for extremely tight budgets. A strong follow-up would train MetaEmbed with joint MMR (on vector count) and MRL (on dimension), producing a grid of operating points: (r_q, r_c) ∈ {(1,1), (2,4), ..., (16,64)} × D ∈ {896, 1792, 3584}. The experiment would measure whether the two nesting axes interact — does dimension truncation degrade the multi-vector advantage, or do they provide independent accuracy-efficiency tradeoffs? The practical impact would be significant for production deployments where both index storage (driven by r_c × D) and scoring latency (driven by r_q × r_c × D) must fit within hard resource budgets.

Investigating whether the Meta Token representations can be used for embedding-based reranking in a two-stage retrieval pipeline, with small-budget MetaEmbed for fast candidate filtering and large-budget MetaEmbed for top-k refinement. The paper only evaluates single-stage retrieval (score all candidates with one budget). In practice, large-scale retrieval systems use a two-stage architecture: a lightweight first-stage retriever (e.g., single-vector dot product) filters the corpus to, say, 1,000 candidates, and a heavier reranker scores only those candidates with a more expensive model. MetaEmbed's nested design is architecturally suited for this: use (1,1) or (2,4) for the first-stage retrieval over the full corpus, then apply (16,64) scoring only to the top-K candidates. The key question is whether the (1,1) MetaEmbed representation is sufficiently discriminative as a first-stage filter — does recall@1000 remain high enough that the reranker can recover the relevant candidates? A strong experiment would measure recall@K for first-stage retrieval at various budgets against a full-budget oracle, and end-to-end accuracy for the two-stage pipeline, on both MMEB and ViDoRe v2 with realistic corpus sizes (100K–1M candidates). The paper's Table 3 provides the latency budget: (1,1) scoring over 100K candidates takes 1.67 ms, while encoding the query takes 788 ms, so the first-stage scoring is essentially free relative to encoding. If recall is high (e.g., >95% at K=1000), this pipeline would make full-budget accuracy achievable on corpora where brute-force (16,64) scoring over all candidates is prohibitively slow.

Stress-testing MetaEmbed's robustness to backbone quality by systematically degrading the backbone VLM and measuring the impact on retrieval performance. The paper shows that MetaEmbed-11B (Llama) underperforms MetaEmbed-7B (Qwen2.5-VL) because the backbone is weaker on VQA. But this is a single comparison between two different architectures. A more systematic study would take a single VLM architecture (e.g., Qwen2.5-VL-7B) and degrade it in controlled ways — early-stopping pretraining at different checkpoints, fine-tuning on corrupted data, ablating specific pretraining objectives — then apply the MetaEmbed recipe and measure how retrieval performance degrades relative to the baseline VLM's generative performance. The goal is to map out which backbone capabilities are critical for MetaEmbed: is it VQA performance specifically? Cross-modal alignment quality as measured by some probe? Representation dimensionality or expressiveness? The result would provide guidance for VLM developers: if you want your model to serve as a strong backbone for retrieval fine-tuning, which pretraining objectives or data mixtures should you prioritize? This experiment would also test whether MetaEmbed's gains are robust to moderate backbone degradation (e.g., does a slightly weaker backbone still benefit from multi-vector representations, or does the advantage collapse once backbone quality falls below some threshold?).

Training MetaEmbed with independently selectable query and candidate budgets during MMR, rather than the paired (r_q, r_c) groups, to enable full cross-product flexibility at test time. The current MMR design trains five paired groups: (1,1), (2,4), (4,8), (8,16), (16,64). At test time, a user must select one of these five pairs; they cannot, for instance, index at (16,64) and query at (2,4) — the combination (2,64) was never explicitly trained, though it may work in practice since the query-side vectors at position 1 and 2 were trained with (2,4) and the candidate-side vectors at positions 1–64 were trained with (16,64). An extension would decouple the nesting: train with a set of query-side group sizes {1, 2, 4, 8, 16} and a set of candidate-side group sizes {1, 4, 8, 16, 64} independently, computing contrastive losses for all 25 combinations (or a sampled subset). This would guarantee that any (r_q, r_c) combination is well-formed at test time. The experiment would measure whether the added training complexity (25× more losses) degrades peak performance due to optimization difficulty, and whether the cross-product combinations actually achieve better accuracy-efficiency tradeoffs than the paired groups. A negative result — where the paired groups already provide sufficient flexibility and decoupling adds training cost without deployment benefit — would validate the paper's simpler design.

Practical Applications and Downstream Use Cases

Cost-efficient multi-tier visual document retrieval for enterprise search. Consider an enterprise with 10 million scanned documents (contracts, invoices, reports) that need to be searchable by text query. Indexing at full MetaEmbed-7B budget (16, 64) would require approximately 4,272 GiB of index memory — infeasible without massive sharding. A three-tier architecture enabled by MetaEmbed's nested design: (1) index all 10M documents at (1,1), requiring 68 GiB — fits on a single A100; (2) for each query, score all 10M at (1,1) in ~167 ms (extrapolating Table 3: 1.67 ms per 100K × 100 = 167 ms); (3) re-rank the top 1,000 candidates at (16,64) in 6.25 ms (cost per 100K, so 0.0625 ms for 1K). Total query latency: ~167 ms + 788 ms encoding = ~955 ms, acceptable for non-real-time search. This is only possible because the same trained model serves all three tiers — without MMR, the (1,1) first-stage filter would be a different model, or the full ColBERT-style index at patch granularity would be hundreds of terabytes. The paper's Table 2 ViDoRe v2 results (MetaEmbed-7B at 61.3% NDCG@5, leading all baselines) provide the accuracy justification for the (16,64) reranking tier; the (1,1) accuracy of 71.3% on MMEB (Figure 3b) suggests the first-stage filter retains high recall, though recall-at-K experiments are needed to confirm.

On-device multimodal retrieval with adaptive quality for varying network conditions. A mobile application that lets users search their personal photo library (10,000 images) using natural language queries. The application can ship with a quantized MetaEmbed-3B model encoding all photos at (2,4) — index size: 10K × 4 × 2,048 dimensions × 2 bytes (bfloat16) ≈ 164 MiB, easily stored on-device. Under normal conditions, queries are processed on-device at (2,4) with scoring over 10K candidates in <1 ms (ballpark from Table 3 scaling). When the user is on Wi-Fi and can offload to cloud, the app sends the (16,64) query embeddings and retrieves from a cloud index built at (16,64) with higher accuracy. The same trained model serves both tiers — the app ships the full MetaEmbed-3B weights, uses (2,4) locally, and (16,64) remotely. The paper's Figure 3a shows the accuracy gap: 65.8% at (2,4) vs. 69.1% at (16,64) for MetaEmbed-3B on MMEB. The +3.3-point gain from the cloud tier is meaningful for user satisfaction on difficult queries, while the on-device tier provides instant, private results for most queries. This application leverages the architecture's test-time scalability directly — the same model weights, different retrieval budgets, no model switching.

Training data filtering for multimodal generative models. Large-scale training of vision-language models (like the next generation of CLIP or SigLIP) requires filtering billions of image-text pairs for relevance — a retrieval problem where the text is the query and the image is the candidate. Using a single-vector filter (e.g., a CLIP model) may discard pairs where the text describes a specific detail in the image that a single vector cannot capture. Using a full ColBERT-style filter on billions of pairs is computationally infeasible. MetaEmbed offers a middle ground: filter at (2,4) to quickly eliminate obviously irrelevant pairs (scoring cost: 5.73 GFLOPs per query-candidate pair at 7B, or ~1 ms per 100K candidates), then re-rank borderline pairs at (16,64) for final inclusion decisions. The paper's MMEB Retrieval score for MetaEmbed-7B (78.7%, Table 1) substantially exceeds the best single-vector baseline on Retrieval tasks (75.0% for MoCa-7B), suggesting MetaEmbed would identify relevant pairs that single-vector filters miss. The ViDoRe v2 results (Table 2) further support this for document-image pairs. The economic value is in data quality: more relevant training pairs means better downstream model performance, and MetaEmbed's multi-vector expressiveness may recover pairs that single-vector filters incorrectly discard.